Python Runtime Environment
Welcome to Google App Engine for Python! With App Engine, you can build web applications using the Python programming language, and take advantage of the many libraries, tools and frameworks for Python that professional developers use to build world-class web applications. Your Python application runs on Google's scalable infrastructure, and uses large-scale persistent storage and services.
- Selecting the Python runtime
- Requests and domains
- Request headers
- The request timer
- The sandbox
- Pure Python
- App caching
- The environment
- Quotas and limits
- The Datastore and services
- Scheduled tasks
- Python tools
App Engine executes your Python application code using a pre-loaded Python interpreter in a safe "sandboxed" environment. Your app receives web requests, performs work, and sends responses by interacting with this environment.
A Python web app interacts with the App Engine web server using the WSGI protocol, so apps can use any WSGI-compatible web application framework. App Engine includes a simple web application framework, called webapp2, to make it easy to get started. For larger applications, mature third-party frameworks, such as Django, work well with App Engine.
The Python interpreter can run any Python code, including Python modules you include with your application, as well as the Python standard library. The interpreter cannot load Python modules with C code; it is a "pure" Python environment.
The secured "sandbox" environment isolates your application for service and security. It ensures that apps can only perform actions that do not interfere with the performance and scalability of other apps. For instance, an app cannot write data to the local file system or make arbitrary network connections. Instead, apps use scalable services provided by App Engine to store data and communicate over the Internet. The Python interpreter raises an exception when an app attempts to import a module from the standard library known to not work within the sandbox restrictions.
If you haven't already, see the Python 2.7 Getting Started Guide for an interactive introduction to developing web applications with Python and Google App Engine.
Selecting the Python runtime
App Engine knows to use the Python runtime environment for your application code when you use the tool named
appcfg.py from the Python SDK with a configuration file named
You specify the
runtime element in
app.yaml. To use Python 2.7, add the following to
runtime: python27 api_version: 1 threadsafe: true ...
The first element,
runtime, selects the Python runtime environment. App Engine supports four runtime environments:
python27– Python 2.7.
python– the deprecated Python 2.5 runtime. See the differences from Python 2.7 for more information.
java– See the Java runtime documentation for more information.
go– See the Go runtime documentation for more information.
The second element,
api_version, selects which version of the Python runtime environment to use. As of this writing, App Engine only has one version of the Python environment,
1. If the App Engine team ever needs to release changes to the environment that may not be compatible with existing code, they will do so with a new version identifier. Your app will continue to use the selected version until you change the
api_version setting and upload your app.
Requests and domains
App Engine determines that an incoming request is intended for your application using the domain name of the request. A request whose domain name is
http://your_app_id.appspot.com is routed to the application whose ID is
your_app_id. Every application gets an
appspot.com domain name for free.
appspot.com domains also support subdomains of the form
subdomain can be any string allowed in one part of a domain
.). Requests sent to any subdomain in this way are routed to your application.
You can set up a custom top-level domain using Google Apps. With Google Apps, you assign subdomains of your business's domain to various applications, such as Google Mail or Sites. You can also associate an App Engine application with a subdomain. For convenience, you can set up a Google Apps domain when you register your application ID, or later from the Administrator Console. See Deploying your Application on your Google Apps URL for more information.
Requests for these URLs all go to the version of your application that you have selected as the default version in the Administration Console. Each version of your application also has its own URL, so you can deploy and test a new version before making it the default version. The version-specific URL uses the version identifier from your app's configuration file in addition to the
appspot.com domain name, in this pattern:
http://version_id-dot-latest-dot-your_app_id.appspot.com You can also use subdomains with the version-specific URL:
The domain name used for the request is included in the request data passed to the application. If you want your app to respond differently depending on the domain name used to access it (such as to restrict access to certain domains, or redirect to an official domain), you can check the request data (such as the
Host request header) for the domain from within the application code and respond accordingly.
If your app uses backends, you can address requests to a specific backend and a specific instance with that backend. For more information about backend addressability, please see Properties of Backends.
Please note that in April of 2013, Google stopped issuing SSL certificates for double-wildcard domains hosted at
*.*.appspot.com). If you rely on such URLs for HTTPS access to your application, please change any application logic to use "-dot-" instead of ".". For example, to access version "1" of application "myapp" use "https://1-dot-myapp.appspot.com" instead of "https://1.myapp.appspot.com." If you continue to use "https://1.myapp.appspot.com" the certificate will not match, which will result in an error for any User-Agent that expects the URL and certificate to match exactly.
When App Engine receives a web request for your application, it calls the handler script that corresponds to the URL, as described in the application's app.yaml configuration file. If your application uses backends, it may also call the handler script defined in
backends.yaml. See Backend Configuration for more information.
The Python 2.7 runtime supports the WSGI standard and the CGI standard for backwards compatibility. WSGI is preferred, and some features of Python 2.7 do not work without it. The configuration of your application's script handlers determines whether a request is handled using WSGI or CGI. The Python 2.5 runtime always uses the CGI standard.
App Engine uses multiple web servers to run your application, and automatically adjusts the number of servers it is using to handle requests reliably. A given request may be routed to any server, and it may not be the same server that handled a previous request from the same user.
If you mark your application as thread-safe, concurrent requests will be enabled, which means that App Engine may dispatch multiple requests to each web server in parallel. To do so, simply set
threadsafe: true in
app.yaml as described in Using Concurrent Requests. Concurrent requests are only available for Python 2.7 apps, and are not allowed if any script handler uses CGI.
Requests and WSGI
The server determines which Python application object to call by comparing the URL of the request to the URL patterns in the app's configuration file. It then calls the application object using the arguments as defined in the WSGI standard. The application object performs actions appropriate to the request, then prepares a response and returns it as a list of strings.
Most applications use a framework, such as webapp2, to handle WSGI requests.
webapp2 is included as part of the Python 2.7 runtime.
Requests and CGI
The server determines which Python handler script to run by comparing the URL of the request to the URL patterns in the app's configuration file. It then runs the script in a CGI environment populated with the request data. As described in the CGI standard, the server puts the request data in environment variables and the standard input stream. The script performs actions appropriate to the request, then prepares a response and puts it on the standard output stream.
Most applications use a library to parse CGI requests and return CGI responses, such as the cgi module from the Python standard library, or a web framework that knows the CGI protocol (such as webapp). You can refer to the CGI documentation for details about the environment variables and the format of the input stream data.
The following example handler script displays a message on the user's browser. It prints an HTTP header that identifies the type of the message and the content of message to the standard output stream.
print "Content-Type: text/plain" print "" print "Hello, world!"
An incoming HTTP request includes the HTTP headers sent by the client. For security purposes, some headers are sanitized or amended by intermediate proxies before they reach the application.
The following headers are removed from the request:
In addition, the header
Strict-Transport-Security is removed from requests served to any domains other than
These headers relate to the transfer of the HTTP data between the client and server, and are transparent to the application. For example, the server may automatically send a gzipped response, depending on the value of the
Accept-Encoding request header. The application itself does not need to know which content encodings the client can accept.
As a service to the app, App Engine adds some headers:
- Country from which the request originated, as an ISO 3166-1 alpha-2 country code. App Engine determines this code from the client's IP address.
- Name of region from which the request originated. This value only makes sense in the context of the country in
X-AppEngine-Country. For example, if the country is "US" and the region is "ca", that "ca" means "California", not Canada.
- Name of the city from which the request originated. For example, a request from the city of Mountain View might have the header value
- Latitude and longitude of the city from which the request originated. This string might look like "37.386051,-122.083851" for a request from Mountain View.
App Engine collects all of the data the request handler script writes to the standard output stream, then waits for the script to return. When the handler completes, all of the output data is sent to the user.
App Engine does not support sending data to the user's browser before the handler returns. Some web servers use this technique to "stream" data to the user's browser over a period of time in response to a single request. App Engine does not support this streaming technique.
Dynamic responses are limited to 32MB. If a script handler generates a response larger than this limit, the server sends back an empty response with a 500 Internal Server Error status code. This limitation does not apply to responses that serve data from the Blobstore or Google Cloud Storage.
If the client sends HTTP headers with the request indicating that the client can accept compressed (gzipped) content, App Engine compresses the response data automatically and attaches the appropriate response headers. It uses both the
User-Agent request headers to determine if the client can reliably receive compressed responses. Custom clients can indicate that they are able to receive compressed responses by specifying both
User-Agent headers with a value of
Content-Type of the response is also used to determine whether compression is appropriate; in general, text-based content types are compressed, whereas binary content types are not.
The following headers are ignored and removed from the response:
In addition, the header
Strict-Transport-Security is removed from responses served from any domains other than
Headers with non-ASCII characters in either the name or value are also removed. In addition, the following headers are added or replaced in the response:
These headers specify caching policy to intermediate web proxies (such as Internet Service Providers) and browsers. If your script sets these headers, they will usually be unmodified, unless the response has a Set-Cookie header, or is generated for a user who is signed in using an administrator account. Static handlers will set these headers as directed by the configuration file. If you do not specify a
Cache-Control, the server may set it to
private, and add a
If you have a Set-Cookie response header, the
Cache-Controlheader will be set to
private(if it is not already more restrictive) and the
Expiresheader will be set to the current date (if it is not already in the past). Generally, this will allow browsers to cache the response, but not intermediate proxy servers. This is for security reasons, since if the response was cached publicly, another user could subsequently request the same resource, and retrieve the first user's cookie.
- Depending upon the request headers and response
Content-Type, the server may automatically compress the response body, as described above. In this case, it adds a
Content-Encoding: gzipheader to indicate that the body is compressed.
- The server always ignores the
Content-Lengthheader returned by the application. It will either set
Content-Lengthto the length of the body (after compression, if compression is applied), or delete
Content-Length, and use chunked transfer encoding (adding a
If not specified by the application, the server will set a default
- Set to the current date and time.
- Set to
Google Frontend. The development server sets this to
Development/x, where x is the version number.
If you access your site while signed in using an administrator account, App Engine includes per-request statistics in the response headers:
- An estimate of what 1,000 requests similar to this request would cost in US dollars.
- The resources used by the request, including server-side time as a number of milliseconds.
Responses with resource usage statistics will be made uncacheable.
X-AppEngine-BlobKey header is in the application's response, it and the optional
X-AppEngine-BlobRange header will be used to replace the body with all or part of a blobstore blob's content. If
Content-Type is not specified by the application, it will be set to the blob's MIME type. If a range is requested, the response status will be changed to
206 Partial Content, and a
Content-Range header will be added. The
X-AppEngine-BlobRange headers will be removed from the response. You do not normally need to set these headers yourself, as the
blobstore_handlers.BlobstoreDownloadHandler class sets them. See Serving a Blob for details.
The request timer
A request handler has a limited amount of time to generate and return a response to a request, typically around 60 seconds. Once the deadline has been reached, the request handler is interrupted.
The Python runtime environment interrupts the request handler by raising a
DeadlineExceededError, from the package
google.appengine.runtime. If the request handler does not catch this exception, as with all uncaught exceptions, the runtime environment will return an HTTP 500 server error to the client.
The request handler can catch this error to customize the response. The runtime environment gives the request handler a little bit more time (less than a second) after raising the exception to prepare a custom response.
from google.appengine.runtime import DeadlineExceededError class MainPage(webapp2.RequestHandler): def get(self): try: # Do stuff... except DeadlineExceededError: self.response.clear() self.response.set_status(500) self.response.out.write("This operation could not be completed in time...")
If the handler hasn't returned a response or raised an exception by the second deadline, the handler is terminated and a default error response is returned.
While a request can take as long as 60 seconds to respond, App Engine is optimized for applications with short-lived requests, typically those that take a few hundred milliseconds. An efficient app responds quickly for the majority of requests. An app that doesn't will not scale well with App Engine's infrastructure.
Refer to Dealing with DeadlineExceededErrors for common DeadlineExceededError causes and suggested workarounds.
Backends allow you to avoid this request timer; with backends, there is no time limit for generating and returning a request.
To allow App Engine to distribute requests for applications across multiple web servers, and to prevent one application from interfering with another, the application runs in a restricted "sandbox" environment. In this environment, the application can execute code, store and query data in the App Engine datastore, use the App Engine mail, URL fetch and users services, and examine the user's web request and prepare the response.
An App Engine application cannot:
write to the filesystem. Applications must use the App Engine datastore for storing persistent data. Reading from the filesystem is allowed, and all application files uploaded with the application are available.
respond slowly. A web request to an application must be handled within a few seconds. Processes that take a very long time to respond are terminated to avoid overloading the web server.
make other kinds of system calls.
Sandboxing in Python
The Python 2.7 runtime does not restrict access to Python bytecode. Libraries that generate or manipulate bytecode (e.g. the jinja2 templating library) can do so in this runtime. The Python 2.5 runtime does not allow bytecode manipulation.
You can upload and use
.pyc files when using the Python 2.7 runtime, but you cannot upload a
.py and a
.pyc version of the same file. You can upload zip files containing
.pyc files (or a combination). A number of important caveats apply if you upload
- For a CGI script, the script handler should still use the
.pyfile extension, even if you upload a
- By default,
.pycfiles are excluded by
appcfg. To work around this, you will need to override the
app.yamlto not include
- You must use Python 2.7 to build the
.pycfile. If you have a different version of Python (such as Python 2.6) on your development machine, you will need to obtain version 2.7 to build a compatible
Threads can be created in Python 2.7 using the
threading modules. Note that threads will be joined by the runtime when the request ends so the threads cannot run past the end of the request. On a backend server, you can spawn a background thread; a background thread can "outlive" the request that spawns it.
All code for the Python runtime environment must be pure Python, and not include any C extensions or other code that must be compiled.
The environment includes the Python standard library. Some modules have been disabled because their core functions are not supported by App Engine, such as networking or writing to the filesystem. In addition, the
os module is available, but with unsupported features disabled. An attempt to import an unsupported module or use an unsupported feature will raise an exception.
You can include other pure Python libraries with your application by putting the code in your application directory. If you make a symbolic link to a module's directory in your application directory, appcfg.py will follow the link and include the module in your app.
The Python module include path includes your application's root directory (the directory containing the
app.yaml file). Modules you create in your application's root directory are available using a path from the root. Don't forget to create
__init__.py files in sub-directories, so Python will recognize the sub-directories as packages.
A few modules from the standard library have been replaced or customized to work with App Engine. These modules vary between the two Python runtimes, as described below.
Customized Libraries in Python 2.7
In the Python 2.7 runtime, the following modules have been replaced or customized:
In addition to the Python standard library and the App Engine libraries, the Python 2.7 runtime includes several third-party libraries.
The Python runtime environment caches imported modules between requests on a single web server, similar to how a standalone Python application loads a module only once even if the module is imported by multiple files. Since WSGI handlers are modules, they are cached between requests. CGI handler scripts are only cached if they provide a
main() routine; otherwise, the CGI handler script is loaded for every request. See Requests for more information about the differences between WSGI and CGI.
App caching provides a significant benefit in response time. We recommend that all CGI handler scripts use a main() routine, as described below.
Imports are cached
For efficiency, the web server keeps imported modules in memory and does not re-load or re-evaluate them on subsequent requests to the same application on the same server. Most modules do not initialize any global data or have other side effects when they are imported, so caching them does not change the behavior of the application.
If your application imports a module that depends on the module being evaluated for every request, the application must accommodate this caching behavior.
The following example demonstrates how an imported module is cached. Because
mymodule is only imported once for a single web server, the global
mymodule.counter is only initialized to
0 on the first request served by the server. Subsequent requests use the value from the previous request.
### mymodule.py counter = 0 def increment(): global counter counter += 1 return counter ### myhandler.py import webapp2 import mymodule class MyHandler(webapp2.RequestHandler): def get(self): self.response.headers["Content-Type"] = "text/plain" self.response.out.write("My number: %d" % mymodule.increment()) app = webapp2.WSGIApplication([("/", MyHandler)], debug=True)
My number: # where
# is the number of times this handler has been called by the web server that handled the request.
CGI handler scripts can also be cached
You can tell App Engine to cache the CGI handler script itself, in addition to imported modules. If the handler script defines a function named
main(), then the script and its global environment will be cached like an imported module. The first request for the script on a given web server evaluates the script normally. For subsequent requests, App Engine calls the
main() function in the cached environment.
To cache a handler script, App Engine must be able to call
main() with no arguments. If the handler script does not define a
main() function, or the
main() function requires arguments (that don't have defaults), then App Engine loads and evaluates the entire script for every request.
Keeping the parsed Python code in memory saves time and allows for faster responses. Caching the global environment has other potential uses as well:
Compiled regular expressions. All regular expressions are parsed and stored in a compiled form. You can store compiled regular expressions in global variables, then use app caching to re-use the compiled objects between requests.
GqlQuery objects. The GQL query string is parsed when the GqlQuery object is created. Re-using a GqlQuery object with parameter binding and the bind() method is faster than re-constructing the object each time. You can store a GqlQuery object with parameter binding for the values in a global variable, then re-use it by binding new parameter values for each request.
- Configuration and data files. If your application loads and parses configuration data from a file, it can retain the parsed data in memory to avoid having to re-load the file with each request.
The handler script should call
main() when imported. App Engine expects that importing the script calls
main(), so App Engine does not call it when loading the request handler for the first time on a server.
The following example does the same thing as the previous example, using caching of the handler script's global environment:
### myhandler.py # A global variable, cached between requests on this web server. counter = 0 def main(): global counter counter += 1 print "Content-Type: text/plain" print "" print "My number: " + str(counter) if __name__ == "__main__": main()
App caching with
main() provides a significant improvement in your CGI handler's response time. We recommend it for all applications that use CGI.
The App Engine web server captures everything the handler script writes to the standard output stream for the response to the web request. It also captures everything the handler script writes to the standard error stream, and stores it as log data. Each request is assigned a request ID, a globally unique identifier based on the request's start time. Log data for your application can be viewed and analyzed using the Administration Console, or downloaded using appcfg.py request_logs.
The App Engine Python runtime environment includes special support for the logging module from the Python standard library to understand logging concepts such as log levels ("debug", "info", "warning", "error", "critical").
import logging from google.appengine.api import users from google.appengine.ext import db user = users.get_current_user() if user: q = db.GqlQuery("SELECT * FROM UserPrefs WHERE user = :1", user) results = q.fetch(2) if len(results) > 1: logging.error("more than one UserPrefs object for user %s", str(user)) if len(results) == 0: logging.debug("creating UserPrefs object for user %s", str(user)) userprefs = UserPrefs(user=user) userprefs.put() else: userprefs = results else: logging.debug("creating dummy UserPrefs for anonymous user")
The execution environment automatically sets several environment variables; you can set more in
app.yaml. Of the automatically-set variables, some are special to App Engine, while others are part of the WSGI or CGI standards. Python code can access these variables using the
The following environment variables are specific to App Engine:
CURRENT_VERSION_ID: The major and minor version of the currently running application, as "X.Y". The major version number ("X") is specified in the app's
app.yamlfile. The minor version number ("Y") is set automatically when each version of the app is uploaded to App Engine. On the development web server, the minor version is always "1".
AUTH_DOMAIN: The domain used for authenticating users with the Users API. Apps hosted on appspot.com have an
gmail.com, and accept any Google account. Apps hosted on a custom domain using Google Apps have an
AUTH_DOMAINequal to the custom domain.
INSTANCE_ID: Contains the instance ID of the frontend instance handling a request. The ID is a hex string (for example,
00c61b117c7f7fd0ce9e1325a04b8f0df30deaaf). A logged-in admin can use the id in a url: http://[INSTANCE_ID].myApp.appspot.com/. The request will be routed to that specific frontend instance. If the instance cannot handle the request it returns an immediate 503.
The following environment variables are part of the WSGI and CGI standards, with special behavior in App Engine:
SERVER_SOFTWARE: In the development web server, this value is "Development/X.Y" where "X.Y" is the version of the runtime. When running on App Engine, this value is "Google App Engine/X.Y.Z".
You can also set environment variables in the
env_variables: DJANGO_SETTINGS_MODULE: 'myapp.settings'
import os import webapp2 class PrintEnvironmentHandler(webapp2.RequestHandler): def get(self): for name in os.environ.keys(): self.response.out.write("%s = %s<br />\n" % (name, os.environ[name]))
At the time of the request, you can save the request ID, which is unique to that request. The request ID can be used later to look up the logs for that request, using the
The following sample code shows how to get the request ID in the context of a request:
import os import webapp2 class MainPage(webapp2.RequestHandler): def get(self): self.response.headers['Content-Type'] = 'text/plain' self.response.out.write('REQUEST_LOG_ID=' + (os.environ.get('REQUEST_LOG_ID')) + '\n') app = webapp2.WSGIApplication([('/', MainPage)])
Quotas and limits
Google App Engine automatically allocates resources to your application as traffic increases. However, this is bound by the following restrictions:
- App Engine reserves automatic scaling capacity for applications with low latency, where the application responds to requests in less than one second. Applications with very high latency (over one second per request for many requests) and high throughput require Silver, Gold, or Platinum support. Customers with this level of support can request higher throughput limits by contacting their support representative.
- Applications that are heavily CPU-bound may also incur some additional latency in order to efficiently share resources with other applications on the same servers. Requests for static files are exempt from these latency limits.
Each incoming request to the application counts toward the Requests limit. Data sent in response to a request counts toward the Outgoing Bandwidth (billable) limit.
Both HTTP and HTTPS (secure) requests count toward the Requests, Incoming Bandwidth (billable), and Outgoing Bandwidth (billable) limits. The Quota Details page of the Admin Console also reports Secure Requests, Secure Incoming Bandwidth, and Secure Outgoing Bandwidth as separate values for informational purposes. Only HTTPS requests count toward these values. See the Quotas page, and the "Quota Details" section of the Admin Console for more information.
In addition to system-wide safety limits, the following limits apply specifically to the use of request handlers:
|request size||32 megabytes|
|response size||32 megabytes|
|request duration||60 seconds|
|maximum total number of files (app files and static files)||10,000 total
1,000 per directory
|maximum size of an application file||32 megabytes|
|maximum size of a static file||32 megabytes|
|maximum total size of all application and static files||first 1 gigabyte is free
$0.13 per gigabyte per month after first 1 gigabyte
App Engine applications will automatically use the SPDY protocol when accessed over SSL by a browser that supports SPDY. This is a replacement for HTTP designed by Google and intended to reduce the latency of web page downloads. The use of SPDY should be entirely transparent to both applications and users (applications can be written as if normal HTTP was being used). For more information, see the SPDY project page.
The Datastore and services
Apps can use the App Engine Datastore for reliable, scalable persistent storage of data. The Python API to the App Engine datastore includes rich data modeling tools for managing data schemas. The API supports two interfaces for performing datastore queries, including GQL, a SQL-like query language that is also used in the Admin Console.
The App Engine Memcache provides fast, transient distributed storage for caching the results of datastore queries and calculations. The Python interface to the App Engine memcache is compatible with the Python Memcached API.
Apps use the URL Fetch service to access resources over the web, and to communicate with other hosts using the HTTP and HTTPS protocols. Python applications can use the urllib, urllib2, or httplib modules from the Python standard library to access this service, or they can use the App Engine URL Fetch service API.
An app can use the Mail service to send email messages on behalf of the application's administrators, or on behalf of the currently signed-in user.
The Images service lets applications transform and manipulate image data in several formats, including cropping, rotating, resizing, and photo color enhancement.
An application can use Google Accounts for user authentication. Google Accounts handles user account creation and sign-in, and a user that already has a Google account (such as a Gmail account) can use that account with your app. An app can detect when the current user is signed in, and can access the user's email address. The Python API can return user data in an object that can be stored directly in the datastore.
An application can configure scheduled tasks that will call URLs of the application at specified intervals. For more on this, see Cron Jobs.
The App Engine Python SDK includes tools for testing your application, uploading your application files, managing datastore indexes, downloading log data, and uploading large amounts of data to the datastore.
The development server runs your application on your local computer for testing your application. The server simulates the App Engine datastore, services and sandbox restrictions. The development server can also generate configuration for datastore indexes based on the queries the app performs during testing.
A multipurpose tool called appcfg.py handles all command-line interaction with your application running on App Engine.
appcfg.py can upload your application to App Engine, or just update the datastore index configuration so you can build new indexes before updating the code. It can also download the app's log data, so you can analyze your app's performance using your own tools.
The Python SDK includes a data upload tool that can add data to your app's datastore from your local data files. The tool can extract data from CSV files, a spreadsheet format supported by most spreadsheet software such as Google Docs or Microsoft Excel. You can customize how CSV files are converted into datastore entities using Python code.