Behind the Data Sensing Lab: Gathering, Processing, and Analyzing Data at Scale using the Google Cloud Platform

Highly scalable and rapid data collection and analysis is a key need for many mobile and gaming apps, as well as for sensor networks and the "Internet of Things." We'll show how the Data Sensing Lab incorporates a key Google Cloud Platform pattern: a high-throughput pipeline for data collection, processing, and analysis. We use the Cloud Endpoints API to collect constantly streaming data; process large amounts of data with high throughput using App Engine, Cloud Storage, and data transformation on Compute Engine; and query many GBs of collected data in just a few seconds using BigQuery.