Behind the Data Sensing Lab: Gathering, Processing, and Analyzing Data at Scale using the Google Cloud Platform
Amy Unruh, Kim Cameron
Highly scalable and rapid data collection and analysis is a key need for many mobile and gaming apps, as well as for sensor networks and the "Internet of Things." We'll show how the Data Sensing Lab incorporates a key Google Cloud Platform pattern: a high-throughput pipeline for data collection, processing, and analysis. We use the Cloud Endpoints API to collect constantly streaming data; process large amounts of data with high throughput using App Engine, Cloud Storage, and data transformation on Compute Engine; and query many GBs of collected data in just a few seconds using BigQuery.
Amy is a Developer Programs Engineer for the Google Cloud Platform, with a current focus on App Engine. She has an academic background, and has also worked at several startups, worked in industrial R&D, done technical training and course development, and has published a book on "Google App Engine Java and GWT Application Development". Based in Sydney, Amy supports the Cloud Platform in APAC.
Kim is the technical writer for BigQuery with a computer science degree and a passion for all things cloud. In her past life, Kim worked for many years on phone SDK docs and code. Outside of work she loves cooking, photography, gaming and robots, amongst other nerdy inclinations.