Opening Keynote - Testable JavaScript - Architecting Your Application for Testability
Mark Trostler (Google)
Testable JavaScript is a process. Whether starting with a blank slate or an already implemented application (or somewhere in-between) being able to test your JavaScript code simply, cleanly, and effectively is a necessary feature. Code that cannot be tested will be rewritten.
While JavaScript is unique due to the myriad of environments within which it runs, there are several tried and true 'testable' methodologies from other languages which also hold true for JavaScript. And of course there remain the unique challenges that JavaScript developers must face while writing and testing their code.
What patterns make code testable? Which anti-patterns hinder testing? What metrics and common sense guideposts can be used to measure the testability of our code? Once the process of creating testable code has started now what?
Join me to break down the process of writing testable JavaScript. We will investigate ideas, patterns, and methodologies that greatly increase the testability, and hence the maintainability, correctness, and longevity of your code. Whether you write client- or server-side JavaScript mastering this process will greatly enhance the quality of your code.
Breaking the Matrix - Android Testing at Scale
Thomas Knych (Google), Stefan Ramsauer (Google) and Valera Zakharov (Google)
Are you ready to take the red pill?
Mobile has changed the way humans interact with computers. This is great, but as engineers we're faced with an ever growing matrix of environments our code runs on. The days of considering only a handful of browsers and screen resolutions are not coming back. How can engineers cope with the Matrix? We'll cover how Google is fighting this testing problem on workstations, in the cloud and in your head...
"I'm trying to free your mind, Neo. But I can only show you the door. You're the one that has to walk through it."
Android UI Automation
Guang Zhu (朱光) (Google) and Adam Momtaz (Google)
As Android gains popularity in the mobile world, application developers and OEM vendors are exploring ways to perform end-to-end UI driven testing of applications or entire platform. With a brief review of existing UI Automation solutions on Android, this talk introduces the recently released Android UI Automator framework, and continues to give an inside look of the framework, typical use cases and workflows.
Appium: Automation for Mobile Apps
Jonathan Lipps (Sauce Labs)
Appium is is a Node.js server that automates native and hybrid mobile applications (both iOS and Android). Appium's philosophy dictates that apps should not be modified in order to be automated, and that you should be able to write your test code in any language or framework. The result is a Selenium WebDriver server that speaks mobile like a native. Appium runs on real devices and emulators and is completely open source, making it a wonderfully friendly way to get started with mobile test automation. In this talk I will outline the principles that inform Appium's design, talk about Appium in the space of other mobile automation frameworks, and introduce the architecture that makes the magic happen. Finally, I'll dig into the code for a simple test of a brand-new mobile app, and demonstrate Appium running this test on iPhone and Android.
Building Scalable Mobile Test Infrastructure for Google+ Mobile
Eduardo Bravo (Google)
Testing native apps in a meaningful, stable and scalable way is a challenge. The G+ have developed efficient solutions to tackle these problems by providing the right infrastructure for each of the complex test scenarios that mobile presents. Our current test infrastructure provides the right tools to both iOS and Android apps to give our development team the confidence that new changes won't break existing clients.
Espresso: Fresh Start to Android UI Testing
Valera Zakharov (Google)
Developing a reliable Android test should be as quick and easy as pulling a shot of espresso. Unfortunately, with existing tools, it may feel more like making a double-shot-caramel-sauce-upside-down-single-whip-half-decaf-latte - confusing and rarely consistent. Espresso is a new Android test framework that lets you write concise, beautiful, and reliable UI tests quickly. The core API is small, predictable, and easy to learn - yet it is also open for customization. Espresso tests state their expectations, interactions, and assertions clearly without distracting boilerplate, custom infrastructure, or messy implementation details getting in the way. Tests run optimally fast - leave your waits, syncs, sleeps, and polls behind and let the framework gracefully manipulate and assert on your UI when it is at rest. Start enjoying writing and executing UI tests - try a shot of Espresso.
Web Performance Testing with WebDriver
Michael Klepikov (Google)
In web performance testing, we know pretty well how to analyze a page load. We need to move beyond a page load though: modern apps are highly interactive, and operations tend not to reload the entire page, but rather update it. Different people, myself included, have integrated WebDriver into web performance test harnesses, which helps, but still keeps performance tests separate from the rest of the UI test suite. I propose to build performance testing features right into WebDriver itself, leveraging its recently added Logging API. This makes it possible to collect performance metrics while running regular functional tests, allowing for a much more seamless integration of performance tests into the overall development and test flow. It is also much less disruptive to the custom build/test toolchains that almost every large organization creates.
I will demonstrate this with the new-generation ChromeDriver (WebDriver for the Chromium browser).
Continuous Maps Data Testing
Yvette Nameth (Google) and Brendan Dhein (Google)
Continuous testing is generally about running unit tests and integration tests. But when the data that your server processes is actually the biggest cause of change, how do you ensure that consumers of the data still find it useable and that nothing crashes under the rate of change or a bad change? We'll discuss techniques for continuous data testing with examples from Google Maps.
Finding Culprits Automatically in Failing Builds - i.e. Who Broke the Build?
Celal Ziftci (UCSD) and Vivek Ramavajjala (UCSD)
Continuous build is one of the key infrastructures in Google. When a build fails, it is vital to pinpoint the culprit changelist (CL)/changelists quickly, so that it can be fixed to get the build back to green.
Culprit detection solutions exist for small and medium sized builds, but not for large integration builds.
Our culprit finder targets finding the culprit CL for large builds automatically, in a very short time-frame with high success. Based on production usage on multiple projects in the last 9 months, culprit finder provides very promising results. Come see our talk to see how we implemented the culprit finder, how successful it is in production and what it feels and looks like.
Empirical Investigation of Software Product Line Quality
Katerina Goseva-Popstojanova (West Virginia University)
Software product lines exhibit high degree of commonality among the systems in the product line and a well specified number of possible variations. Based on data extracted from two case studies - a medium size industrial product line and a large, evolving open source product line - we explored empirically if the systematic reuse improves the quality and supports successful prediction of potential future faults from previously experienced faults, source code metrics, and change metrics. Our research results confirmed, in a software product line setting, the findings of others that faults are more highly correlated to change metrics than to static code metrics. The quality assessment results showed that although older packages (including commonalities) continually changed, they retained low fault densities. Furthermore, the open source product line improved in quality as it evolved through releases. The prediction based on generalized linear regression models accurately ranked the packages according to their post-release faults using the models built on the previous release. The results also revealed that post-release fault predictions benefit from additional product line information.
AddressSanitizer, ThreadSanitizer and MemorySanitizer -- Dynamic Testing Tools for C++
Kostya Serebryany (Google)
AddressSanitizer (ASan) is a tool that finds buffer overflows (in stack, heap and globals) and use-after-free bugs in C/C++ programs. ThreadSanitizer (TSan) finds data races in C/C++ and Go programs. MemorySanitizer (MSan) is a work-in-progress tool that finds uses of uninitialized memory (C++). These tools are based on compiler instrumentation (LLVM and GCC), which makes them very fast (e.g. ASan incurs just 2x slowdown). We will share our experience in huge scale testing using these tools.
Closing Keynote - Drinking the Ocean - Finding XSS at Google Scale
Claudio Criscione (Google)
Cross site scripting, or XSS, is the modern-day equivalent of the middle-ages black plague in the web application world: it's widespread, it's bad and there are little or no technical ways to detect it until it's too late. DOM XSS is a particularly nasty variant of those, as it requires a real browser or equivalent to be detected: a difficult problem with little automated solution available.
We needed powerful, self-driving tools to identify DOM XSS early in the development lifecycle, usable by engineers outside of the security team: all we wanted was a product which could scan our huge, fast moving, highly complex and arcane corpus of applications... and of course, we found none. So we built our own: a web application scanner targeting DOM XSS designed on top of standard Google technologies. It runs in App Engine and leverages the powerful Chrome browser and some hundreds of CPUs as a security scanning platform.
It is also a nice citizen of the testing arsenal at Google: it lives inside our testing infrastructure, instead of being the instrument of the security team.
In this talk we outline our novel approach, the challenges we faced in scaling our system to Google size and the ideas behind our detection and crawling models on JavaScript intensive applications.