Opening Remarks
Tony Voellm (Google)
Opening Keynote - Evolution from Quality Assurance to Test Engineering
Ari Shamash (Google)
You built an app. You launched it. You figured you'd get it out there, build up some volume, get some funding, throw it all out, and then start from scratch so you can "do it right". But, demands for new features are sky high, you are now being asked to push towards unprecedented scale at an unheard of velocity. Yikes! Now what?
You cannot throw it away and start from scratch, you'll just need to evolve what you have, while continuing to add high quality features at breathtaking speed. In addition, you need to ensure that what is already there doesn't break. How do you do this? Fortunately, a new field is forming within the software engineering industry that addresses this common scenario: at Google, we call this "test engineering".
This talk will focus on what test engineering is, how it evolved from quality assurance, and how the industry as a whole has implemented test engineering (with specific examples of how it is implemented at Google).
Testing Systems at Scale @Twitter
James Waldrop (Twitter)
James will be discussing the tools, process, and philosophy that goes into performance testing at Twitter. Particular emphasis will be placed on the Iago open source load testing library, which he wrote to enable Twitter's engineering teams to perform load tests before deploying code to production. The talk will dive into implementation details of some of these tests (including source code) and how complicating factors such as OAuth and arbitrary Thrift protocols are managed.
How Do You Test a Mobile OS?
David Burns (Mozilla) and Malini Das (Mozilla)
This is the problem that confronted Mozilla when we decided to venture into the world of FirefoxOS. Where to start and how to do it was going to prove an interesting task. Come listen to how we solved this problem and how we created a new framework.
Mobile Automation in Continuous Delivery Pipeline
Igor Dorovskikh (Expedia) and Kaustubh Gawande (Expedia)
Expedia started investing in Mobile Web and iOS/Android apps in early 2012. At the same time Test Engineers started developing test automation solutions to build quality and testability into the products from the beginning. In this talk, we will share our experience and learning of utilizing open source tools to build automated testing in Expedia's Agile development and continuous delivery environment. We will talk about Test Pyramid and go into more detail of specific open source tools that have worked well for us. Some of the open source tools we use are BDD tools such as Cucumber, web automation tool Selenium-WebDriver, iOS automation tool Frank, Android automation tools Robotium and Calabash, and Continuous Integration system Jenkins. In addition, we will share some of the Agile delivery principles we strive to adopt like TDD, Pair Programming, Build and Test Radiators. Finally, we will share some of the benefits we have realized from our investment in Agile and test automation and how that is getting us to our Continuous Delivery goals.
Automated Set-Top Box Testing with GStreamer and OpenCV
David Röthlisberger (YouView)
We'll build a video-capturing image recognition system in 3 minutes, using GStreamer's command-line tools and OpenCV. (GStreamer is an open-source media-handling framework; OpenCV —"Open Computer Vision"— is an open-source image-processing library.)
A leading example of such a system is http://stb-tester.com, an open-source tool developed at YouView to automate the UI testing of our set-top boxes. We'll describe stb-tester, the flexibility offered by its GStreamer underpinnings, some of the possibilities it opens up, and the challenges ahead.
Webdriver for Chrome
Ken Kania (Google)
From its start as a Windows-only browser, Chrome has expanded to Mac, Linux, ChromeOS, and most recently, Android and iOS. User level testing of web applications across these platforms has been difficult and necessitated various automation approaches. This talk will describe the work the Chrome team is doing to make WebDriver available for Chrome on all platforms. This will include a technical look at the underlying approach but will focus on how developers can use the new ChromeDriver to write tests for Chrome's various platforms. Also, the current state of the project and a roadmap for its future will be covered.
Karma - Test Runner for JavaScript
Vojta Jina (Google)
Introduction to Karma - test runner that makes testing JavaScript applications in real browsers frictionless and enjoyable.
Testing is not optional when one is building a JavaScript application that must work across many browsers and devices. However executing tests in all of these various environments is hard. Karma turns this typically painstaking task into a piece of cake. It allows you to execute JavaScript tests in real browsers or devices such as your phone or tablet directly from the comfort of your terminal or your favorite IDE.
Automated Video Quality Measurements
Patrik Höglund (Google)
Yes, it is possible to automatically test complex, subjective measurements such as video quality! This talk will show how we constructed a continuous, automated end-to-end test of a WebRTC video call. We'll take a look at the toolchain at a high level and what challenges we ran into while constructing it. This is perfect if you want inspiration for how to take your media testing to the next level.
When Bad Things Happen to Good Applications...
Minal Mishra (Netflix)
The boom of mobile and tablet computing has inundated the software industry with application development platforms. Developing consumer applications on computing platforms have their own magical experience for the end users. Consumer facing software companies always attempt to put their best foot forward when they develop an application for these platforms. However, the biggest challenge in application development only begins after companies roll out the first version of the application. Consumers and the software companies want the latest features and functionalities out of development as soon as possible with the highest quality. This leads to constant code churn in every layer of the stack. We, UI automation engineers, build a variety of detection systems to catch application issues sooner than later. In this talk I will share some of our challenges and successes behind one such detection system, that helped find problems outside the application layer but still adversely impacted the user experience.
Testing for Educational Gaming and Educational Gaming for Testing
Tao Xie (North Carolina State University)
This talk presents Pex4Fun (http://www.pexforfun.com/), which leverages automated test generation to underpin automatic grading in an online programming system that can scale to hundreds of thousands of users. It provides a programming-oriented gaming experience outside of the classroom, training users to learn various programming and software engineering skills, including testing skills such as writing parameterized unit tests. Pex4Fun makes a significant contribution to the known problem of assignment grading, as well as providing a fun learning experience based on interactive gaming. Pex4Fun has been gaining high popularity in the community: since it was released to the public in June 2010, the number of clicks of the "Ask Pex!" button (indicating the attempts made by users to solve games at Pex4Fun) has reached over one million as of early 2013.
Closing Keynote - How Facebook Tests Facebook on Android
Simon Stewart (Facebook)
Facebook is one of the most popular Android applications there is. In this talk, you'll find what Facebook does to ensure that each release is as good as it can be. We'll cover everything from how we manage our code, through our approaches to testing and all the way out to dogfooding.