# GTAC 2015: Presentations

## Opening Remarks

[Yvette Nameth](https://developers.google.com/google-test-automation-conference/2015/speakers#YvetteNameth) (Google)

## Opening Keynote

Jürgen Allgayer (Google)

## The Uber Challenge of Cross-Application/Cross-Device Testing

[Apple Chow](https://developers.google.com/google-test-automation-conference/2015/speakers#AppleChow) (Uber) and
[Bian Jiang](https://developers.google.com/google-test-automation-conference/2015/speakers#BianJiang) (Uber)


Links:
[Video](https://www.youtube.com/watch?v=p6gsssppeT0&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=3),
[Slides](https://docs.google.com/presentation/d/1vYXhkvgLKun72Ix91LQDDWZQdcY5VOBqKVvI1Y6riYo/pub)


Soon after joining Uber in March 2015, we encountered an Uber-unique challenge while investigating
UI testing tools for our mobile applications. Many of our sanity tests require our rider application
and driver application communicating/coordinating their actions with each other in order to complete
the end-to-end testing scenario. In this talk, we will present our platform agnostic solution,
called Octopus, and discuss how it coordinates communication across different apps running on
different devices. This solution can be adopted for any tests that require coordination/communication
across different apps or devices (e.g. a multi-user game, multi-user messaging/communication app, etc.)

## Robot Assisted Test Automation

[Hans Kuosmanen](https://developers.google.com/google-test-automation-conference/2015/speakers#HansKuosmanen) (OptoFidelity)
and [Natalia Leinonen](https://developers.google.com/google-test-automation-conference/2015/speakers#NataliaLeinonen) (OptoFidelity)


Links:
[Video](https://www.youtube.com/watch?v=oQRrk7S9sUE&index=4&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1DxEX7d3UZKFv-ekHzt1-eRCWtlR8uAyXt6ra1B14IGQ/pub)


OptoFidelity is a Finnish high-tech company with 10 years of experience in developing and delivering
R\&D test automation solutions. This talk will include our experiences and future outlook of
non-intrusive test methods used in mobile device UI performance testing. Did you know that Chrome
OS team uses a robot solution from OptoFidelity to measure end-to-end latency of Android and ChromeOS devices?

## Juggling Chainsaws for Fun and Profit: Lessons Learned from Mobile Cross-Platform Integration Testing

[Dan Giovannelli](https://developers.google.com/google-test-automation-conference/2015/speakers#DanGiovannelli) (Google)


Links:
[Video](https://www.youtube.com/watch?v=jaWD60t69Wg&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=5),
[Slides](https://docs.google.com/presentation/d/1Z2lWJDsZu_iyr8wFLodG-6R27vB_EodcbN0euqCXgz0/pub)


Mobile development is hard. Building test infrastructure is hard. Working cross-platform is hard.
Combine the three and you have a recipe for disaster. In this talk, Dan Giovannelli will share his
experiences working on a cross-platform mobile test infrastructure project. He'll discuss the things
that went right, the things that went (very) wrong, and what he wishes now he knew starting out.
Come for insights on designing mobile tools for non-mobile engineers, stay to find out just what
the heck The Matrix is and how to beat it at its own game.

## Mobile Game Test Automation Using Real Devices

[Jouko Kaasila](https://developers.google.com/google-test-automation-conference/2015/speakers#JoukoKaasila) (Bitbar/Testdroid)


Links:
[Video](https://www.youtube.com/watch?v=WFBfRk-GLRo&index=6&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1SrxWmnYsRRkEEMz_g-hLi6kfFVaSWVs4uAK658IXyYI/pub)


Mobile Games are the biggest money making category in today's app stores so ensuring that every
version of each game works on any user's device is a high priority for any game developer. Despite
the importance of validating this, there are very few examples or frameworks for automating testing
of mobile games, forcing game developers to resort to manual testing that doesn't scale to the
extent that gaming companies would need to cover their global market. One main reason is the unique
nature of games as mobile apps because they are accessing the screen directly and bypassing all UI
services provided by the OS and rendering most test automation frameworks useless since traditional
objects are not exposed.


Luckily there are ways to use standard mobile test automation frameworks to drive test automation
on real mobile devices for games by using some creativity and publically available libraries. In
his presentation, Jouko Kaasila from Testdroid will discuss three different approaches with real
world examples and some sample code.

## How to Component Test Soup Dumplings

[Toni Chang](https://developers.google.com/google-test-automation-conference/2015/speakers#ToniChang) (Google)


Links:
[Video](https://www.youtube.com/watch?v=OLesBZSd-cM&index=7&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1x7FjkFqLjwOTNUM_Mgag8Ar_MXQ5HZsQ-18CVqoR5MQ/pub)


People that have spent too much time stabilizing flaky tests would agree that we need to decompose
tests. However, some find it difficult and not sure how, others may be challenged by teammates that
believe we need E2E test to validate all scenarios. Since it is sometimes difficult to get the idea
when you are not used to viewing your product in components, I will use an abstract example of a Soup
Dumpling to demonstrate how to break down what seems to be an inseparable piece to components and
apply tests to them.


I will take you through a fun journey of translating E2E test into component test that will give
you confidence in the final product. Hopefully this will give you a fresh perspective when you look
at your own product.

## Chromecast Test Automation

[Brian Gogan](https://developers.google.com/google-test-automation-conference/2015/speakers#BrianGogan) (Google)


Links:
[Video](https://www.youtube.com/watch?v=TfAVJLhb_k0&index=8&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1gBGS5UFyyR0mcwg990I1YDOFQwAhL-nmonNPGvGYnlk/pub)


The internet of things has led to a proliferation of connected devices. Validating behavior across
diverse interoperating devices is a significant testing challenge. To test Chromecast, several
approaches were taken. We outline test frameworks, lab infrastructure and test tooling that we
developed to generate reliable quality signals from the product. We detail the challenges of testing
a product that operates in noisy networked environments. We propose that test tooling for devices
such as Chromecast is in its infancy and presents opportunities for innovation in software test engineering.

## Using Robots for Android App Testing

[Dr.Shauvik Roy Choudhary](https://developers.google.com/google-test-automation-conference/2015/speakers#DrShauvikRoyChoudhary)
(Georgia Tech/Checkdroid)


Links:
[Video](https://www.youtube.com/watch?v=Vt8EUQtqBuA&index=9&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1pvAzl2L6X6CLdPI3IgZscZV6gCPz-n56pZ1VRRsYlZI/pub)


Software robots, such as Monkey can be used to test an android application without much manual effort.
There are several such tools proposed in academia whose goal is to automatically generate test input
to drive android applications. In this talk, I will introduce a set of representative test input
generation tools and present a comparative study to highlight their strengths and limitations.
You will learn about the internals of these tools and how you can use them to test your application.
The details of the study along with a VM setup with the tools is available at:
[http://bear.cc.gatech.edu/\~shauvik/androtest/](http://bear.cc.gatech.edu/~shauvik/androtest/)

## Your Tests Aren't Flaky


[Alister Scott](https://developers.google.com/google-test-automation-conference/2015/speakers#AlisterScott) (Automattic)


Links:
[Video](https://www.youtube.com/watch?v=hmk1h40shaE&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=10),
[Slides](https://docs.google.com/presentation/d/1L9hGYqCAgjZyXE9ch4Toh4ziuYYkB2OiMCdFpgfTko0/pub)


Flaky tests are the bugbear of any automated test engineer; as someone (probably Alister) once said
"insanity is running the same tests over and over again and getting different results".
Flaky tests cause no end of despair, but perhaps there's no such thing as a flaky or non-flaky test,
perhaps we need to look at this problem through a different lens. We should spend more time building
more deterministic, more testable systems than spending time building resilient and persistent tests.
Alister will share some examples of when test flakiness hid real problems underneath the system,
and how it's possible to solve test flakiness by building better systems.

## Large-Scale Automated Visual Testing

[Adam Carmi](https://developers.google.com/google-test-automation-conference/2015/speakers#AdamCarmi) (Applitools)


Links:
[Video](https://www.youtube.com/watch?v=euJ2OrlBEqQ&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=11),
[Slides](https://docs.google.com/presentation/d/1TslcNggA6xx_GQbYu4QYU9vozdHhZ2pQE1Kj52ch1VQ/pub)


Automated visual testing is a major emerging trend in the dev / test community. In this talk you will
learn what visual testing is and why it should be automated. We will take a deep dive into some of
the technological challenges involved with visual test automation and show how modern tools address
them. We will demo cutting edge technologies that enable running cross browser and cross device
visual tests, and provide key tips for succeeding with large scale visual testing.

## Hands Off Regression Testing

[Karin Lundberg](https://developers.google.com/google-test-automation-conference/2015/speakers#KarinLundberg) (Twitter) and
[Puneet Khanduri](https://developers.google.com/google-test-automation-conference/2015/speakers#PuneetKhanduri) (Twitter)


Links:
[Video](https://www.youtube.com/watch?v=2zjhKmV0UFA&index=12&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1KITWbK46rm-lDZce_SqxegHTPjc9Ez-broxDTpfBpK8/pub)


Your team has just finished a major refactor of a service and all your unit and integration tests pass.
Nice work! But you're not done just yet. Now you need to make extra sure that you didn't break anything
and that there aren't any lurking bugs that you haven't caught yet. It's time to put Diffy to work.


Unlike tools that ensure that your code is sound, like unit or integration tests, Diffy compares
the behavior of your modified service by standing up instances of your new service and your old
service side by side, routing example requests to each, comparing the responses, and provides back
any regressions that have surfaced from those comparisons.


Also, we've just open sourced the tool and it is quickly becoming one of the most popular among
Twitter's open source projects.

## Automated Accessibility Testing for Android Applications

[Casey Burkhardt](https://developers.google.com/google-test-automation-conference/2015/speakers#CaseyBurkhardt) (Google)


Links:
[Video](https://www.youtube.com/watch?v=vvwc8MVlusY&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=13),
[Slides](https://docs.google.com/presentation/d/18ZUiXTZIGx0ydeLcKxnp5BxnI5kv7Cxm6rPaTf-c0ms/pub)


This talk will introduce the core accessibility affordances on the Android platform and illustrate
some common developer pitfalls related to accessibility. You'll learn about the new Android Accessibility
Test Framework and its integration into the Espresso and Robolectric testing frameworks. Finally,
you'll learn how easy it is to add automated accessibility checking to your your existing Android project tests.

## Statistical Data Sampling

[Celal Ziftci](https://developers.google.com/google-test-automation-conference/2015/speakers#CelalZiftci) (Google) and
[Ben Greenberg](https://developers.google.com/google-test-automation-conference/2015/speakers#BenjaminGreenberg) (MIT graduate student)


Links:
[Video](https://www.youtube.com/watch?v=cXi1Jo5V7UM&index=15&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1zAgKXFOQn02PVik9b4YkV0ZJ2wJIaGAf5oFY_dUyDD8/pub)


It is common practice to use a sample of production data in tests. Examples are:

- Sanity Test: Feed a sample of production data into your system to see if anything fails.
- A/B Test: Take a large chunk of production data, run it through the current and new versions of your system, and diff the outputs for inspection.


To get a sample of production data, teams typically use ad-hoc solutions, such as:

- Manually looking at the distribution of specific fields (e.g. numeric fields),
- Choosing a totally random sample


However, these approaches have a serious downside: They can miss out rare events (e.g. edge cases),
which increases the risk of uncaught bugs in production. To mitigate this risk, teams choose very
large samples. However, with such large samples, there are even more downsides:

- Rare events can still be missed,
- Runtime of tests greatly increases,
- Diffs are too large for a human being to comprehend, and there is a lot of repetition.


In this talk, we propose a novel statistical data sampling technique to "smartly" choose a "good"
sample from production data that:

- Guarantees rare events will not be missed,
- Minimizes the size of the chosen sample by eliminating duplicates.


Our technique catches rare/boundary cases, keeps the sample size to a minimum, and implicitly
decreases the manual burden of looking at test outputs/diffs on the developers. It also supports
parallel execution (e.g. MapReduce) so that vast amounts of data can be processed in a short time-frame
to choose the sample.

## Nest Automation Infrastructure

[Usman Abdullah](https://developers.google.com/google-test-automation-conference/2015/speakers#UsmanAbdullah) (Nest),
[Giulia Guidi](https://developers.google.com/google-test-automation-conference/2015/speakers#GiuliaGuidi) (Nest) and
[Sam Gordon](https://developers.google.com/google-test-automation-conference/2015/speakers#SamGordon) (Nest)


Links:
[Video](https://www.youtube.com/watch?v=lIoAq2Mjjas&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=16),
[Slides](https://docs.google.com/presentation/d/1x-l6xb9uUzFA9n0EFxcsvIEj6dQPQBcCu3JucAVe-8E/pub)


Nest's vision for the Thoughtful Home involves interconnected, intelligent devices working together
to make your home safer, more energy efficient, and more aware. This talk will focus on the automation
infrastructure and test tools that have been built to help make that vision a reality. Various teams
within Nest have been working on both cross-platform and device/feature specific systems for automated
regression test and analysis. Using specific examples from real world product testing, we will
cover cross-product hardware in the loop testing infrastructure and power regression analysis tools,
along with camera and motion detection specific tool sets.

## Event Generators

[Roussi Roussev](https://developers.google.com/google-test-automation-conference/2015/speakers#RoussiRossev) (Splunk)


Links:
[Video](https://www.youtube.com/watch?v=oDaQ8hvnPzM&index=17&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/10wJIlD2Jnt6CQiPKJdcgYtC6XPy5czHKs_jYEk1TpWo/pub)


This talk covers our experiences in developing and using software event generators at Splunk.
Inspired by particle physics where event generators have been indispensable in understanding the
physical world without running large experimental machines, log generators have improved the way
we test our numerous integrations with modern and legacy third-party software. The talk covers the
basic functionality and the challenges in generating realistic logs.

## Multithreaded Test Synthesis

[Murali Krishna Ramanathan](https://developers.google.com/google-test-automation-conference/2015/speakers#MuraliKrishmanRamanathan)
(Indian Institute of Science, Bangalore)


Links:
[Video](https://www.youtube.com/watch?v=FkcSPVpE5og&index=18&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://drive.google.com/file/d/0ByHjpj7XroZ5dGJKUHp3NHM0Ylk/view)


Subtle concurrency errors in multithreaded libraries that arise because of incorrect or inadequate
synchronization are often difficult to pinpoint precisely using only static techniques. On the other
hand, the effectiveness of dynamic detectors is critically dependent on multithreaded test suites
whose execution can be used to identify and trigger concurrency bugs including data races, deadlocks
and atomicity violations. Usually, such multithreaded tests need to invoke a specific combination of
methods with objects involved in the invocations being shared appropriately to expose a bug.
Without a priori knowledge of the bug, construction of such tests can be challenging.


In this talk, I will present a lightweight and scalable technique for synthesizing tests for detecting
thread-safety violations. Given a multi-threaded library and a sequential test suite, I will describe a
fully automated analysis that examines sequential execution traces, and produces as its output a concurrent
client program that drives shared objects via library method calls to states conducive for triggering a
concurrency bug. Experimental results on a variety of well-tested Java libraries demonstrate the
effectiveness of our approach in revealing many complex bugs.

## Enabling Streaming Experiments at Netflix

[Minal Mishra](https://developers.google.com/google-test-automation-conference/2015/speakers#MinalMishra) (Netflix)


Links:
[Video](https://www.youtube.com/watch?v=5aDqes-EUKg&index=19&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1K16NtKd-vqdjFmMVVOJusSmBTXJESTxNPXF4s36BjXI/pub)


69+ million user's streaming experience is of paramount importance to Netflix. In order to swiftly
improve this, we moved the adaptive streaming algorithms into the Javascript layer. This posed a
unique challenge of frequently releasing client javascript software which directly impacted consumer's
streaming experience. Borrowing the continuous delivery paradigm, which has been widely and successfully
adopted for service applications, we used it to retire risk over the lifecycle of a checkin and
deliver updates frequently. In this talk we will describe a key component of this paradigm to enable
software updates. We will dive into rollout procedure of the javascript client and tools to accurately
compare the health against the current version. We will also share the challenges faced with this process.

## Mock the Internet

[Yabin Kang](https://developers.google.com/google-test-automation-conference/2015/speakers#YabinKang) (LinkedIn)


Links:
[Video](https://www.youtube.com/watch?v=6gPNrujpmn0&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=20),
[Slides](https://docs.google.com/presentation/d/1CDtpZahRBunKXZzOasHOZtJFW8QcYO-qy3ho144BZXg/pub)


Mock the internet, talking about a new mocking system in Linkedin that helps to mock all the outbound
traffic for service level integration tests, will also talk a little bit about the overview of Linkedin
Mocking strategy. Share the knowledge and what we learnt with everyone.

## Effective Testing of a GPS Monitoring Station Receiver

[Andrew Knodt](https://developers.google.com/google-test-automation-conference/2015/speakers#AndrewKnodt) (Lockheed Martin)


Links:
[Video](https://www.youtube.com/watch?v=UyZ71_VmroI&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=21),
[Slides](https://docs.google.com/presentation/d/1zKnH9LrsPTtinqWzJg9GpZv5sXaMF8rd314irYvPtH4/pub)


The existing GPS monitoring stations used by the Air Force have become difficult to maintain and
an effort is underway to replace them with a GPU accelerated Software Defined Radio (SDR) approach.
An overview of the unique testing challenges of this specialized GPS receiver along with an examination
of several testing approaches will be presented. While focused on a GPS application, these
testing approaches could easily be applied to other production level SDR efforts.

## Automation on Wearable Devices

[Anurag Routroy](https://developers.google.com/google-test-automation-conference/2015/speakers#AnuragRoutroy) (Intel)


Links:
[Video](https://www.youtube.com/watch?v=1agsqcPXPHo&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=22),
[Slides](https://docs.google.com/presentation/d/1nOvdDY4k7grcPSo83GpsBJRj6z8Ty1rscd2A8VSG0W0/pub)


As wearable technology is on the rise in personal and business use, all the companies which have
a solid space in the Android market have switched their focus on this upcoming technology. Thus
creating their apps with wearable support, which also increases the effort to test their app on
the wearable devices. Hence automation on the wearables becomes important to reduce testing effort
and increase efficiency.

## Unified Infra and CI Integration Testing (Docker/Vagrant)

[Maxim Guenis](https://developers.google.com/google-test-automation-conference/2015/speakers#MaximGuenis) (Supersonic)


Links:
[Video](https://www.youtube.com/watch?v=KtDDGZOdrbA&index=23&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1fRZUWG4ER2Z2MVnVZ7UmAHeC_NqqqFwlhyZKm2iLWbQ/pub)


Developers struggle every day to have a working local development environment ready when developing,
debugging and going through the continues integration cycle.We can to solve that by integrating docker
and vagrant to be used with CI tool. This combination allows to control applications at stack level
on development machines, while able to use the same stack in integration tests.
In this talk we will discuss:

- Use of Docker in CI integration tests
- Control of stack instead of single docker or app.
- Version control of development and test environments, easily ddistributed with git and docker tools.
- Seamless support for running Dockers on Mac and windows.

## Eliminating Useless Test Bits


[Patrick Lam](https://developers.google.com/google-test-automation-conference/2015/speakers#PatrickLam) (University of Waterloo)


Links:
[Video](https://www.youtube.com/watch?v=dH7bXocxPR0&index=24&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1yoOf_fyeN-AtQQQsTemxrranKVBUc9i-VzF_o4OqsJQ/pub)


Specializing static analysis techniques for test suites has yielded interesting results. We've
previously learned that most tests are simple straight-line code, namely a sequence of setup statements
followed by a payload consisting of asserts. We show how static analysis can identify useless setup
statements, enabling developers to simplify and speed up their test cases.

## Coverage is Not Strongly Correlated with Test Suite Effectiveness


[Laura Inozemtseva](https://developers.google.com/google-test-automation-conference/2015/speakers#LauraInozemtseva) (University of Waterloo)


Links:
[Video](https://www.youtube.com/watch?v=sAfROROGujU&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=25),
[Slides](https://docs.google.com/presentation/d/1ttVaFhAfnbO0027Hd_Tz47P5xPHtLf4mLr12xFRMKig/pub)


The coverage of a test suite is often used as a proxy for its ability to detect faults.
However, previous studies that investigated the correlation between code coverage and test suite
effectiveness have failed to reach a consensus about the nature and strength of the relationship
between these test suite characteristics. Moreover, many of the studies were done with small or
synthetic programs, making it unclear whether their results generalize to larger programs, and some
of the studies did not account for the confounding influence of test suite size. We have extended
these studies by evaluating the relationship between test suite size, coverage, and effectiveness for
realistic Java programs; our study is the largest to date in the literature. We measured the statement
coverage, decision coverage, and modified condition coverage of these suites and used mutation testing
to evaluate their fault detection effectiveness. We found that there is a low to moderate correlation
between coverage and effectiveness when the number of test cases in the suite is controlled for.
In addition, we found that stronger forms of coverage do not provide greater insight into the effectiveness
the suite.

## Fake Backends with RpcReplay


[Matt Garrett](https://developers.google.com/google-test-automation-conference/2015/speakers#MattGarrett) (Google)


Links:
[Video](https://www.youtube.com/watch?v=wJPfsyRRmyw&index=26&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1),
[Slides](https://docs.google.com/presentation/d/1IF30CK1z8xzj8xZxJiE7LBQoyzK770yZuBzJR2pfHng/pub)


Keeping tests fast and stable is critically important. This is hard when servers depend on many
backends. Developers must choose between long and flaky tests, or writing and maintaining fake
implementations. Instead, tests can be run using recorded traffic from these backends.
This provides the best of both worlds, allowing developers to test quickly against real backends.

## ChromeOS Test Automation Lab


[Simran Basi](https://developers.google.com/google-test-automation-conference/2015/speakers#SimranBasi) (Google) and
[Chris Sosa](https://developers.google.com/google-test-automation-conference/2015/speakers#ChrisSosa) (Google)


Links:
[Video](https://www.youtube.com/watch?v=08CyrK2d1t0&list=PLSIUOFhnxEiCWGsN9t5A-XOhRbmz54IS1&index=27),
[Slides](https://docs.google.com/presentation/d/1ZbtToNrkpG6jLZBXbuUXVFzHilrr6_ZTFFAFdgpl4FQ/pub)


ChromeOS is currently shipping 60+ different Chromebook/boxes each running their own software. On
the field, customers are getting a fresh system every 6 weeks. This would not be possible without a
robust Continuous Integration System vetting check-ins from our 200+ developers. In this talk, we
describe the overall architecture with specific emphasis on our test automation lab. In addition,
we discuss Moblab (short for mobile (test) lab), our entire test automation infrastructure running
from one chromebox. This system is used by many of our partners so that they too can run tests the
way we do.