The Continuous Integration Environment
The Continuous Integration Environment, like all other environments except the Development Environments which are entirely individual, is shared by all developers. There is no CI server in the Ocean system. Instead, we're using AWS CodeBuild for testing and artifact building.
The Test Agents
A Test Agent, as allocated by AWS CodeBuild, runs tests on a specific version of an operating system (such as Windows 7/8, Linux Ubuntu 16, Mac OS 10.6/7/8, etc). Test Agents are hosted on Amazon.
- Back-end tests are run only on Linux Ubuntu - the version of Linux used to deploy services on the production servers. Back-end tests do not involve browser testing.
- Front-end tests are run on all operating systems the clients support - that is, on as many of the operating systems the end users will be running, but the tests must also be run for each version of each browser supported by each version of those operating systems. Thus there is a matrix of OSs and browsers. Moreover, testing of front-end applications is done using real browsers, using Watir, Selenium, Xvfb and other browser testing frameworks.
AWS CodeBuild manages all test agent instances. It starts and stops them dynamically and runs tests in parallel. Payment is per minute for test agent instances, not per hour. Prices start at $0.005 per minute. The first 100 build minutes per month are free.
Exhaustive Testing of Front-end Clients
For web client applications, the Continuous Integration system tests all combinations of operating systems and supported browsers (including multiple release versions of each). All branches of the application flow are tested this way, which ensures that the UI layer works consistently for all end users. The parallel nature of AWS CodeDeploy testing reduces overall test times to a minimum.