Search “how to test microservices” and you’ll find Martin Fowlers article. This excellent article provides an in depth analysis of the risks and considerations required while testing such an architecture. The diagram below sourced from this article, describes potential types of testing to be considered.
Figure 1: Martin Fowler Testing Microservices
The article explores the architecture of a microservice, and how clear, well defined boundaries enable faster feedback in testing and allow for different testing risks to be explored.
This case study is different. It explores how we approached the testing of microservices from a qualitye engineering perspective. By considering a developer and quality engineer perspective, allows for early risk identification allowing for reliability and consistency of release.
Our client, a fintech bank, was an early adopter of microservices architecture and many of the strategies now in place for testing microservices where not then known. The business requirement was to develop a deposit banking product. The product approach was founded in LEAN principles. That is, develop an MVP (Minimal Viable Product) using that source consumer that drives further design.
In parallel to this process, the fintech was seeking a banking license from APRA. A secondary objective of the development of the deposit product was to demonstrate to APRA an agile approach to software development, one that accommodated customer feedback and allowed for an agile and responsive approach to customer demand.
The best practice approach to developing a product using microservices is to build from an existing monolith. Being a new product, we didn’t have an existing system. Knowing precisely where a system boundary should exist is difficult when the product functionality itself is uncertain. Boundaries were often refactored as the system design became more coherent.
In parallel to product development, was the transition of our infrastructure to cloud computing. End to End banking services require infrastructure that is secure and reliable. Significant investigation and exploration was put into the appropriate software that provided us the greatest reliability with the most security yet still allowing us to move with speed.
The consequence of this approach directly impacted our quality engineering strategy in two ways. Firstly, we didn’t yet have the luxury of ephemeral test environments. An unfortunate consequence of this was a limited number of test environments that enabled end to end testing.
Secondly, we realised that we needed a quality engineering strategy that reacted and responded to change. Our context was rapidly changing and our strategy had to be designed in a way enabled rapid response regardless of churn.
The client was experiencing a period of rapid growth with new teams and new products being created at a rapid rate. Rapid company growth meant the need to change processes and approaches. For instance, how we onboarded our people change to accommodate this growth.
In summary, we wanted a quality engineering strategy for a bespoke banking product, designed to be able to respond rapidly to consumer feedback. It was to be developed using a microservices architecture, the scope of which was yet to be determined. It was to be built on a devops platform that was being built and rolled out in parallel to product development.
What could possibly go wrong?
We were confident though. There was some excellent engineering practices in place that enabled us to build with confidence. All software developers used TDD (Test Driven development). This provided a ‘think testing first’ mindset and resulted in excellent software test automation coverage. This also meant that engineers were comfortable having the responsibility of software testing and with quality being a team responsibility.
A vital part of TDD process is refactoring, and so the concept of modifying and amending code was built into engineering culture, making it easier to rebuild and redefine the service boundaries when that need came along.
Pair Programming meant that peer reviews were performed as code was developed, providing immediate fast feedback on test design. Trunk Based Development. Developers were encouraged to frequently commit their code directly to the trunk on passing both manual and automated tests . Pushing code frequently means any conflicts are found earlier rather than later. Testing in Production The engineers made a conscious decision to deploy code continuously to production from iteration 1. The idea is to minimise risk through identifying problems as early as possible. By deploying software once stories were complete, enabled code to be tested in production for iteration one too. The nature of this testing was initially basic. For example, testing to see if a deployment could be successfully made, or testing the heartbeat on a new service. This concept of test early and often in production allowed us to identify early on potential risks and work to fix those.
The software engineers focused on extensive test automation. Their strategy was similar to that described by Martin Fowlers article on test automation. They performed:
Cross System Testing (Selenium Testing between Systems)
In addition to some excellent engineering practices the culture encouraged innovation and experimentation. You didn’t have to have all the answers, you needed a vision and a strategy, and you need to be able to explain the reasoning behind that approach. If it made sense, then there was little to block experimentation.
With such a test infected engineering culture, you may ask, was there a need for quality engineers? What could they possibly offer when such great testing was in place?
Quality Engineers provided a secondary, alternative approach to quality. While software engineers focused on predictability and reliability, quality engineers focused on identifying risk. It’s not easy to predict risks, especially given the large amount of uncertainty. Software testing was used as a tool to provide clarity and understanding. This mindset required a culture based in investigation and exploration. By encouraging a sceptical mindset focused firmly on business outcomes, the quality engineering team provided the space and opportunity to discuss the potential impacts these risks might have on the overall business goals.
This blend of approaches enabled us to design and develop at speed. By focusing on potential risks and exploring those, quality engineers were able to highlight early on areas which required attention. This strategy complemented the engineering efficiency approach that provides reliability and consistency of delivery.
Quality Engineers identified risk at multiple levels through the use of exploratory testing. Rather than limit their exploration to exploring within stories, they were encouraged to identify risks at technical and product levels. If a risk was identified, the quality engineers created a temporary team (known as a swarm) which allowed them to target and focus investigation on that particular risk.
A quality engineer in each team was able to focus on key integration risks, often mitigating risks through collaboration with other teams to identify early on potential changes that might break the overall system. This strategy of early risk identification enabled us to reduce the coverage of automated end-to-end testing. These tests are notoriously brittle, so keeping this test suite small was vital to our ability to deliver at speed.
One key area the quality engineering team focused on was testing the deployment process. A key benefit to microservices is the ability to deploy each service independently of the other. As the number of microservices grew, it became crucial that teams had access to test environments with minimal setup time. We also knew when the CD platform was completed a lot of these logistics would be solved. A temporary measure was to purchase production ready test environments . Post deployment, a snapshot was taken and a production-like test environment was supplied to each team. This allowed them to fully test the service with production like data.
One of our key testing strategies was to reduce risk through diversification. Rather than look at one approach we applied multiple smaller strategies. This enabled greater flexibility and reduced the impact of one strategy failing to deliver. Some areas we diversified our strategy was in the types of test environments we made available. We provided teams with production ready test environments that could be spun up to perform production like testing. Ideally, we would have liked to test in production, but separation of duty policies prevented us from doing so.
We diversified our test data. Some test data was created manually, others we used API’s to load test data. We also sanitised production data on a regular fortnightly basis ensuring the data we test against was fresh and relevant.
But most importantly we diversified our quality engineering skillset. Some quality engineers focused on supporting engineering efficiencies, and devops teams, while others worked closely with business teams, better understanding their processes and the risks associated with how we operated as a business. Knowing the business helped us identify business related risks as software was being developed.
Our software testing strategy was designed with some key requirements. The ability to respond rapidly to changes in scope and design. We focused on diversification to help mitigate risks. We minimised over engineering our solutions, focusing on small explorative steps that enabled us to move and respond rapidly and were inexpensive to adopt.
And while a testing strategy focused on decoupling, bounded context and fast feedback is critical to moving at speed, this is not our consumers perspective. The sum of the parts doesn’t necessarily make the whole. A good quality engineering strategy must consider both in order to deliver business value.
It wasn’t the prettiest solution, and there was little ‘best’ practise about it. But the solution enabled the fintech company to deploy a deposit product in six months and be awarded our APRA banking license. Importantly, we enabled the engineering team to move with speed, without any reduction in quality.
Want a microservices quality engineering quality that supports fast deployment? Contact us ! firstname.lastname@example.org
Testing Times, Level 5, 155 Clarence Street, Sydney NSW 2000