MultiSystemTestingAndWhatToDoAboutIt

From SPA Wiki

Jump to: navigation, search

Outputs from SPA2014 Session 'Multi-System Testing and What to Do About It'

We defined the area of multi-system testing (that is testing systems-of-systems) in the session and then identified problem areas that cause a lot of pain when you're trying to test systems which are really collections of other systems (also known as "front to back" testing or "end to end" testing).

The problem areas that the teams identified as most important (or perhaps most interesting!) and their solutions were:

  • Configuration - how do we configure large and complex test environments across systems and how do we prevent dangers to the production environment from accidental configuration?
    • Use obviously fake data, in case it ends up in the wrong place
    • Just use a single environment for everything (!)
    • Entirely script environments so that they can be created and destroyed cheaply (Docker, LXC, Glu, ...)
    • Keep configuration in a central place (e.g. Zookeeper)
    • Segregate environments physically (e.g. firewalls)
  • Test Data - where does it come from?
    • Have test data owned by development teams on a per domain type basis, that team is responsible for generating the test data set for their types (and everyone has to use it)
    • Use automatically generate data for automated tests, use small manual or production data sets for special cases with manual testing
    • Generate representative simulated data (perhaps machine learning based on production data sets?)
  • Test Diagnostics - how you you know if tests have passed, and if they fail how do you find out what has gone wrong?
    • feed all logs into a temporal database (e.g. Splunk) to allow a correlated view
    • Build strict interfaces that return very specific errors
    • correlate infrastructure and application logs to allow failures to be correlated
  • Inconsistent Formats, Dates, Units - how do you avoid catastrophic misunderstandings a la Adriane 5?
    • Use a shared data dictionary that defines this well
    • Use facades to external systems that convert to agreed representations
    • Define units carefully as part of interface definitions (no assumptions)
  • Interface Semantics - how do you make sure that you both mean the same thing?
    • use literate interface naming to make semantics and intent clear
    • choose test data sets that expose possible semantic ambiguities (e.g. dates that cross month ends to reveal month formatting conventions)
  • Simulators - how can you simulate external systems reliably and safely?
    • consider using event streams captured from production rather than generating event streams
    • parallel run simulators with production systems to compare their behaviour
    • include assertions in simulators to compare with expected behaviour and reveal violations of constraints
  • Volumes - how do you know you have tested with the right volume of workload?
    • take copies of production event streams to understand typical production loads


The slides from the session are here: Media:SPA2014-MultiSystemTesting.pdf

Group Work

The groups captured their outputs on flip charts for presentation during the session and these outputs can be found below:

MultiSystemTesting-Ex1GOM.jpg

MultiSystemTesting-Ex1GOM2.jpg

MultiSystemTesting-Ex1CBE.jpg

MultiSystemTesting-Ex1FG.jpg

MultiSystemTesting-Ex1STIA.jpg

MultiSystemTestingEx2Problems1.jpg

MultiSystemTestingEx2Problems2.jpg

MultiSystemTestingEx2Problems3.jpg

MultiSystemTestingEx2Problems4.jpg

MultiSystemTestingEx3Solutions1.jpg

MultiSystemTestingEx3Solutions2.jpg

MultiSystemTestingEx3Solutions3.jpg

MultiSystemTestingEx3Solutions4.jpg

MultiSystemTestingEx3Solutions5.jpg

MultiSystemTestingEx3Solutions6.jpg