Blog

Demanding Government regulation? Highly frequent audits? Multiple and distributed development teams? Software validation nightmare, right? Not so much if you put some effort into automating it.

Some industries (like health, fintech,..) demand more validations in place than others. If you build a news portal and add the NBA game overview to the politics section it would be bad, but if you mix up meds for a patient it would be a disaster! Naturally, companies buying the products like these expect detailed proof of validation before having the software see the light of the day. In this use case we will imagine we are working for a client from the Health Industry and explain how we would automate the validation process.

Some of the concerns that drove us to automating validation process:

• Strict Government regulations
• Frequent and demanding audits
• Manual Validation is slow and error prone

Then, proof of each release should have:

• Only features implemented as designed
• All new implementations have been tested properly (manually and also covered with automation tests)
• Not have existing functionality broken in any way

Validation Requirements

• JIRA tickets and implementation descriptions for all features (changes) going into a release, plus proof that all regression tests have been run successfully
• Documented Validation Plan (supplied through a list of automated and/or manual tests executed)
• Documentation stating installation, configurations, and performance of the software has been tested for this release
• Documented summary of the results of the validation plan (i.e. whether tests have passed or not)
• Signature on the above (most likely through Engineering Manager and Release Manager)
• *Health Industry deeper level of Validation requirements: https://www.datacor.com/the-datacor-blog/fda-software-validation 

Hypothesis

The majority of the manual work for the Validation process can be automated and added into the CI/CD pipeline. Hereafter we will use the term Validation Process to refer to any series of scripts, code, etc. that are used to produce the Validation Documents.

For the purpose of this document, we define done as “A majority of the Validation work is handled automatically with minimum human involvement”.

This Validation Process will, in preparation for a release, be able to access appropriate systems (Project Management tool, SVC tool, Test Management tool, storage for validation artifacts) to build and store the validation documentation we need. The output will contain ‘Validation Plan’ and ‘Validation Summary Report’ in a PDF document (or separately) which will contain all the information but will have to be manually signed.

Collecting validation information

In the process of software validation we will need to be able to access 4 external systems: JIRA, Github, TestRail and AWS S3.

• SVC will be used to retrieve information about a release (milestone), which mainly is new features (changes) that are being added. For the sake of the context we will reference Github in this article. Make sure that every Pull Request has a Jira ticket referenced to it. Validation Software will fail in its purposes if this rule is not enforced.
• Project Management tool will be our source of truth for the feature description and the implementation requirements (as dictated by Product). In this article we will reference Jira. JIRA tickets will have been validated and referenced to the proof of validation (Test Case).
• Test Case Management Tool holds information of exact steps needed for feature validation, which will be used for the Validation Documents. (any Test Case Management tool is an option as long as we can easily fetch Test Cases from it, like Tuskr, Qase,… In this article we will reference TestRail)
• Storage is used to store validation artifacts (videos, screenshots, and plan/summary reports). Retention time for these artifacts should match business needs. In this article we will reference AWS S3.

Documentation of features correctly implemented

In order to comply with regulations a proof of verification and validation will come in the form of not just test steps but also screenshots and videos produced by test automation software we use (most if not all of the top ones support this either out-of-the-box or with third party integrations). Verification and validation will be done in a multi-environment in order to maintain high quality standards, shift left and support validation documentation.

Data in the form of videos and screenshots will be stored on S3. While validating a new feature, if it is not covered with automation, we can manually take video and/or screenshot as a proof of validation.

Ticket Validation Flow Example

Example of validating the ticket flow, as a part of the complete SDLC, is as follows:

• Create a new TestRail Test Case(s) for the User Story we are validating
• Reference TestRail Test Case to JIRA
• Make Test Cases Test Run associated with the JIRA ticket we are validating (both Automated and Manual Test Cases 
• Record the results of the Test Run inside of the JIRA ticket

It is also worth mentioning that proper naming is advised when creating Test Cases in

Test Rail (i.e. all of them should start with the Jira ticket ID)

Architecture Diagrams

Context Diagram

Container Diagram

Code Flow Diagram

Reporting

This work should have no impact on BI as it should not change the data stored. It will assist in creating Validation documents which is a requirement for Quality Assurance but that is not within the purview of BI.