All Classes Namespaces Files Functions Variables Enumerator Groups Pages
Continuous Integration

This chapter describes approach for continuous performance testing with help of the Jagger framework

Continuous performance testing

Anti patterns of performance testing
Unfortunately performance testing quite often:
  • is provided directly before release when code changes are already locked. As results issues can be found too late
  • is provided not regularly. As result it is difficult to identify when performance issues were introduced
  • is provided outside of development process (e.g. by regression team or external vendor). As a result team doesn't have access to all details

To resolve mentioned issues, we are recommending to make performance testing part of the Continuous Integration process

Continuous Performance Testing
Continuous Performance Testing (CPT) helps to catch problems during development (not release) process. When performance tests are the part of CI we can split load testing to several types and execute them on the regular basis.
As example
Performance testing as a part of CI pipeline
In this example we separate tests to following types:
  • Performance smoke testing: Applied to an isolated service via an API, this test measures the response times of certain critical performance-sensitive operations. It is fast and cheap. It rises alerts when degradation is detected
  • Performance regression testing: These tests measure simulated performance characteristics of typical user scenarios, such as placing item in a shopping bag, placing a product order, etc. We are focused on business scenarios performance. We collect performance metrics of the users steps and API calls. Test rises alerts when degradation is detected
  • End-to-end (E2E) performance testing: Full user scenarios, including UI, backend, and all required external 3rd party services, configured in production-like environments with realistic data volumes
You can find more reading about CPT at our blog

Requirements to the framework to support CPT

Framework should:

  • store test results
  • build trends for measured parameters
  • compare test results to baseline session (golden result) or reference values
  • take automatic decisions and rise flags
  • support CI integration

Jagger framework as a tool for CPT

From the very start framework is designed to be a part of CI. It is small, has high performance and low resources consumption, as pure java it can be launched on any platform (on premises or in the cloud), it has small deployment time.

Framework is storing all measured results and is able to build results trends in the Web UI - deviation of the results from execution to execution

Trend plots

Framework is supporting decision making and alert rising

Decision per particular metrics

Framework can be part of the Jenkins CI pipelines

Jagger deployment plugin

Deploys Jagger via ssh to specified environment. Allows to configure and deploy Jagger components (Master/Coordinator, Kernels, Agents) to the dedicated environment without scripting. Plugin allows to launch component in the single node and distributed modes

Latest version of the deployment plugin you can find in the nexus

Plugin allows to:

  • setup Master/Coordinator node and set additional system properties if necessary
    Master node setup
  • optional: setup Kernel node(s) and set additional system properties if necessary
  • optional: setup Agent node(s) and set additional system properties if necessary
  • specify where test results will be stores
    Results database setup
  • define what test scenario will be executed and provide additional setup via properties
    Test project setup

Jagger JaaS plugin

Controls test execution on the performance test environment(s) via JaaS REST API. It allows to submit test projects to the running test environment with Jagger load generating components. As you know from the Running in the Stand by mode chapter load generating components can run on some environment in the stand by mode waiting for test projects to execute. This plugin can submit test projects for execution

Latest version of the deployment plugin you can find in the nexus

Possible workflow

  • you are configuring and starting performance test environment in your CI pipeline
  • you are configuring and starting Jagger load generating components on this environment
  • you submit one or multiple performance test projects to this environment with help of the Jenkins JaaS plugin without restarting load generating components

In this case Jagger framework is working as a test service and you can send new tests to this service via API

Plugin allows to:

  • send your test project to the test environment via API
  • start particular load scenario from your project
    Test project start via API