Creating load test

• Main terms
• Java test project
• Load scenario
• Load scenario configuration
• Load test
• Test definition
• Test definition for HTTP configuration
• Test definition for Custom configuration
• Parametrization via properties

Main terms

Before discussing load tests development, let's agree on the following terms. These names will be used in the whole documentation

• System under test (SUT) - application (or group of applications) under test
• Endpoint - where we are applying load. In case of http protocol, it can be URI
• Endpoint provider - source of the test data (endpoints)
• Query - what parameters we are sending
• Query provider - source of the test data (queries)
• Distributor - component, responsible to combine endpoints and queries in different order (one by one, round robin, unique combinations, etc)
• Invoker - component, providing requests to SUT during load test. In case of http protocol - http client
• Validator - component that verifies responses from SUT - decides if response is valid or not. In case of http we can verify that return codes are always 200
• Metric - component that measures some parameter of the response (response time, throughput, success rate of requests, custom parameters)

Java test project

We consider that at this point, you have already installed the framework and learned how to execute the test

Now let's look inside the test project and review what is there. Created test project will have following components:

• Load scenario provider(s) - component where load scenarios are defined. Every provider is marked with annotation @Configuration. Every method returning load scenario is marked with annotation @Bean. Provider can contain multiple methods with @Bean annotation
• Test data provider(s) - endpoint and query providers. Sources of the test data
• Util - support component to work with test and environment properties

In short, test development will consist of creating test data sources and writing load scenarios

Load scenario

Load scenario describes the execution sequence of the load tests. While there are multiple scenarios in the test project, only single one can be executed at a time
Let's take a look what components does the load scenario consist of

• Load scenario. It describes the sequence of the parallel test groups execution. This is the entity that can be executed by the framework

• Parallel test group. It describes what load tests will be executed in parallel. Parallel test group can contain one or multiple load tests

• Load test. It describes:
  • What load profile will be used
  • When the load test should be terminated
    see more information in the section below

• Test definition. It describes
  • Where and how to read test data
  • What protocol to use for communication with the SUT
  • How to verify if responses from the SUT are valid
    see more information in the section below

Load scenario configuration

Create load scenario

// begin: following section is used for docu generation - Load test scenario configuration
@Configuration
public class SimpleJLoadScenarioProvider {

    @Bean
    public JLoadScenario exampleSimpleJaggerLoadScenario() {

        JTestDefinition jTestDefinition = JTestDefinition.builder(Id.of("td_example"), new EndpointsProvider()).build();

        JLoadProfile jLoadProfileRps = JLoadProfileRps.builder(RequestsPerSecond.of(10)).withMaxLoadThreads(10).withWarmUpTimeInMilliseconds(10000).build();
        
        JTerminationCriteria jTerminationCriteria = JTerminationCriteriaIterations.of(IterationsNumber.of(500), MaxDurationInSeconds.of(30));
        
        JLoadTest jLoadTest = JLoadTest.builder(Id.of("lt_example"), jTestDefinition, jLoadProfileRps, jTerminationCriteria).build();
        
        JParallelTestsGroup jParallelTestsGroup = JParallelTestsGroup.builder(Id.of("ptg_example"), jLoadTest).build();
        
        // To launch your load scenario, set 'jagger.load.scenario.id.to.execute' property's value equal to the load scenario id
        // You can do it via system properties or in the 'environment.properties' file
        return JLoadScenario.builder(Id.of("ls_example"), jParallelTestsGroup).build();
    }
}
// end: following section is used for docu generation - Load test scenario configuration

Load test

Below is the list of available load profiles and termination criteria:

• Load profiles
• Termination criteria

Test definition

Test definition components

Test definition components

On the image above you can see relation of the different components of the test definition. Test data, provided
by endpoint and query providers, is combined by the distributor (aka load balancer) to the set of the requests. During the load test
this set of the requests is passed to the invoker. Invoker is providing communication with the SUT and returning responses.
Responses are verified by validators to take pass/fail decision and processed by metrics to collect some measurements.
During the run time every virtual user is working with it's own invoker, thus allowing to generate load.

Below are the links to related information:

• Load balancers (Distributors)
• Validators and metrics collection

Test definition for HTTP configuration

Support of Http

Out of the box Jagger support http load tests. To make work with http easier, we have prepared set of classes to describe Test definition for HTTP

• DefaultHttpInvoker - Http invoker. Used by default for all test definitions
• JHttpEndpoint - Http endpoint.
• JHttpQuery - Http query.
• JHttpResponse - Http response. Represents response from the SUT
• JHttpResponseStatusValidatorProvider - Provider of the Http status validators

To create your first load test using http, you need to create an endpoint provider, like in the example below.
Query provider is optional

Examples of the endpoint provider for Test definition configuration

// begin: following section is used for docu generation - Endpoint provider
public class EndpointsProvider implements Iterable {
    private List<JHttpEndpoint> endpoints = new ArrayList<>();
    
    // Simple example of endpoint provider
    // Constructor will be triggered during spring bean creation at Jagger startup
    // Later distributor will invoke iterator method to get endpoints
    public EndpointsProvider() {
        // Put custom code here to get endpoints
        // In our case they will be hardcoded
        JHttpEndpoint httpEndpoint = new JHttpEndpoint(URI.create("https://jagger.griddynamics.net:443"));
        endpoints.add(httpEndpoint);
    }
    
    @Override
    public Iterator<JHttpEndpoint> iterator() {
        return endpoints.iterator();
    }
}
// end: following section is used for docu generation - Endpoint provider

Examples of the query provider for Test definition configuration

// begin: following section is used for docu generation - Query provider
public class QueriesProvider  implements Iterable {
    @Override
    public Iterator iterator() {
        List<JHttpQuery> queries = new ArrayList<>();
        queries.add(new JHttpQuery()
                .get()
                .path("index.html"));

        queries.add(new JHttpQuery()
                .get()
                .responseBodyType(String.class)
                .path("screenshots.html"));

        queries.add(new JHttpQuery()
                .get()
                .responseBodyType(String.class)
                .path("download.html"));
        
        return queries.iterator();
    }
}
// end: following section is used for docu generation - Query provider

Test definition for Custom configuration

Custom component to work with non http protocol

Based on the http example above, you can create performance test for any other protocol. You will need to:

• decide what objects will describe you endpoints, queries and responses. It can be any serializable object. You can use JHttpEndpoint, JHttpQuery, JHttpResponse as example
• create test data sources - providers of your endpoints and queries. Providers are implementing Iterable like in the examples above.
• create custom invoker. This component implements Invoker<Q,R,E> and can use some library to communicate with you SUT. E.g. jdbc to load database. You can use DefaultHttpInvoker as example
• if necessary, add custom validators and metrics

Jagger archetype contains example of the custom invoker provider CustomHttpInvokerProvider.java and its usage in the JLoadScenarioProvider.java

Parametrization via properties

To make your test more flexible and avoid recompilation we are recommending to parametrize your test. You can define parameters like load value or test duration in the test.properties file and later change them in the file or via system properties
Jagger archetype contains util class JaggerPropertiesProvider to work with properties. Your load scenario provider should just extend this class like in the example below

Example of the test properties file:

# exampleJaggerLoadScenario properties
example.jagger.test.definition.comment=no comments
example.jagger.load.scenario.termination.iterations=1000
example.jagger.load.scenario.termination.max.duration.seconds=20

Example of the test properties usage in the code:

// begin: following section is used for docu generation - Detailed load test scenario configuration

@Configuration
public class JLoadScenarioProvider extends JaggerPropertiesProvider {

    @Bean
    public JLoadScenario exampleJaggerLoadScenario() {

        // Example of using JaggerPropertiesProvider
        String testDefinitionComment = getTestPropertyValue("example.jagger.test.definition.comment");

        JTestDefinition jTestDefinition = JTestDefinition
                .builder(Id.of("exampleJaggerTestDefinition"), new EndpointsProvider())
                // optional
                .withComment(testDefinitionComment)
                .withInvoker(CustomHttpInvokerProvider.nonVerbose())
                .withQueryProvider(new QueriesProvider())
                .withLoadBalancer(JLoadBalancer.builder(ROUND_ROBIN).withRandomSeed(42).build())
                .addValidator(new ExampleResponseValidatorProvider("we are always good"))
                .addValidator(DefaultResponseValidatorProvider.of(NotNullResponseValidator.class))
                .addListener(new NotNullInvocationListener())
                .build();

        // Example of using JaggerPropertiesProvider
        Long iterationsNumber = Long.valueOf(getTestPropertyValue("example.jagger.load.scenario.termination.iterations"));
        Long maxDurationInSeconds = Long.valueOf(getTestPropertyValue("example.jagger.load.scenario.termination.max.duration.seconds"));
        JTerminationCriteria jTerminationCriteria = JTerminationCriteriaIterations
                .of(IterationsNumber.of(iterationsNumber), MaxDurationInSeconds.of(maxDurationInSeconds));

        JLoadProfile jLoadProfileRps = JLoadProfileRps
                .builder(RequestsPerSecond.of(10))
                .withMaxLoadThreads(10)
                .withWarmUpTimeInMilliseconds(10000)
                .build();

        // For standard metrics use JMetricName.
        // JLimitVsRefValue is used to compare the results with the referenced value.
        JLimit successRateLimit = JLimitVsRefValue.builder(JMetricName.PERF_SUCCESS_RATE_OK, RefValue.of(1D))
                // the threshold is relative.
                .withOnlyWarnings(LowWarnThresh.of(0.99), UpWarnThresh.of(1.01))
                .build();

        // For standard metrics use JMetricName.
        // JLimitVsBaseline is used to compare the results with the baseline.
        // Use 'chassis.engine.e1.reporting.session.comparison.baseline.session.id' to set baseline.
        JLimit throughputLimit = JLimitVsBaseline.builder(JMetricName.PERF_THROUGHPUT)
                // the threshold is relative.
                .withOnlyErrors(LowErrThresh.of(0.99), UpErrThresh.of(1.00001))
                .build();

        // For standard metrics use JMetricName.
        // JMetricName.PERF_LATENCY_PERCENTILE is used to set limits for latency percentile metrics.
        JLimit latencyPercentileLimit = JLimitVsRefValue.builder(JMetricName.PERF_LATENCY_PERCENTILE(95D), RefValue.of(0.1D))
                // the threshold is relative.
                .withOnlyWarnings(LowWarnThresh.of(0.50), UpWarnThresh.of(1.5))
                .build();


        JLoadTest jLoadTest = JLoadTest
                .builder(Id.of("exampleJaggerLoadTest"), jTestDefinition, jLoadProfileRps, jTerminationCriteria)
                .addListener(new CollectThreadsTestListener())
                .withLimits(successRateLimit, throughputLimit, latencyPercentileLimit)
                .build();

        JParallelTestsGroup jParallelTestsGroup = JParallelTestsGroup
                .builder(Id.of("exampleJaggerParallelTestsGroup"), jLoadTest)
                .addListener(new ExampleTestGroupListener())
                .build();

        // For JLoadScenario which is supposed to be executed by Jagger its ID must be set to 'jagger.load.scenario.id.to.execute' property's value
        return JLoadScenario.builder(Id.of("exampleJaggerLoadScenario"), jParallelTestsGroup)
                .addListener(new ExampleLoadScenarioListener())
                .withLatencyPercentiles(Arrays.asList(10D, 25.5D, 42D, 95D))
                .build();
    }
    // end: following section is used for docu generation - Detailed load test scenario configuration