Tempest - The OpenStack Integration Test Suite

The documentation for Tempest is officially hosted at: https://docs.openstack.org/tempest/latest/

This is a set of integration tests to be run against a live OpenStack cluster. Tempest has batteries of tests for OpenStack API validation, scenarios, and other specific tests useful in validating an OpenStack deployment.

Team and repository tags

.SS Design Principles

Tempest Design Principles that we strive to live by.

Quickstart

To run Tempest, you first need to create a configuration file that will tell Tempest where to find the various OpenStack services and other testing behavior switches. Where the configuration file lives and how you interact with it depends on how you'll be running Tempest. There are 2 methods of using Tempest. The first, which is a newer and recommended workflow treats Tempest as a system installed program. The second older method is to run Tempest assuming your working dir is the actually Tempest source repo, and there are a number of assumptions related to that. For this section we'll only cover the newer method as it is simpler, and quicker to work with.

$ git clone https://opendev.org/openstack/tempest
$ pip install tempest/

$ tempest init cloud-01

$ mkdir cloud-01 && cd cloud-01 && tempest init

$ tempest run

$ tempest run --workspace cloud-01

$ stestr run --exclude-regex '\[.*\bslow\b.*\]' '^tempest\.(api|scenario)'

$ tox -e full

$ tox -e smoke

Library

Tempest exposes a library interface. This interface is a stable interface and should be backwards compatible (including backwards compatibility with the old tempest-lib package, with the exception of the import). If you plan to directly consume Tempest in your project you should only import code from the Tempest library interface, other pieces of Tempest do not have the same stable interface and there are no guarantees on the Python API unless otherwise stated.

For more details refer to the library documentation

Release Versioning

Tempest Release Notes shows what changes have been released on each version.

Tempest's released versions are broken into 2 sets of information. Depending on how you intend to consume Tempest you might need

The version is a set of 3 numbers:

X.Y.Z

While this is almost semver like, the way versioning is handled is slightly different:

X is used to represent the supported OpenStack releases for Tempest tests in-tree, and to signify major feature changes to Tempest. It's a monotonically increasing integer where each version either indicates a new supported OpenStack release, the drop of support for an OpenStack release (which will coincide with the upstream stable branch going EOL), or a major feature lands (or is removed) from Tempest.

Y.Z is used to represent library interface changes. This is treated the same way as minor and patch versions from semver but only for the library interface. When Y is incremented we've added functionality to the library interface and when Z is incremented it's a bug fix release for the library. Also note that both Y and Z are reset to 0 at each increment of X.

Configuration

Detailed configuration of Tempest is beyond the scope of this document, see Tempest Configuration Documentation for more details on configuring Tempest. The etc/tempest.conf.sample attempts to be a self-documenting version of the configuration.

You can generate a new sample tempest.conf file, run the following command from the top level of the Tempest directory:

$ tox -e genconfig

The most important pieces that are needed are the user ids, OpenStack endpoints, and basic flavors and images needed to run tests.

Unit Tests

Tempest also has a set of unit tests which test the Tempest code itself. These tests can be run by specifying the test discovery path:

$ stestr --test-path ./tempest/tests run

By setting --test-path option to ./tempest/tests it specifies that test discover should only be run on the unit test directory. The default value of test_path is test_path=./tempest/test_discover which will only run test discover on the Tempest suite.

Alternatively, there are the py27 and py36 tox jobs which will run the unit tests with the corresponding version of python.

One common activity is to just run a single test, you can do this with tox simply by specifying to just run py27 or py36 tests against a single test:

$ tox -e py36 -- -n tempest.tests.test_microversions.TestMicroversionsTestsClass.test_config_version_none_23

Or all tests in the test_microversions.py file:

$ tox -e py36 -- -n tempest.tests.test_microversions

You may also use regular expressions to run any matching tests:

$ tox -e py36 -- test_microversions

Additionally, when running a single test, or test-file, the -n/--no-discover argument is no longer required, however it may perform faster if included.

For more information on these options and details about stestr, please see the stestr documentation.

Python 3.x

Starting during the Pike cycle Tempest has a gating CI job that runs Tempest with Python 3. Any Tempest release after 15.0.0 should fully support running under Python 3 as well as Python 2.7.

Legacy run method

The legacy method of running Tempest is to just treat the Tempest source code as a python unittest repository and run directly from the source repo. When running in this way you still start with a Tempest config file and the steps are basically the same except that it expects you know where the Tempest code lives on your system and requires a bit more manual interaction to get Tempest running. For example, when running Tempest this way things like a lock file directory do not get generated automatically and the burden is on the user to create and configure that.

To start you need to create a configuration file. The easiest way to create a configuration file is to generate a sample in the etc/ directory

$ cd $TEMPEST_ROOT_DIR
$ oslo-config-generator --config-file \


    tempest/cmd/config-generator.tempest.conf \


    --output-file etc/tempest.conf

After that, open up the etc/tempest.conf file and edit the configuration variables to match valid data in your environment. This includes your Keystone endpoint, a valid user and credentials, and reference data to be used in testing.

NOTE:

Tempest is not tied to any single test runner, but stestr is the most commonly used tool. Also, the nosetests test runner is not recommended to run Tempest.

After setting up your configuration file, you can execute the set of Tempest tests by using stestr. By default, stestr runs tests in parallel

$ stestr run

To run one single test serially

$ stestr run --serial tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON.test_reboot_non_existent_server

Tempest Field Guide Overview

Tempest is designed to be useful for a large number of different environments. This includes being useful for gating commits to OpenStack core projects, being used to validate OpenStack cloud implementations for both correctness, as well as a burn in tool for OpenStack clouds.

As such Tempest tests come in many flavors, each with their own rules and guidelines. Below is the overview of the Tempest repository structure to make this clear.

tempest/


   api/ - API tests


   scenario/ - complex scenario tests


   tests/ - unit tests for Tempest internals

Each of these directories contains different types of tests. What belongs in each directory, the rules and examples for good tests, are documented in a README.rst file in the directory.

Tempest Field Guide to API tests

API tests are validation tests for the OpenStack API. They should not use the existing Python clients for OpenStack, but should instead use the Tempest implementations of clients. Having raw clients let us pass invalid JSON to the APIs and see the results, something we could not get with the native clients.

When it makes sense, API testing should be moved closer to the projects themselves, possibly as functional tests in their unit test frameworks.

Tempest Field Guide to Scenario tests

Scenario tests are complex "through path" tests for OpenStack functionality. They are typically a series of steps where complicated state requiring multiple services is set up exercised, and torn down.

Scenario tests should not use the existing Python clients for OpenStack, but should instead use the Tempest implementations of clients.

Tempest Field Guide to Unit tests

Unit tests are the self checks for Tempest. They provide functional verification and regression checking for the internal components of Tempest. They should be used to just verify that the individual pieces of Tempest are working as expected.

Tempest Field Guide to API tests

What are these tests?

One of Tempest's prime function is to ensure that your OpenStack cloud works with the OpenStack API as documented. The current largest portion of Tempest code is devoted to test cases that do exactly this.

It's also important to test not only the expected positive path on APIs, but also to provide them with invalid data to ensure they fail in expected and documented ways. The latter type of tests is called negative tests in Tempest source code. Over the course of the OpenStack project Tempest has discovered many fundamental bugs by doing just this.

In order for some APIs to return meaningful results, there must be enough data in the system. This means these tests might start by spinning up a server, image, etc, then operating on it.

Why are these tests in Tempest?

This is one of the core missions for the Tempest project, and where it started. Many people use this bit of function in Tempest to ensure their clouds haven't broken the OpenStack API.

It could be argued that some of the negative testing could be done back in the projects themselves, and we might evolve there over time, but currently in the OpenStack gate this is a fundamentally important place to keep things.

Scope of these tests

API tests should always use the Tempest implementation of the OpenStack API, as we want to ensure that bugs aren't hidden by the official clients.

They should test specific API calls, and can build up complex state if it's needed for the API call to be meaningful.

They should send not only good data, but bad data at the API and look for error codes.

They should all be able to be run on their own, not depending on the state created by a previous test.

Tempest Field Guide to Scenario tests

What are these tests?

Scenario tests are "through path" tests of OpenStack function. Complicated setups where one part might depend on completion of a previous part. They ideally involve the integration between multiple OpenStack services to exercise the touch points between them.

Any scenario test should have a real-life use case. An example would be:

Why are these tests in Tempest?

This is one of Tempest's core purposes, testing the integration between projects.

Scope of these tests

Scenario tests should always use the Tempest implementation of the OpenStack API, as we want to ensure that bugs aren't hidden by the official clients.

Tests should be tagged with which services they exercise, as determined by which client libraries are used directly by the test.

Example of a good test

While we are looking for interaction of 2 or more services, be specific in your interactions. A giant "this is my data center" smoke test is hard to debug when it goes wrong.

A flow of interactions between Glance and Nova, like in the introduction, is a good example. Especially if it involves a repeated interaction when a resource is setup, modified, detached, and then reused later again.

Tempest Field Guide to Unit tests

What are these tests?

Unit tests are the self checks for Tempest. They provide functional verification and regression checking for the internal components of Tempest. They should be used to just verify that the individual pieces of Tempest are working as expected. They should not require an external service to be running and should be able to run solely from the Tempest tree.

Why are these tests in Tempest?

These tests exist to make sure that the mechanisms that we use inside of Tempest are valid and remain functional. They are only here for self validation of Tempest.

Scope of these tests

Unit tests should not require an external service to be running or any extra configuration to run. Any state that is required for a test should either be mocked out or created in a temporary test directory. (see test_wrappers.py for an example of using a temporary test directory)

Tempest Configuration Guide

Tempest Configuration Guide

This guide is a starting point for configuring Tempest. It aims to elaborate on and explain some of the mandatory and common configuration settings and how they are used in conjunction. The source of truth on each option is the sample config file which explains the purpose of each individual option. You can see the sample config file here: Sample Configuration File

Test Credentials

Tempest allows for configuring a set of admin credentials in the auth section, via the following parameters:

Admin credentials are not mandatory to run Tempest, but when provided they can be used to:

When Keystone uses a policy that requires domain scoped tokens for admin actions, the flag admin_domain_scope must be set to True. The admin user configured, if any, must have a role assigned to the domain to be usable.

Tempest allows for configuring pre-provisioned test credentials as well. This can be done using the accounts.yaml file (see Pre-Provisioned Credentials). This file is used to specify an arbitrary number of users available to run tests with. You can specify the location of the file in the auth section in the tempest.conf file. To see the specific format used in the file please refer to the accounts.yaml.sample file included in Tempest.

Keystone Connection Info

In order for Tempest to be able to talk to your OpenStack deployment you need to provide it with information about how it communicates with keystone. This involves configuring the following options in the identity section:

The auth_version option is used to tell Tempest whether it should be using Keystone's v2 or v3 api for communicating with Keystone. The two uri options are used to tell Tempest the url of the keystone endpoint. The uri option is used for Keystone v2 request and uri_v3 is used for Keystone v3. You want to ensure that which ever version you set for auth_version has its uri option defined.

Credential Provider Mechanisms

Tempest currently has two different internal methods for providing authentication to tests: dynamic credentials and pre-provisioned credentials. Depending on which one is in use the configuration of Tempest is slightly different.

Dynamic Credentials

Dynamic Credentials (formerly known as Tenant isolation) was originally created to enable running Tempest in parallel. For each test class it creates a unique set of user credentials to use for the tests in the class. It can create up to three sets of username, password, and project names for a primary user, an admin user, and an alternate user. To enable and use dynamic credentials you only need to configure two things:

This is also currently the default credential provider enabled by Tempest, due to its common use and ease of configuration.

It is worth pointing out that depending on your cloud configuration you might need to assign a role to each of the users created by Tempest's dynamic credentials. This can be set using the tempest_roles option. It takes in a list of role names each of which will be assigned to each of the users created by dynamic credentials. This option will not have any effect when Tempest is not configured to use dynamic credentials.

When the admin_domain_scope option is set to True, provisioned admin accounts will be assigned a role on domain configured in default_credentials_domain_name. This will make the accounts provisioned usable in a cloud where domain scoped tokens are required by Keystone for admin operations. Note that the initial pre-provision admin accounts, configured in tempest.conf, must have a role on the same domain as well, for Dynamic Credentials to work.

Pre-Provisioned Credentials

For a long time using dynamic credentials was the only method available if you wanted to enable parallel execution of Tempest tests. However, this was insufficient for certain use cases because of the admin credentials requirement to create the credential sets on demand. To get around that the accounts.yaml file was introduced and with that a new internal credential provider to enable using the list of credentials instead of creating them on demand. With pre-provisioned credentials (also known as locking test accounts) each test class will reserve a set of credentials from the accounts.yaml before executing any of its tests so that each class is isolated like with dynamic credentials.

To enable and use pre-provisioned credentials you need do a few things:

It is worth pointing out that each set of credentials in the accounts.yaml should have a unique project. This is required to provide proper isolation to the tests using the credentials, and failure to do this will likely cause unexpected failures in some tests. Also, ensure that these projects and users used do not have any pre-existing resources created. Tempest assumes all tenants it's using are empty and may sporadically fail if there are unexpected resources present.

When the Keystone in the target cloud requires domain scoped tokens to perform admin actions, all pre-provisioned admin users must have a role assigned on the domain where test accounts a provisioned. The option admin_domain_scope is used to tell Tempest that domain scoped tokens shall be used. default_credentials_domain_name is the domain where test accounts are expected to be provisioned if no domain is specified.

Note that if credentials are pre-provisioned via tempest account-generator the role on the domain will be assigned automatically for you, as long as admin_domain_scope as default_credentials_domain_name are configured properly in tempest.conf.

Pre-Provisioned Credentials are also known as accounts.yaml or accounts file.

Keystone Scopes & Roles Support in Tempest

For details on scope and roles support in Tempest, please refer to this document

Compute

Flavors

For Tempest to be able to create servers you need to specify flavors that it can use to boot the servers with. There are two options in the Tempest config for doing this:

Both of these options are in the compute section of the config file and take in the flavor id (not the name) from Nova. The flavor_ref option is what will be used for booting almost all of the guests; flavor_ref_alt is only used in tests where two different-sized servers are required (for example, a resize test).

Using a smaller flavor is generally recommended. When larger flavors are used, the extra time required to bring up servers will likely affect the total run time and probably require tweaking timeout values to ensure tests have ample time to finish.

Images

Just like with flavors, Tempest needs to know which images to use for booting servers. There are two options in the compute section just like with flavors:

Both options are expecting an image id (not name) from Nova. The image_ref option is what will be used for booting the majority of servers in Tempest. image_ref_alt is used for tests that require two images such as rebuild. If two images are not available you can set both options to the same image id and those tests will be skipped.

There are also options in the scenario section for images:

However, unlike the other image options, these are used for a very small subset of scenario tests which are uploading an image. These options are used to tell Tempest where an image file is located and describe its metadata for when it is uploaded.

You first need to specify full path of the image using img_file option. If it is found then the img_container_format and img_disk_format options are used to upload that image to glance. If it's not found, the tests requiring an image to upload will fail.

It is worth pointing out that using cirros is a very good choice for running Tempest. It's what is used for upstream testing, they boot quickly and have a small footprint.

Networking

OpenStack has a myriad of different networking configurations possible and depending on which of the two network backends, nova-network or Neutron, you are using things can vary drastically. Due to this complexity Tempest has to provide a certain level of flexibility in its configuration to ensure it will work against any cloud. This ends up causing a large number of permutations in Tempest's config around network configuration.

Enabling Remote Access to Created Servers

Network Creation/Usage for Servers

When Tempest creates servers for testing, some tests require being able to connect those servers. Depending on the configuration of the cloud, the methods for doing this can be different. In certain configurations, it is required to specify a single network with server create calls. Accordingly, Tempest provides a few different methods for providing this information in configuration to try and ensure that regardless of the cloud's configuration it'll still be able to run. This section covers the different methods of configuring Tempest to provide a network when creating servers.

Fixed Network Name

This is the simplest method of specifying how networks should be used. You can just specify a single network name/label to use for all server creations. The limitation with this is that all projects and users must be able to see that network name/label if they are to perform a network list and be able to use it.

If no network name is assigned in the config file and none of the below alternatives are used, then Tempest will not specify a network on server creations, which depending on the cloud configuration might prevent them from booting.

To set a fixed network name simply:

In the case that the configured fixed network name can not be found by a user network list call, it will be treated like one was not provided except that a warning will be logged stating that it couldn't be found.

Accounts File

If you are using an accounts file to provide credentials for running Tempest then you can leverage it to also specify which network should be used with server creations on a per project and user pair basis. This provides the necessary flexibility to work with more intricate networking configurations by enabling the user to specify exactly which network to use for which projects. You can refer to the accounts.yaml.sample file included in the Tempest repo for the syntax around specifying networks in the file.

However, specifying a network is not required when using an accounts file. If one is not specified you can use a fixed network name to specify the network to use when creating servers just as without an accounts file. However, any network specified in the accounts file will take precedence over the fixed network name provided. If no network is provided in the accounts file and a fixed network name is not set then no network will be included in create server requests.

If a fixed network is provided and the accounts.yaml file also contains networks this has the benefit of enabling a couple more tests which require a static network to perform operations like server lists with a network filter. If a fixed network name is not provided these tests are skipped. Additionally, if a fixed network name is provided it will serve as a fallback in case of a misconfiguration or a missing network in the accounts file.

With Dynamic Credentials

With dynamic credentials enabled and using nova-network, your only option for configuration is to either set a fixed network name or not. However, in most cases, it shouldn't matter because nova-network should have no problem booting a server with multiple networks. If this is not the case for your cloud then using an accounts file is recommended because it provides the necessary flexibility to describe your configuration. Dynamic credentials are not able to dynamically allocate things as necessary if Neutron is not enabled.

With Neutron and dynamic credentials enabled there should not be any additional configuration necessary to enable Tempest to create servers with working networking, assuming you have properly configured the network section to work for your cloud. Tempest will dynamically create the Neutron resources necessary to enable using servers with that network. Also, just as with the accounts file, if you specify a fixed network name while using Neutron and dynamic credentials it will enable running tests which require a static network and it will additionally be used as a fallback for server creation. However, unlike accounts.yaml this should never be triggered.

However, there is an option create_isolated_networks to disable dynamic credentials's automatic provisioning of network resources. If this option is set to False you will have to either rely on there only being a single/default network available for the server creation, or use fixed_network_name to inform Tempest which network to use.

SSH Connection Configuration

There are also several different ways to actually establish a connection and authenticate/login on the server. After a server is booted with a provided network there are still details needed to know how to actually connect to the server. The validation group gathers all the options regarding connecting to and remotely accessing the created servers.

To enable remote access to servers, there are 3 options at a minimum that are used:

The run_validation is used to enable or disable ssh connectivity for all tests (with the exception of scenario tests which do not have a flag for enabling or disabling ssh) To enable ssh connectivity this needs be set to True.

The connect_method option is used to tell Tempest what kind of IP to use for establishing a connection to the server. Two methods are available: fixed and floating, the later being set by default. If this is set to floating Tempest will create a floating ip for the server before attempted to connect to it. The IP for the floating ip is what is used for the connection.

For the auth_method option there is currently, only one valid option, keypair. With this set to keypair Tempest will create an ssh keypair and use that for authenticating against the created server.

Configuring Available Services

OpenStack is really a constellation of several different projects which are running together to create a cloud. However which projects you're running is not set in stone, and which services are running is up to the deployer. Tempest, however, needs to know which services are available so it can figure out which tests it is able to run and certain setup steps which differ based on the available services.

The service_available section of the config file is used to set which services are available. It contains a boolean option for each service (except for Keystone which is a hard requirement) set it to True if the service is available or False if it is not.

Service Catalog

Each project which has its own REST API contains an entry in the service catalog. Like most things in OpenStack this is also completely configurable. However, for Tempest to be able to figure out which endpoints should get REST API calls for each service, it needs to know how that project is defined in the service catalog. There are three options for each service section to accomplish this:

Setting catalog_type and endpoint_type should normally give Tempest enough information to determine which endpoint it should pull from the service catalog to use for talking to that particular service. However, if your cloud has multiple regions available and you need to specify a particular one to use a service you can set the region option in that service's section.

It should also be noted that the default values for these options are set to what DevStack uses (which is a de facto standard for service catalog entries). So often nothing actually needs to be set on these options to enable communication to a particular service. It is only if you are either not using the same catalog_type as DevStack or you want Tempest to talk to a different endpoint type instead of publicURL for a service that these need to be changed.

NOTE:

Service Feature Configuration

OpenStack provides its deployers a myriad of different configuration options to enable anyone deploying it to create a cloud tailor-made for any individual use case. It provides options for several different backend types, databases, message queues, etc. However, the downside to this configurability is that certain operations and features aren't supported depending on the configuration. These features may or may not be discoverable from the API so the burden is often on the user to figure out what is supported by the cloud they're talking to. Besides the obvious interoperability issues with this, it also leaves Tempest in an interesting situation trying to figure out which tests are expected to work. However, Tempest tests do not rely on dynamic API discovery for a feature (assuming one exists). Instead, Tempest has to be explicitly configured as to which optional features are enabled. This is in order to prevent bugs in the discovery mechanisms from masking failures.

The service feature-enabled config sections are how Tempest addresses the optional feature question. Each service that has tests for optional features contains one of these sections. The only options in it are boolean options with the name of a feature which is used. If it is set to false any test which depends on that functionality will be skipped. For a complete list of all these options refer to the sample config file.

API Extensions

The service feature-enabled sections often contain an api-extensions option (or in the case of Swift a discoverable_apis option). This is used to tell Tempest which API extensions (or configurable middleware) is used in your deployment. It has two valid config states: either it contains a single value all (which is the default) which means that every API extension is assumed to be enabled, or it is set to a list of each individual extension that is enabled for that service.

Sample Configuration File

The following is a sample Tempest configuration for adaptation and use. It is auto-generated from Tempest when this documentation is built, so if you are having issues with an option, please compare your version of Tempest with the version of this documentation.

The sample configuration can also be viewed in file form.

Command Documentation

Tempest Test-Account Generator Utility

Utility for creating accounts.yaml file for concurrent test runs. Creates one primary user, one alt user, one swift admin, one stack owner and one admin (optionally) for each concurrent thread. The utility creates user for each tenant. The accounts.yaml file will be valid and contain credentials for created users, so each user will be in separate tenant and have the username, tenant_name, password and roles.

Usage: tempest account-generator [-h] [OPTIONS] accounts_file.yaml

Positional Arguments

accounts_file.yaml (Required) Provide an output accounts yaml file. Utility creates a .yaml file in the directory where the command is ran. The appropriate name for the file is accounts.yaml and it should be placed in tempest/etc directory.

Authentication

Account generator creates users and tenants so it needs the admin credentials of your cloud to operate properly. The corresponding info can be given either through CLI options or environment variables.

You're probably familiar with these, but just to remind:

Optional Arguments

To see help on specific argument, please do: tempest account-generator [OPTIONS] <accounts_file.yaml> -h.

Post Tempest Run Cleanup Utility

Utility for cleaning up environment after Tempest test run

Usage: tempest cleanup [--help] [OPTIONS]

If run with no arguments, tempest cleanup will query your OpenStack deployment and build a list of resources to delete and destroy them. This list will exclude the resources from saved_state.json and will include the configured admin account if the --delete-tempest-conf-objects flag is specified. By default the admin project is not deleted and the admin user specified in tempest.conf is never deleted.

Example Run

WARNING:

$ tempest cleanup --init-saved-state
$ # Actual running of Tempest tests
$ tempest cleanup

Runtime Arguments

[1]

The _projects_to_clean dictionary in dry_run.json lists the projects that tempest cleanup will loop through to delete child objects, but the command will, by default, not delete the projects themselves. This may differ from the projects list as you can clean the Tempest and alternate Tempest users and projects but they will not be deleted unless the --delete-tempest-conf-objects flag is used to force their deletion.

NOTE:

Subunit Describe Calls Utility

subunit-describe-calls is a parser for subunit streams to determine what REST API calls are made inside of a test and in what order they are called.

Runtime Arguments

Usage

subunit-describe-calls will take in either stdin subunit v1 or v2 stream or a file path which contains either a subunit v1 or v2 stream passed via the --subunit parameter. This is then parsed checking for details contained in the file_bytes of the --non-subunit-name parameter (the default is pythonlogging which is what Tempest uses to store logs). By default the OpenStack default ports are used unless a file is provided via the --ports option. The resulting output is dumped in JSON output to the path provided in the --output-file option.

Ports file JSON structure

{


    "<port number>": "<name of service>",


    ...
}

Output file JSON structure

{


    "full_test_name[with_id_and_tags]": [


        {


            "name": "The ClassName.MethodName that made the call",


            "verb": "HTTP Verb",


            "service": "Name of the service",


            "url": "A shortened version of the URL called",


            "status_code": "The status code of the response",


            "request_headers": "The headers of the request",


            "request_body": "The body of the request",


            "response_headers": "The headers of the response",


            "response_body": "The body of the response"


        }


    ]
}

Tempest Workspace

Manages Tempest workspaces

This command is used for managing tempest workspaces

Commands

list

Outputs the name and path of all known tempest workspaces

register

Registers a new tempest workspace via a given --name and --path

rename

Renames a tempest workspace from --old-name to --new-name

move

Changes the path of a given tempest workspace --name to --path

remove

Deletes the entry for a given tempest workspace --name

--rmdir Deletes the given tempest workspace directory

General Options

Tempest Run

Runs tempest tests

This command is used for running the tempest tests

Test Selection

Tempest run has several options:

There are also the --exclude-list and --include-list options that let you pass a filepath to tempest run with the file format being a line separated regex, with '#' used to signify the start of a comment on a line. For example:

# Regex file
^regex1 # Match these tests
.*regex2 # Match those tests

These arguments are just passed into stestr, you can refer to the stestr selection docs for more details on how these operate: http://stestr.readthedocs.io/en/latest/MANUAL.html#test-selection

You can also use the --list-tests option in conjunction with selection arguments to list which tests will be run.

You can also use the --load-list option that lets you pass a filepath to tempest run with the file format being in a non-regex format, similar to the tests generated by the --list-tests option. You can specify target tests by removing unnecessary tests from a list file which is generated from --list-tests option.

You can also use --worker-file option that let you pass a filepath to a worker yaml file, allowing you to manually schedule the tests run. For example, you can setup a tempest run with different concurrences to be used with different regexps. An example of worker file is showed below:

# YAML Worker file
- worker:


  # you can have more than one regex per worker


  - tempest.api.*


  - neutron_tempest_tests
- worker:


  - tempest.scenario.*

This will run test matching with 'tempest.api.*' and 'neutron_tempest_tests' against worker 1. Run tests matching with 'tempest.scenario.*' under worker 2.

You can mix manual scheduling with the standard scheduling mechanisms by concurrency field on a worker. For example:

# YAML Worker file
- worker:


  # you can have more than one regex per worker


  - tempest.api.*


  - neutron_tempest_tests


  concurrency: 3
- worker:


  - tempest.scenario.*


  concurrency: 2

This will run tests matching with 'tempest.scenario.*' against 2 workers.

This worker file is passed into stestr. For some more details on how it operates please refer to the stestr scheduling docs: https://stestr.readthedocs.io/en/stable/MANUAL.html#test-scheduling

Test Execution

There are several options to control how the tests are executed. By default tempest will run in parallel with a worker for each CPU present on the machine. If you want to adjust the number of workers use the --concurrency option and if you want to run tests serially use --serial/-t

Running with Workspaces

Tempest run enables you to run your tempest tests from any setup tempest workspace it relies on you having setup a tempest workspace with either the tempest init or tempest workspace commands. Then using the --workspace CLI option you can specify which one of your workspaces you want to run tempest from. Using this option you don't have to run Tempest directly with you current working directory being the workspace, Tempest will take care of managing everything to be executed from there.

Running from Anywhere

Tempest run provides you with an option to execute tempest from anywhere on your system. You are required to provide a config file in this case with the --config-file option. When run tempest will create a .stestr directory and a .stestr.conf file in your current working directory. This way you can use stestr commands directly to inspect the state of the previous run.

Test Output

By default tempest run's output to STDOUT will be generated using the subunit-trace output filter. But, if you would prefer a subunit v2 stream be output to STDOUT use the --subunit flag

Combining Runs

There are certain situations in which you want to split a single run of tempest across 2 executions of tempest run. (for example to run part of the tests serially and others in parallel) To accomplish this but still treat the results as a single run you can leverage the --combine option which will append the current run's results with the previous runs.

Supported OpenStack Releases and Python Versions

Supported OpenStack Releases and Python Versions

This document lists the officially supported OpenStack releases and python versions by Tempest.

Compatible OpenStack Releases

Tempest master supports the below OpenStack Releases:

For older OpenStack Release:

For any older OpenStack Release than the listed above, Tempest master might work. But if Tempest master starts failing then, you can use the respective Tempest tag listed in OpenStack release page.

For example: OpenStack Stein: Tempest 20.0.0

How to use Tempest tag on Extended Maintenance stable branch:

Supported Python Versions

Tempest master supports the below python versions:

Description of Tests

Description of Tests

OpenStack Services Integration Tests

OpenStack Services API Tests

So You Want to Contribute...

For general information on contributing to OpenStack, please check out the contributor guide to get started. It covers all the basics that are common to all OpenStack projects: the accounts you need, the basics of interacting with our Gerrit review system, how we communicate as a community, etc.

Below will cover the more project specific information you need to get started with Tempest.

Communication

Contacting the Core Team

Please refer to the Tempest Core Team contacts.

New Feature Planning

If you want to propose a new feature please read Feature Proposal Process Tempest features are tracked on Launchpad BP.

Task Tracking

We track our tasks in Launchpad.

If you're looking for some smaller, easier work item to pick up and get started on, search for the 'low-hanging-fruit' tag.

Reporting a Bug

You found an issue and want to make sure we are aware of it? You can do so on Launchpad. More info about Launchpad usage can be found on OpenStack docs page

Getting Your Patch Merged

All changes proposed to the Tempest require single Code-Review +2 votes from Tempest core reviewers by giving Workflow +1 vote. More detailed guidelines for reviewers are available at Reviewing Tempest Code.

Project Team Lead Duties

All common PTL duties are enumerated in the PTL guide.

The Release Process for QA is documented in QA Release Process.

Development

Tempest Coding Guide

Tempest Specific Commandments

It is recommended to use tox -eautopep8 before submitting a patch.

Test Data/Configuration

Supported OpenStack Components

Tempest's Tempest Library Documentation and plugin interface can be leveraged to support integration testing for virtually any OpenStack component.

However, Tempest only offers in-tree integration testing coverage for the following components:

Historically, Tempest offered in-tree testing for other components as well, but since the introduction of the External Plugin Interface, Tempest's in-tree testing scope has been limited to the projects above. Integration tests for projects not included above should go into one of the relevant plugin projects.

Exception Handling

According to the The Zen of Python the Errors should never pass silently. Tempest usually runs in special environment (jenkins gate jobs), in every error or failure situation we should provide as much error related information as possible, because we usually do not have the chance to investigate the situation after the issue happened.

In every test case the abnormal situations must be very verbosely explained, by the exception and the log.

In most cases the very first issue is the most important information.

Try to avoid using try blocks in the test cases, as both the except and finally blocks could replace the original exception, when the additional operations leads to another exception.

Just letting an exception to propagate, is not a bad idea in a test case, at all.

Try to avoid using any exception handling construct which can hide the errors origin.

If you really need to use a try block, please ensure the original exception at least logged. When the exception is logged you usually need to raise the same or a different exception anyway.

Use of self.addCleanup is often a good way to avoid having to catch exceptions and still ensure resources are correctly cleaned up if the test fails part way through.

Use the self.assert* methods provided by the unit test framework. This signals the failures early on.

Avoid using the self.fail alone, its stack trace will signal the self.fail line as the origin of the error.

Avoid constructing complex boolean expressions for assertion. The self.assertTrue or self.assertFalse without a msg argument, will just tell you the single boolean value, and you will not know anything about the values used in the formula, the msg argument might be good enough for providing more information.

Most other assert method can include more information by default. For example self.assertIn can include the whole set.

It is recommended to use testtools matcher for the more tricky assertions. You can implement your own specific matcher as well.

If the test case fails you can see the related logs and the information carried by the exception (exception class, backtrack and exception info). This and the service logs are your only guide to finding the root cause of flaky issues.

Test cases are independent

Every test_method must be callable individually and MUST NOT depends on, any other test_method or test_method ordering.

Test cases MAY depend on commonly initialized resources/facilities, like credentials management, testresources and so on. These facilities, MUST be able to work even if just one test_method is selected for execution.

Service Tagging

Service tagging is used to specify which services are exercised by a particular test method. You specify the services with the tempest.common.utils.services decorator. For example:

@utils.services('compute', 'image')

Valid service tag names are the same as the list of directories in tempest.api that have tests.

For scenario tests having a service tag is required. For the API tests service tags are only needed if the test method makes an API call (either directly or indirectly through another service) that differs from the parent directory name. For example, any test that make an API call to a service other than Nova in tempest.api.compute would require a service tag for those services, however they do not need to be tagged as compute.

Test Attributes

Tempest leverages test attributes which are a simple but effective way of distinguishing between different "types" of API tests. A test can be "tagged" with such attributes using the decorators.attr decorator, for example:

@decorators.attr(type=['negative'])
def test_aggregate_create_aggregate_name_length_less_than_1(self):


    [...]

These test attributes can be used for test selection via regular expressions. For example, (?!.*\[.*\bslow\b.*\])(^tempest\.scenario) runs all the tests in the scenario test module, except for those tagged with the slow attribute (via a negative lookahead in the regular expression). These attributes are used in Tempest's tox.ini as well as Tempest's Zuul job definitions for specifying particular batches of Tempest test suites to run.

Negative Attribute

The type='negative' attribute is used to signify that a test is a negative test, which is a test that handles invalid input gracefully. This attribute should be applied to all negative test scenarios.

This attribute must be applied to each test that belongs to a negative test class, i.e. a test class name ending with "Negative.*" substring.

Slow Attribute

The type='slow' attribute is used to signify that a test takes a long time to run, relatively speaking. This attribute is usually applied to scenario tests, which involve a complicated series of API operations, the total runtime of which can be relatively long. This long runtime has performance implications on Zuul jobs, which is why the slow attribute is leveraged to run slow tests on a selective basis, to keep total Zuul job runtime down to a reasonable time frame.

Smoke Attribute

The type='smoke' attribute is used to signify that a test is a so-called smoke test, which is a type of test that tests the most vital OpenStack functionality, like listing servers or flavors or creating volumes. The attribute should be sparingly applied to only the tests that sanity-check the most essential functionality of an OpenStack cloud.

Test fixtures and resources

Test level resources should be cleaned-up after the test execution. Clean-up is best scheduled using addCleanup which ensures that the resource cleanup code is always invoked, and in reverse order with respect to the creation order.

Test class level resources should be defined in the resource_setup method of the test class, except for any credential obtained from the credentials provider, which should be set-up in the setup_credentials method. Cleanup is best scheduled using addClassResourceCleanup which ensures that the cleanup code is always invoked, and in reverse order with respect to the creation order.

In both cases - test level and class level cleanups - a wait loop should be scheduled before the actual delete of resources with an asynchronous delete.

The test base class BaseTestCase defines Tempest framework for class level fixtures. setUpClass and tearDownClass are defined here and cannot be overwritten by subclasses (enforced via hacking rule T105).

Set-up is split in a series of steps (setup stages), which can be overwritten by test classes. Set-up stages are:

Tear-down is also split in a series of steps (teardown stages), which are stacked for execution only if the corresponding setup stage had been reached during the setup phase. Tear-down stages are:

Skipping Tests

Skipping tests should be based on configuration only. If that is not possible, it is likely that either a configuration flag is missing, or the test should fail rather than be skipped. Using discovery for skipping tests is generally discouraged.

When running a test that requires a certain "feature" in the target cloud, if that feature is missing we should fail, because either the test configuration is invalid, or the cloud is broken and the expected "feature" is not there even if the cloud was configured with it.

Negative Tests

Error handling is an important aspect of API design and usage. Negative tests are a way to ensure that an application can gracefully handle invalid or unexpected input. However, as a black box integration test suite, Tempest is not suitable for handling all negative test cases, as the wide variety and complexity of negative tests can lead to long test runs and knowledge of internal implementation details. The bulk of negative testing should be handled with project function tests. All negative tests should be based on API-WG guideline . Such negative tests can block any changes from accurate failure code to invalid one.

If facing some gray area which is not clarified on the above guideline, propose a new guideline to the API-WG. With a proposal to the API-WG we will be able to build a consensus across all OpenStack projects and improve the quality and consistency of all the APIs.

In addition, we have some guidelines for additional negative tests.

The above guidelines don't cover all cases and we will grow these guidelines organically over time. Patches outside of the above guidelines are left up to the reviewers' discretion and if we face some conflicts between reviewers, we will expand the guideline based on our discussion and experience.

Test skips because of Known Bugs

If a test is broken because of a bug it is appropriate to skip the test until bug has been fixed. You should use the skip_because decorator so that Tempest's skip tracking tool can watch the bug status.

Example:

@skip_because(bug="980688")
def test_this_and_that(self):


  ...

Guidelines

Parallel Test Execution

Tempest by default runs its tests in parallel this creates the possibility for interesting interactions between tests which can cause unexpected failures. Dynamic credentials provides protection from most of the potential race conditions between tests outside the same class. But there are still a few of things to watch out for to try to avoid issues when running your tests in parallel.

Sample Configuration File

The sample config file is autogenerated using a script. If any changes are made to the config variables in tempest/config.py then the sample config file must be regenerated. This can be done running:

tox -e genconfig

Unit Tests

Unit tests are a separate class of tests in Tempest. They verify Tempest itself, and thus have a different set of guidelines around them:

Test Documentation

For tests being added we need to require inline documentation in the form of docstrings to explain what is being tested. In API tests for a new API a class level docstring should be added to an API reference doc. If one doesn't exist a TODO comment should be put indicating that the reference needs to be added. For individual API test cases a method level docstring should be used to explain the functionality being tested if the test name isn't descriptive enough. For example:

def test_get_role_by_id(self):


    """Get a role by its id."""

the docstring there is superfluous and shouldn't be added. but for a method like:

def test_volume_backup_create_get_detailed_list_restore_delete(self):


    pass

a docstring would be useful because while the test title is fairly descriptive the operations being performed are complex enough that a bit more explanation will help people figure out the intent of the test.

For scenario tests a class level docstring describing the steps in the scenario is required. If there is more than one test case in the class individual docstrings for the workflow in each test methods can be used instead. A good example of this would be:

class TestServerBasicOps(manager.ScenarioTest):


    """The test suite for server basic operations


    This smoke test case follows this basic set of operations:


     * Create a keypair for use in launching an instance


     * Create a security group to control network access in instance


     * Add simple permissive rules to the security group


     * Launch an instance


     * Perform ssh to instance


     * Verify metadata service


     * Verify metadata on config_drive


     * Terminate the instance


    """

Test Identification with Idempotent ID

Every function that provides a test must have an idempotent_id decorator that is a unique uuid-4 instance. This ID is used to complement the fully qualified test name and track test functionality through refactoring. The format of the metadata looks like:

@decorators.idempotent_id('585e934c-448e-43c4-acbf-d06a9b899997')
def test_list_servers_with_detail(self):


    # The created server should be in the detailed list of all servers


    ...

Tempest.lib includes a check-uuid tool that will test for the existence and uniqueness of idempotent_id metadata for every test. If you have Tempest installed you run the tool against Tempest by calling from the Tempest repo:

check-uuid

It can be invoked against any test suite by passing a package name:

check-uuid --package <package_name>

Tests without an idempotent_id can be automatically fixed by running the command with the --fix flag, which will modify the source package by inserting randomly generated uuids for every test that does not have one:

check-uuid --fix

The check-uuid tool is used as part of the Tempest gate job to ensure that all tests have an idempotent_id decorator.

Branchless Tempest Considerations

Starting with the OpenStack Icehouse release Tempest no longer has any stable branches. This is to better ensure API consistency between releases because the API behavior should not change between releases. This means that the stable branches are also gated by the Tempest master branch, which also means that proposed commits to Tempest must work against both the master and all the currently supported stable branches of the projects. As such there are a few special considerations that have to be accounted for when pushing new changes to Tempest.

1. New Tests for new features

When adding tests for new features that were not in previous releases of the projects the new test has to be properly skipped with a feature flag. This can be just as simple as using the @utils.requires_ext() or testtools.skipUnless decorators to check if the required extension (or discoverable optional API) or feature is enabled or can be as difficult as adding a new config option to the appropriate section. If there isn't a method of selecting the new feature from the config file then there won't be a mechanism to disable the test with older stable releases and the new test won't be able to merge.

Introduction of a new feature flag requires specifying a default value for the corresponding config option that is appropriate in the latest OpenStack release. Because Tempest is branchless, the feature flag's default value will need to be overridden to a value that is appropriate in earlier releases in which the feature isn't available. In DevStack, this can be accomplished by modifying Tempest's lib installation script for previous branches (because DevStack is branched).

2. Bug fix on core project needing Tempest changes

When trying to land a bug fix which changes a tested API you'll have to use the following procedure:

1. Propose change to the project, get a +2 on the change even with failing
2. Propose skip on Tempest which will only be approved after the


  corresponding change in the project has a +2 on change
3. Land project change in master and all open stable branches (if required)
4. Land changed test in Tempest

Otherwise the bug fix won't be able to land in the project.

Handily, Zuul's cross-repository dependencies. can be leveraged to do without step 2 and to have steps 3 and 4 happen "atomically". To do that, make the patch written in step 1 to depend (refer to Zuul's documentation above) on the patch written in step 4. The commit message for the Tempest change should have a link to the Gerrit review that justifies that change.

3. New Tests for existing features

If a test is being added for a feature that exists in all the current releases of the projects then the only concern is that the API behavior is the same across all the versions of the project being tested. If the behavior is not consistent the test will not be able to merge.

API Stability

For new tests being added to Tempest the assumption is that the API being tested is considered stable and adheres to the OpenStack API stability guidelines. If an API is still considered experimental or in development then it should not be tested by Tempest until it is considered stable.

Reviewing Tempest Code

To start read the OpenStack Common Review Checklist

Ensuring code is executed

For any new or change to a test it has to be verified in the gate. This means that the first thing to check with any change is that a gate job actually runs it. Tests which aren't executed either because of configuration or skips should not be accepted.

If a new test is added that depends on a new config option (like a feature flag), the commit message must reference a change in DevStack or DevStack-Gate that enables the execution of this newly introduced test. This reference could either be a Cross-Repository Dependency or a simple link to a Gerrit review.

Execution time

While checking in the job logs that a new test is actually executed, also pay attention to the execution time of that test. Keep in mind that each test is going to be executed hundreds of time each day, because Tempest tests run in many OpenStack projects. It's worth considering how important/critical the feature under test is with how costly the new test is.

Unit Tests

For any change that adds new functionality to either common functionality or an out-of-band tool unit tests are required. This is to ensure we don't introduce future regressions and to test conditions which we may not hit in the gate runs. API and scenario tests aren't required to have unit tests since they should be self-verifying by running them in the gate. All service clients, on the other hand, must have unit tests, as they belong to tempest/lib.

API Stability

Tests should only be added for published stable APIs. If a patch contains tests for an API which hasn't been marked as stable or for an API which doesn't conform to the API stability guidelines then it should not be approved.

Reject Copy and Paste Test Code

When creating new tests that are similar to existing tests it is tempting to simply copy the code and make a few modifications. This increases code size and the maintenance burden. Such changes should not be approved if it is easy to abstract the duplicated code into a function or method.

Tests overlap

When a new test is being proposed, question whether this feature is not already tested with Tempest. Tempest has more than 1200 tests, spread amongst many directories, so it's easy to introduce test duplication. For example, testing volume attachment to a server could be a compute test or a volume test, depending on how you see it. So one must look carefully in the entire code base for possible overlap. As a rule of thumb, the older a feature is, the more likely it's already tested.

Being explicit

When tests are being added that depend on a configurable feature or extension, polling the API to discover that it is enabled should not be done. This will just result in bugs being masked because the test can be skipped automatically. Instead the config file should be used to determine whether a test should be skipped or not. Do not approve changes that depend on an API call to determine whether to skip or not.

Configuration Options

With the introduction of the Tempest external test plugin interface we needed to provide a stable contract for Tempest's configuration options. This means we can no longer simply remove a configuration option when it's no longer used. Patches proposed that remove options without a deprecation cycle should not be approved. Similarly when changing default values with configuration we need to similarly be careful that we don't break existing functionality. Also, when adding options, just as before, we need to weigh the benefit of adding an additional option against the complexity and maintenance overhead having it costs.

Test Documentation

When a new test is being added refer to the Test Documentation section in hacking to see if the requirements are being met. With the exception of a class level docstring linking to the API ref doc in the API tests and a docstring for scenario tests this is up to the reviewers discretion whether a docstring is required or not.

Test Removal and Refactoring

Make sure that any test that is renamed, relocated (e.g. moved to another class), or removed does not belong to the interop testing suite -- which includes a select suite of Tempest tests for the purposes of validating that OpenStack vendor clouds are interoperable -- or a project's whitelist or blacklist files.

It is of critical importance that no interop, whitelist or blacklist test reference be broken by a patch set introduced to Tempest that renames, relocates or removes a referenced test.

Please check the existence of code which references Tempest tests with: http://codesearch.openstack.org/

Interop

Make sure that modifications to an interop test are backwards-compatible. This means that code modifications to tests should not undermine the quality of the validation currently performed by the test or significantly alter the behavior of the test.

Removal

Reference the Tempest Test Removal Procedure guidelines for understanding best practices associated with test removal.

Release Notes

Release notes are how we indicate to users and other consumers of Tempest what has changed in a given release. Since Tempest 10.0.0 we've been using reno to manage and build the release notes. There are certain types of changes that require release notes and we should not approve them without including a release note. These include but aren't limited to, any addition, deprecation or removal from the lib interface, any change to configuration options (including deprecation), CLI additions or deprecations, major feature additions, and anything backwards incompatible or would require a user to take note or do something extra.

Deprecated Code

Sometimes we have some bugs in deprecated code. Basically, we leave it. Because we don't need to maintain it. However, if the bug is critical, we might need to fix it. When it will happen, we will deal with it on a case-by-case basis.

When to approve

Microversion Testing With Tempest

Many OpenStack Services provide their APIs with microversion support and want to test them in Tempest.

This document covers how to test microversions for each project and whether tests should live in Tempest or on project side.

Tempest Scope For Microversion Testing

APIs microversions for any OpenStack service grow rapidly and testing each and every microversion in Tempest is not feasible and efficient way. Also not every API microversion changes the complete system behavior and many of them only change the API or DB layer to accept and return more data on API.

Tempest is an integration test suite, but not all API microversion testing fall under this category. As a result, Tempest mainly covers integration test cases for microversions, Other testing coverage for microversion should be hosted on project side as functional tests or via Tempest plugin as per project guidelines.

NOTE:

Tempest will cover only integration testing of applicable microversions with below exceptions:

Project Scope For Microversion Testing

All microversions testing which are not covered under Tempest as per above section, should be tested on project side as functional tests or as Tempest plugin as per project decision.

Configuration options for Microversion

[compute]
min_microversion = None
max_microversion = latest

How To Implement Microversion Tests

Tempest provides stable interfaces to test API Microversion. For Details, see: API Microversion testing Framework This document explains how to implement Microversion tests using those interfaces.

Step1: Add skip logic based on configured Microversion range

Add logic to skip the tests based on Tests class and configured Microversion range. api_version_utils.check_skip_with_microversion function can be used to automatically skip the tests which do not fall under configured Microversion range. For example:

class BaseTestCase1(api_version_utils.BaseMicroversionTest):


    [..]


    @classmethod


    def skip_checks(cls):


        super(BaseTestCase1, cls).skip_checks()


        api_version_utils.check_skip_with_microversion(cls.min_microversion,


                                                       cls.max_microversion,


                                                       CONF.compute.min_microversion,


                                                       CONF.compute.max_microversion)

Skip logic can be added in tests base class or any specific test class depends on tests class structure.

Step2: Selected API request microversion

Select appropriate Microversion which needs to be used to send with API request. api_version_utils.select_request_microversion function can be used to select the appropriate Microversion which will be used for API request. For example:

@classmethod
def resource_setup(cls):


    super(BaseTestCase1, cls).resource_setup()


    cls.request_microversion = (


        api_version_utils.select_request_microversion(


            cls.min_microversion,


            CONF.compute.min_microversion))

Step3: Set Microversion on Service Clients

Microversion selected by Test Class in previous step needs to be set on service clients so that APIs can be requested with selected Microversion.

Microversion can be defined as global variable on service clients which can be set using fixture. Also Microversion header name needs to be defined on service clients which should be constant because it is not supposed to be changed by project as per API contract. For example:

COMPUTE_MICROVERSION = None
class BaseClient1(rest_client.RestClient):


    api_microversion_header_name = 'X-OpenStack-Nova-API-Version'

Now test class can set the selected Microversion on required service clients using fixture which can take care of resetting the same once tests is completed. For example:

def setUp(self):


    super(BaseTestCase1, self).setUp()


    self.useFixture(api_microversion_fixture.APIMicroversionFixture(


        self.request_microversion))

Service clients needs to add set Microversion in API request header which can be done by overriding the get_headers() method of rest_client. For example:

COMPUTE_MICROVERSION = None
class BaseClient1(rest_client.RestClient):


    api_microversion_header_name = 'X-OpenStack-Nova-API-Version'


    def get_headers(self):


        headers = super(BaseClient1, self).get_headers()


        if COMPUTE_MICROVERSION:


            headers[self.api_microversion_header_name] = COMPUTE_MICROVERSION


        return headers

Step4: Separate Test classes for each Microversion

This is last step to implement Microversion test class.

For any Microversion tests, basically we need to implement a separate test class. In addition, each test class defines its Microversion range with class variable like min_microversion and max_microversion. Tests will be valid for that defined range. If that range is out of configured Microversion range then, test will be skipped.

NOTE:

For example:

Below test is applicable for Microversion from 2.2 till 2.9:

class BaseTestCase1(api_version_utils.BaseMicroversionTest,


                    tempest.test.BaseTestCase):


    [..]
class Test1(BaseTestCase1):


    min_microversion = '2.2'


    max_microversion = '2.9'


    [..]

Below test is applicable for Microversion from 2.10 till latest:

class Test2(BaseTestCase1):


    min_microversion = '2.10'


    max_microversion = 'latest'


    [..]

Notes about Compute Microversion Tests

Some of the compute Microversion tests have been already implemented with the Microversion testing framework. So for further tests only step 4 is needed.

Along with that JSON response schema might need versioning if needed.

Compute service clients strictly validate the response against defined JSON schema and does not allow additional elements in response. So if that Microversion changed the API response then schema needs to be versioned. New JSON schema file needs to be defined with new response attributes and service client methods will select the schema based on requested microversion.

If Microversion tests are implemented randomly meaning not in sequence order(v2.20 tests added and previous Microversion tests are not yet added) then, still schema might need to be version for older Microversion if they changed the response. This is because Nova Microversion includes all the previous Microversions behavior.

That can be done by using the get_schema method in below module:

The base_compute_client module

Microversion tests implemented in Tempest

Tempest Test Removal Procedure

Historically, Tempest was the only way of doing functional testing and integration testing in OpenStack. This was mostly only an artifact of Tempest being the only proven pattern for doing this, not an artifact of a design decision. However, moving forward, as functional testing is being spun up in each individual project, we really only want Tempest to be the integration test suite it was intended to be: testing the high-level interactions between projects through REST API requests. In this model, there are probably existing tests that aren't the best fit living in Tempest. However, since Tempest is largely still the only gating test suite in this space we can't carelessly rip out everything from the tree. This document outlines the procedure which was developed to ensure we minimize the risk for removing something of value from the Tempest tree.

This procedure might seem overly conservative and slow-paced, but this is by design to try to ensure we don't remove something that is actually providing value. Having potential duplication between testing is not a big deal especially compared to the alternative of removing something which is actually providing value and is actively catching bugs, or blocking incorrect patches from landing.

Proposing a test removal

3 prong rule for removal

In the proposal etherpad we'll be looking for answers to 3 questions:

The answers to 1 and 2 are easy to verify. For 1 just provide a link to the new test location. If you are linking to the Tempest removal patch please also put a Depends-On in the commit message for the commit which moved the test into another repo.

For prong 2 you can use subunit2sql:

Using subunit2sql directly

SELECT * from tests where test_id like "%test_id%"; (where $test_id is the full test_id, but truncated to the class because of setUpClass or tearDownClass failures)

You can access the infra mysql subunit2sql db w/ read-only permissions with:

For example if you were trying to remove the test with the id: tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON.test_get_flavor_details_for_deleted_flavor you would run the following:

MySQL [subunit2sql]> select * from tests where test_id like \
"tempest.api.compute.admin.test_flavors_negative.FlavorsAdminNegativeTestJSON%";

Eventually, a CLI interface will be created to make that a bit more friendly. Also a dashboard is in the works so we don't need to manually run the command.

The intent of the 2nd prong is to verify that moving the test into a project specific testing is preventing bugs (assuming the Tempest tests were catching issues) from bubbling up a layer into Tempest jobs. If we're seeing failure rates above a certain threshold in the gate checks that means the functional testing isn't really being effective in catching that bug (and therefore blocking it from landing) and having the testing run in Tempest still has value.

However, for the 3rd prong verification is a bit more subjective. The original intent of this prong was mostly for interop/refstack and also for things that running on the stable branches. We don't want to remove any tests if that would break our API consistency checking between releases, or something that interop/refstack is depending on being in Tempest. It's worth pointing out that if a test is used in interop_wg as part of interop testing then it will probably have continuing value being in Tempest as part of the integration/integrated tests in general. This is one area where some overlap is expected between testing in projects and Tempest, which is not a bad thing.

Discussing the 3rd prong

There are 2 approaches to addressing the 3rd prong. Either it can be raised during a QA meeting during the Tempest discussion. Please put it on the agenda well ahead of the scheduled meeting. Since the meeting time will be well known ahead of time anyone who depends on the tests will have ample time beforehand to outline any concerns on the before the meeting. To give ample time for people to respond to removal proposals please add things to the agenda by the Monday before the meeting.

The other option is to raise the removal on the openstack-discuss mailing list. (for example see: http://lists.openstack.org/pipermail/openstack-dev/2016-February/086218.html or http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003574.html ) This will raise the issue to the wider community and attract at least the same (most likely more) attention than discussing it during the irc meeting. The only downside is that it might take more time to get a response, given the nature of ML.

Exceptions to this procedure

For the most part, all Tempest test removals have to go through this procedure there are a couple of exceptions though:

For the first exception type, the only types of testing in the tree which have been declared out of scope at this point are:

For tests that fit into this category, the only criteria for removal is that there is equivalent testing elsewhere.

Tempest Scope

Starting in the liberty cycle Tempest, has defined a set of projects which are defined as in scope for direct testing in Tempest. As of today that list is:

Anything that lives in Tempest which doesn't test one of these projects can be removed assuming there is equivalent testing elsewhere. Preferably using the tempest plugin mechanism to maintain continuity after migrating the tests out of Tempest.

API Compatibility

If an API introduces a non-discoverable, backward-incompatible change, and such a change is not backported to all versions supported by Tempest, tests for that API cannot live in Tempest anymore. This is because tests would not be able to know or control which API response to expect, and thus would not be able to enforce a specific behavior.

If a test exists in Tempest that would meet these criteria as a consequence of a change, the test must be removed according to the procedure discussed in this document. The API change should not be merged until all conditions required for test removal can be met.

Tempest Test Writing Guide

This guide serves as a starting point for developers working on writing new Tempest tests. At a high level, tests in Tempest are just tests that conform to the standard python unit test framework. But there are several aspects of that are unique to Tempest and its role as an integration test suite running against a real cloud.

NOTE:

Adding a New TestCase

The base unit of testing in Tempest is the TestCase (also called the test class). Each TestCase contains test methods which are the individual tests that will be executed by the test runner. But, the TestCase is the smallest self contained unit for tests from the Tempest perspective. It's also the level at which Tempest is parallel safe. In other words, multiple TestCases can be executed in parallel, but individual test methods in the same TestCase can not. Also, all test methods within a TestCase are assumed to be executed serially. As such you can use the test case to store variables that are shared between methods.

In standard unittest the lifecycle of a TestCase can be described in the following phases:

setUpClass

The setUpClass phase is the first phase executed by the test runner and is used to perform any setup required for all the test methods to be executed. In Tempest this is a very important step and will automatically do the necessary setup for interacting with the configured cloud.

To accomplish this you do not define a setUpClass function, instead there are a number of predefined phases to setUpClass that are used. The phases are:

which is executed in that order. Cleanup of resources provisioned during the resource_setup must be scheduled right after provisioning using the addClassResourceCleanup helper. The resource cleanups stacked this way are executed in reverse order during tearDownClass, before the cleanup of test credentials takes place. An example of a TestCase which defines all of these would be:

from tempest.common import waiters
from tempest import config
from tempest.lib.common.utils import test_utils
from tempest import test
CONF = config.CONF
class TestExampleCase(test.BaseTestCase):


    @classmethod


    def skip_checks(cls):


        """This section is used to evaluate config early and skip all test


           methods based on these checks


        """


        super(TestExampleCase, cls).skip_checks()


        if not CONF.section.foo


            cls.skip('A helpful message')


    @classmethod


    def setup_credentials(cls):


        """This section is used to do any manual credential allocation and also


           in the case of dynamic credentials to override the default network


           resource creation/auto allocation


        """


        # This call is used to tell the credential allocator to not create any


        # network resources for this test case. It also enables selective


        # creation of other neutron resources. NOTE: it must go before the


        # super call


        cls.set_network_resources()


        super(TestExampleCase, cls).setup_credentials()


    @classmethod


    def setup_clients(cls):


        """This section is used to setup client aliases from the manager object


           or to initialize any additional clients. Except in a few very


           specific situations you should not need to use this.


        """


        super(TestExampleCase, cls).setup_clients()


        cls.servers_client = cls.os_primary.servers_client


    @classmethod


    def resource_setup(cls):


        """This section is used to create any resources or objects which are


           going to be used and shared by **all** test methods in the


           TestCase. Note then anything created in this section must also be


           destroyed in the corresponding resource_cleanup() method (which will


           be run during tearDownClass())


        """


        super(TestExampleCase, cls).resource_setup()


        cls.shared_server = cls.servers_client.create_server(...)


        cls.addClassResourceCleanup(waiters.wait_for_server_termination,


                                    cls.servers_client,


                                    cls.shared_server['id'])


        cls.addClassResourceCleanup(


            test_utils.call_and_ignore_notfound_exc(


                cls.servers_client.delete_server,


                cls.shared_server['id']))

Allocating Credentials

Since Tempest tests are all about testing a running cloud, every test will need credentials to be able to make API requests against the cloud. Since this is critical to operation and, when running in parallel, easy to make a mistake, the base TestCase class will automatically allocate a regular user for each TestCase during the setup_credentials() phase. During this process it will also initialize a client manager object using those credentials, which will be your entry point into interacting with the cloud. For more details on how credentials are allocated the Test Credentials section of the Tempest Configuration Guide provides more details on the operation of this.

There are some cases when you need more than a single set of credentials, or credentials with a more specialized set of roles. To accomplish this you have to set a class variable credentials on the TestCase directly. For example:

from tempest import test
class TestExampleAdmin(test.BaseTestCase):


    credentials = ['primary', 'admin']


    @classmethod


    def skip_checks(cls):


        ...

In this example the TestExampleAdmin TestCase will allocate 2 sets of credentials, one regular user and one admin user. The corresponding manager objects will be set as class variables cls.os_primary and cls.os_admin respectively. You can also allocate a second user by putting 'alt' in the list too. A set of alt credentials are the same as primary but can be used for tests cases that need a second user/project.

You can also specify credentials with specific roles assigned. This is useful for cases where there are specific RBAC requirements hard coded into an API. The canonical example of this are swift tests which often want to test swift's concepts of operator and reseller_admin. An actual example from Tempest on how to do this is:

class PublicObjectTest(base.BaseObjectTest):


    credentials = [['operator', CONF.object_storage.operator_role],


                   ['operator_alt', CONF.object_storage.operator_role]]


    @classmethod


    def setup_credentials(cls):


        super(PublicObjectTest, cls).setup_credentials()


        ...

In this case the manager objects will be set to cls.os_roles_operator and cls.os_roles_operator_alt respectively.

There is no limit to how many credentials you can allocate in this manner, however in almost every case you should not need more than 3 sets of credentials per test case.

To figure out the mapping of manager objects set on the TestCase and the requested credentials you can reference:

By default cls.os_primary is available since it is allocated in the base Tempest test class (located in tempest/test.py). If your TestCase inherits from a different direct parent class (it'll still inherit from the BaseTestCase, just not directly) be sure to check if that class overrides allocated credentials.

Dealing with Network Allocation

When Neutron is enabled and a testing requires networking this isn't normally automatically setup when a tenant is created. Since Tempest needs isolated tenants to function properly it also needs to handle network allocation. By default the base test class will allocate a network, subnet, and router automatically (this depends on the configured credential provider, for more details see: Network Creation/Usage for Servers). However, there are situations where you do no need all of these resources allocated (or your TestCase inherits from a class that overrides the default in tempest/test.py). There is a class level mechanism to override this allocation and specify which resources you need. To do this you need to call cls.set_network_resources() in the setup_credentials() method before the super(). For example:

from tempest import test
class TestExampleCase(test.BaseTestCase):


    @classmethod


    def setup_credentials(cls):


        cls.set_network_resources(network=True, subnet=True, router=False)


        super(TestExampleCase, cls).setup_credentials()

There are 2 quirks with the usage here. First for the set_network_resources function to work properly it must be called before super(). This is so that children classes' settings are always used instead of a parent classes'. The other quirk here is that if you do not want to allocate any network resources for your test class simply call set_network_resources() without any arguments. For example:

from tempest import test
class TestExampleCase(test.BaseTestCase):


    @classmethod


    def setup_credentials(cls):


        cls.set_network_resources()


        super(TestExampleCase, cls).setup_credentials()

This will not allocate any networking resources. This is because by default all the arguments default to False.

It's also worth pointing out that it is common for base test classes for different services (and scenario tests) to override this setting. When inheriting from classes other than the base TestCase in tempest/test.py it is worth checking the immediate parent for what is set to determine if your class needs to override that setting.

Interacting with Credentials and Clients

Once you have your basic TestCase setup you'll want to start writing tests. To do that you need to interact with an OpenStack deployment. This section will cover how credentials and clients are used inside of Tempest tests.

Manager Objects

The primary interface with which you interact with both credentials and API clients is the client manager object. These objects are created automatically by the base test class as part of credential setup (for more details see the previous Allocating Credentials section). Each manager object is initialized with a set of credentials and has each client object already setup to use that set of credentials for making all the API requests. Each client is accessible as a top level attribute on the manager object. So to start making API requests you just access the client's method for making that call and the credentials are already setup for you. For example if you wanted to make an API call to create a server in Nova:

from tempest import test
class TestExampleCase(test.BaseTestCase):


    def test_example_create_server(self):


        self.os_primary.servers_client.create_server(...)

is all you need to do. As described previously, in the above example the self.os_primary is created automatically because the base test class sets the credentials attribute to allocate a primary credential set and initializes the client manager as self.os_primary. This same access pattern can be used for all of the clients in Tempest.

Credentials Objects

In certain cases you need direct access to the credentials (the most common use case would be an API request that takes a user or project id in the request body). If you're in a situation where you need to access this you'll need to access the credentials object which is allocated from the configured credential provider in the base test class. This is accessible from the manager object via the manager's credentials attribute. For example:

from tempest import test
class TestExampleCase(test.BaseTestCase):


    def test_example_create_server(self):


        credentials = self.os_primary.credentials

The credentials object provides access to all of the credential information you would need to make API requests. For example, building off the previous example:

from tempest import test
class TestExampleCase(test.BaseTestCase):


    def test_example_create_server(self):


        credentials = self.os_primary.credentials


        username = credentials.username


        user_id = credentials.user_id


        password = credentials.password


        tenant_id = credentials.tenant_id

Requirements Upper Constraint for Tempest

Tempest is branchless and supported stable branches use Tempest master and all EM stable branches use old compatible Tempest version for their testing. This means the OpenStack installed upper-constraints might not be compatible with Tempest used for stable branch testing. For example, if Tempest master is used for testing the stable/stein then stable/stein constraint might not be compatible with Tempest master so we need to use master upper-constraints there. That is why we use virtual env for Tempest installation and running tests so that we can control Tempest required constraint from system wide installed constraints.

Devstack takes care of using the master upper-constraints when Tempest master is used. But when old Tempest is used then devstack alone cannot handle the compatible constraints because Tempest in-tree tox.ini also set the upper-constraints which are master constraints so if devstack set the different constraints than what we have in tox.ini we end up re-creation of venv which flush all previously installed tempest plugins in that venv. More details are on this ML thread

To solve that problem we have two ways:

How to pin upper-constraints in tox.ini

This has to be done exactly before we cut the Tempest new major version bump release for the cycle.

Plugins

Tempest Plugins Guide

Tempest Test Plugin Interface

Tempest has an external test plugin interface which enables anyone to integrate an external test suite as part of a Tempest run. This will let any project leverage being run with the rest of the Tempest suite while not requiring the tests live in the Tempest tree.

Creating a plugin

Creating a plugin is fairly straightforward and doesn't require much additional effort on top of creating a test suite using tempest.lib. One thing to note with doing this is that the interfaces exposed by Tempest are not considered stable (with the exception of configuration variables whichever effort goes into ensuring backward compatibility). You should not need to import anything from Tempest itself except where explicitly noted.

Stable Tempest APIs plugins may use

As noted above, several Tempest APIs are acceptable to use from plugins, while others are not. A list of stable APIs available to plugins is provided below:

If there is an interface from Tempest that you need to rely on in your plugin which is not listed above, it likely needs to be migrated to tempest.lib. In that situation, file a bug, push a migration patch, etc. to expedite providing the interface in a reliable manner.

Plugin Cookiecutter

In order to create the basic structure with base classes and test directories you can use the tempest-plugin-cookiecutter project:

> pip install -U cookiecutter && cookiecutter https://opendev.org/openstack/tempest-plugin-cookiecutter.git
Cloning into 'tempest-plugin-cookiecutter'...
remote: Counting objects: 17, done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 17 (delta 1), reused 14 (delta 1)
Unpacking objects: 100% (17/17), done.
Checking connectivity... done.
project (default is "sample")? foo
testclass (default is "SampleTempestPlugin")? FooTempestPlugin

This would create a folder called foo_tempest_plugin/ with all necessary basic classes. You only need to move/create your test in foo_tempest_plugin/tests.

Entry Point

Once you've created your plugin class you need to add an entry point to your project to enable Tempest to find the plugin. The entry point must be added to the "tempest.test_plugins" namespace.

If you are using pbr this is fairly straightforward, in the setup.cfg just add something like the following:

[entry_points]
tempest.test_plugins =


    plugin_name = module.path:PluginClass

Standalone Plugin vs In-repo Plugin

Since all that's required for a plugin to be detected by Tempest is a valid setuptools entry point in the proper namespace there is no difference from the Tempest perspective on either creating a separate python package to house the plugin or adding the code to an existing python project. However, there are tradeoffs to consider when deciding which approach to take when creating a new plugin.

If you create a separate python project for your plugin this makes a lot of things much easier. Firstly it makes packaging and versioning much simpler, you can easily decouple the requirements for the plugin from the requirements for the other project. It lets you version the plugin independently and maintain a single version of the test code across project release boundaries (see the Branchless Tempest Spec for more details on this). It also greatly simplifies the install time story for external users. Instead of having to install the right version of a project in the same python namespace as Tempest they simply need to pip install the plugin in that namespace. It also means that users don't have to worry about inadvertently installing a Tempest plugin when they install another package.

The sole advantage to integrating a plugin into an existing python project is that it enables you to land code changes at the same time you land test changes in the plugin. This reduces some of the burden on contributors by not having to land 2 changes to add a new API feature and then test it and doing it as a single combined commit.

Plugin Class

To provide Tempest with all the required information it needs to be able to run your plugin you need to create a plugin class which Tempest will load and call to get information when it needs. To simplify creating this Tempest provides an abstract class that should be used as the parent for your plugin. To use this you would do something like the following:

from tempest.test_discover import plugins
class MyPlugin(plugins.TempestPlugin):

Then you need to ensure you locally define all of the mandatory methods in the abstract class, you can refer to the api doc below for a reference of what that entails.

Abstract Plugin Class

Plugin Structure

While there are no hard and fast rules for the structure of a plugin, there are basically no constraints on what the plugin looks like as long as the 2 steps above are done. However, there are some recommended patterns to follow to make it easy for people to contribute and work with your plugin. For example, if you create a directory structure with something like:

plugin_dir/


  config.py


  plugin.py


  tests/


    api/


    scenario/


  services/


    client.py

That will mirror what people expect from Tempest. The file

Additionally, when you're creating the plugin you likely want to follow all of the Tempest developer and reviewer documentation to ensure that the tests being added in the plugin act and behave like the rest of Tempest.

Dealing with configuration options

Historically, Tempest didn't provide external guarantees on its configuration options. However, with the introduction of the plugin interface, this is no longer the case. An external plugin can rely on using any configuration option coming from Tempest, there will be at least a full deprecation cycle for any option before it's removed. However, just the options provided by Tempest may not be sufficient for the plugin. If you need to add any plugin specific configuration options you should use the register_opts and get_opt_lists methods to pass them to Tempest when the plugin is loaded. When adding configuration options the register_opts method gets passed the CONF object from Tempest. This enables the plugin to add options to both existing sections and also create new configuration sections for new options.

Service Clients

If a plugin defines a service client, it is beneficial for it to implement the get_service_clients method in the plugin class. All service clients which are exposed via this interface will be automatically configured and be available in any instance of the service clients class, defined in tempest.lib.services.clients.ServiceClients. In case multiple plugins are installed, all service clients from all plugins will be registered, making it easy to write tests which rely on multiple APIs whose service clients are in different plugins.

Example implementation of get_service_clients:

def get_service_clients(self):


    # Example implementation with two service clients


    my_service1_config = config.service_client_config('my_service')


    params_my_service1 = {


        'name': 'my_service_v1',


        'service_version': 'my_service.v1',


        'module_path': 'plugin_tempest_tests.services.my_service.v1',


        'client_names': ['API1Client', 'API2Client'],


    }


    params_my_service1.update(my_service_config)


    my_service2_config = config.service_client_config('my_service')


    params_my_service2 = {


        'name': 'my_service_v2',


        'service_version': 'my_service.v2',


        'module_path': 'plugin_tempest_tests.services.my_service.v2',


        'client_names': ['API1Client', 'API2Client'],


    }


    params_my_service2.update(my_service2_config)


    return [params_my_service1, params_my_service2]

Parameters:

Example usage of the service clients in tests:

# my_creds is instance of tempest.lib.auth.Credentials
# identity_uri is v2 or v3 depending on the configuration
from tempest.lib.services import clients
my_clients = clients.ServiceClients(my_creds, identity_uri)
my_service1_api1_client = my_clients.my_service_v1.API1Client()
my_service2_api1_client = my_clients.my_service_v2.API1Client(my_args='any')

Automatic configuration and registration of service clients imposes some extra constraints on the structure of the configuration options exposed by the plugin.

First service_version should be in the format service_config[.version]. The .version part is optional, and should only be used if there are multiple versions of the same API available. The service_config must match the name of a configuration options group defined by the plugin. Different versions of one API must share the same configuration group.

Second the configuration options group service_config must contain the following options:

The following options will be honoured if defined, but they are not mandatory, as they do not necessarily apply to all service clients.

Third the service client classes should inherit from RestClient, should accept generic keyword arguments, and should pass those arguments to the __init__ method of RestClient. Extra arguments can be added. For instance:

class MyAPIClient(rest_client.RestClient):


    def __init__(self, auth_provider, service, region,


                 my_arg, my_arg2=True, **kwargs):


        super(MyAPIClient, self).__init__(


            auth_provider, service, region, **kwargs)


        self.my_arg = my_arg


        self.my_args2 = my_arg

Finally the service client should be structured in a python module, so that all service client classes are importable from it. Each major API version should have its own module.

The following folder and module structure is recommended for a single major API version:

plugin_dir/


  services/


    __init__.py


    client_api_1.py


    client_api_2.py

The content of __init__.py module should be:

from client_api_1.py import API1Client
from client_api_2.py import API2Client
__all__ = ['API1Client', 'API2Client']

The following folder and module structure is recommended for multiple major API version:

plugin_dir/


  services/


    v1/


       __init__.py


       client_api_1.py


       client_api_2.py


    v2/


       __init__.py


       client_api_1.py


       client_api_2.py

The content each of __init__.py module under vN should be:

from client_api_1.py import API1Client
from client_api_2.py import API2Client
__all__ = ['API1Client', 'API2Client']

Using Plugins

Tempest will automatically discover any installed plugins when it is run. So by just installing the python packages which contain your plugin you'll be using them with Tempest, nothing else is really required.

However, you should take care when installing plugins. By their very nature there are no guarantees when running Tempest with plugins enabled about the quality of the plugin. Additionally, while there is no limitation on running with multiple plugins, it's worth noting that poorly written plugins might not properly isolate their tests which could cause unexpected cross interactions between plugins.

Notes for using plugins with virtualenvs

When using a Tempest inside a virtualenv (like when running under tox) you have to ensure that the package that contains your plugin is either installed in the venv too or that you have system site-packages enabled. The virtualenv will isolate the Tempest install from the rest of your system so just installing the plugin package on your system and then running Tempest inside a venv will not work.

Tempest also exposes a tox job, all-plugin, which will setup a tox virtualenv with system site-packages enabled. This will let you leverage tox without requiring to manually install plugins in the tox venv before running tests.

Stable Branch Support Policy

Stable Branch Support Policy

Since the Extended Maintenance policy for stable branches was adopted OpenStack projects will keep stable branches around after a "stable" or "maintained" period for a phase of indeterminate length called "Extended Maintenance". Prior to this resolution Tempest supported all stable branches which were supported upstream. This policy does not scale under the new model as Tempest would be responsible for gating proposed changes against an ever increasing number of branches. Therefore due to resource constraints, Tempest will only provide support for branches in the "Maintained" phase from the documented Support Phases. When a branch moves from the Maintained to the Extended Maintenance phase, Tempest will tag the removal of support for that branch as it has in the past when a branch goes end of life.

The expectation for Extended Maintenance phase branches is that they will continue running Tempest during that phase of support. Since the REST APIs are stable interfaces across release boundaries, branches in these phases should run Tempest from master as long as possible. But, because we won't be actively testing branches in these phases, it's possible that we'll introduce changes to Tempest on master which will break support on Extended Maintenance phase branches. When this happens the expectation for those branches is to either switch to running Tempest from a tag with support for the branch, or exclude a newly introduced test (if that is the cause of the issue). Tempest will not be creating stable branches to support Extended Maintenance phase branches, as the burden is on the Extended Maintenance phase branche maintainers, not the Tempest project, to support that branch.

Stable Branch Testing Policy

Stable Branch Testing Policy

Tempest and its plugins need to support the stable branches as per Stable Branch Support Policy.

Because of branchless model of Tempest and plugins, all the supported stable branches use the Tempest and plugins master version for their testing. That is done in devstack by using the master branch for the Tempest installation. To make sure the master version of Tempest or plugins (for any changes or adding new tests) is compatible for all the supported stable branches testing, Tempest and its plugins need to add the stable branches job on the master gate. That way can test the stable branches against master code and can avoid breaking supported branches accidentally.

Example:

Once any stable branch is moved to the Extended Maintenance Phases and devstack start using the Tempest older version for that stable branch testing then we can remove that stable branch job from master gate.

Example: https://review.opendev.org/#/c/722183/

Tempest & Plugins Compatible Version Policy

Tempest and Plugins compatible version policy

Tempest and its plugins are responsible for the integrated testing of OpenStack. These tools have two use cases:

Upstream code is tested by the master version of branchless Tempest & plugins for all supported stable branches in Maintained phase.

Production Cloud can be tested by using the compatible version or using master version. It depends on the testing strategy of cloud. To provide the compatible version of Tempest and its Plugins per OpenStack release, we started the coordinated release of all plugins and Tempest per OpenStack release. These versions are the first set of versions from Tempest and its Plugins to officially start the support of a particular OpenStack release. For example: OpenStack Train release first compatible versions Tempest plugins version.

Because of branchless nature of Tempest and its plugins, first version released during OpenStack release is not the last version to support that OpenStack release. This means the next (or master) versions can also be used for upstream testing as well as in production testing.

Since the Extended Maintenance policy for stable branch, Tempest started releasing the end of support version once stable release is moved to EM state, which used to happen on EOL of stable release. This is the last compatible version of Tempest for the OpenStack release moved to EM.

Because of branchless nature as explained above, we have a range of versions which can be considered a compatible version for particular OpenStack release. How we should release those versions is mentioned in the below table.

Tempest & the Plugins should follow the above mentioned policy for the First compatible version and the Last compatible version. so that we provide the right set of compatible versions to Upstream as well as to Production Cloud testing.

Plugins Registry

Tempest Plugin Registry

Since we've created the external plugin mechanism, it's gotten used by a lot of projects. The following is a list of plugins that currently exist.

Detected Plugins

The following are plugins that a script has found in the openstack/ namespace, which includes but is not limited to official OpenStack projects.

Tempest & Plugins Compatible Version Policy

Keystone Scopes & Roles Support in Tempest

Keystone Scopes & Roles Support in Tempest

OpenStack Keystone supports different scopes in token, refer to the Keystone doc. Along with the scopes, keystone supports default roles, one of which is a reader role, for details refer to this keystone document.

Tempest supports those scopes and roles credentials that can be used to test APIs under different scope and roles.

Dynamic Credentials

Dynamic credential supports all the below set of personas and allows you to generate credentials tailored to a specific persona that you can use in your test.

Domain scoped personas:

class TestDummy(base.DummyBaseTest):


    credentials = ['domain_admin']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_d_admin_client = (


            cls.os_domain_admin.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['domain_member']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_d_member_client = (


            cls.os_domain_member.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['domain_reader']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_d_reader_client = (


            cls.os_domain_reader.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = [['domain_my_role1', 'my_own_role1', 'admin']


                   ['domain_my_role2', 'my_own_role2']]


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_d_role1_client = (


            cls.os_domain_my_role1.availability_zone_client)


        cls.az_d_role2_client = (


            cls.os_domain_my_role2.availability_zone_client)

System scoped personas:

class TestDummy(base.DummyBaseTest):


    credentials = ['system_admin']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_s_admin_client = (


            cls.os_system_admin.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['system_member']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_s_member_client = (


            cls.os_system_member.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['system_reader']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_s_reader_client = (


            cls.os_system_reader.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = [['system_my_role1', 'my_own_role1', 'admin']


                   ['system_my_role2', 'my_own_role2']]


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_s_role1_client = (


            cls.os_system_my_role1.availability_zone_client)


        cls.az_s_role2_client = (


            cls.os_system_my_role2.availability_zone_client)

Project scoped personas:

class TestDummy(base.DummyBaseTest):


    credentials = ['project_admin']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_p_admin_client = (


            cls.os_project_admin.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['project_member']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_p_member_client = (


            cls.os_project_member.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['project_reader']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_p_reader_client = (


            cls.os_project_reader.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['project_alt_admin']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_p_alt_admin_client = (


            cls.os_project_alt_admin.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['project_alt_member']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_p_alt_member_client = (


            cls.os_project_alt_member.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = ['project_alt_reader']


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_p_alt_reader_client = (


            cls.os_project_alt_reader.availability_zone_client)

class TestDummy(base.DummyBaseTest):


    credentials = [['project_my_role1', 'my_own_role1', 'admin']


                   ['project_my_role2', 'my_own_role2']]


    @classmethod


    def setup_clients(cls):


        super(TestDummy, cls).setup_clients()


        cls.az_role1_client = (


            cls.os_project_my_role1.availability_zone_client)


        cls.az_role2_client = (


            cls.os_project_my_role2.availability_zone_client)

Pre-Provisioned Credentials

Pre-Provisioned credentials support the below set of personas and can be used in the test as shown above in the Dynamic Credentials Section.

Stable Branch Support Policy

Stable Branch Testing Policy

Library

Tempest Library Documentation

Tempest provides a stable library interface that provides external tools or test suites an interface for reusing pieces of Tempest code. Any public interface that lives in tempest/lib in the Tempest repo is treated as a stable public interface and it should be safe to external consume that. Every effort goes into maintaining backwards compatibility with any change. The library is self contained and doesn't have any dependency on other Tempest internals outside of lib (including no usage of Tempest configuration).

Stability

Any code that lives in tempest/lib will be treated as a stable interface. This means that any public interface under the tempest/lib directory is expected to be a stable interface suitable for public consumption. However, for any interfaces outside of tempest/lib in the tempest tree (unless otherwise noted) or any private interfaces the same stability guarantees don't apply.

Adding Interfaces

When adding an interface to tempest/lib we have to make sure there are no dependencies on any pieces of tempest outside of tempest/lib. This means if for example there is a dependency on the configuration file we need remove that. The other aspect when adding an interface is to make sure it's really an interface ready for external consumption and something we want to commit to supporting.

Making changes

When making changes to tempest/lib you have to be conscious of the effect of any changes on external consumers. If your proposed change set will change the default behaviour of any interface, or make something which previously worked not after your change, then it is not acceptable. Every effort needs to go into preserving backwards compatibility in changes.

Reviewing

When reviewing a proposed change to tempest/lib code we need to be careful to ensure that we don't break backward compatibility. For patches that change existing interfaces, we have to be careful to make sure we don't break any external consumers. Some common red flags are:

Testing

When adding a new interface to the library we need to at a minimum have unit test coverage. A proposed change to add an interface to tempest/lib that doesn't have unit tests shouldn't be accepted. Ideally, these unit tests will provide sufficient coverage to ensure a stable interface moving forward.

Current Library APIs

CLI Testing Framework Usage

The cli.base module

The cli.output_parser module

Collection of utilities for parsing CLI clients output.

Decorators Usage Guide

The decorators module

Rest Client Usage

The rest_client module

Utils Usage

The misc module

API Microversion Testing Support in Tempest

Framework to support API Microversion testing

Many of the OpenStack components have implemented API microversions. It is important to test those microversions in Tempest or external plugins. Tempest now provides stable interfaces to support to test the API microversions. Based on the microversion range coming from the combination of both configuration and each test case, APIs request will be made with selected microversion.

This document explains the interfaces needed for microversion testing.

The api_version_request module

The api_version_utils module

Authentication Framework Usage

The auth module

Service Clients Usage

Tests make requests against APIs using service clients. Service clients are specializations of the RestClient class. The service clients that cover the APIs exposed by a service should be grouped in a service clients module. A service clients module is python module where all service clients are defined. If major API versions are available, submodules should be defined, one for each version.

The ClientsFactory class helps initializing all clients of a specific service client module from a set of shared parameters.

The ServiceClients class provides a convenient way to get access to all available service clients initialized with a provided set of credentials.

The clients management module

# johndoe is a tempest.lib.auth.Credentials type instance
johndoe_clients = clients.ServiceClients(johndoe, identity_uri)
# List servers in default region
johndoe_servers_client = johndoe_clients.compute.ServersClient()
johndoe_servers = johndoe_servers_client.list_servers()
# List servers in Region B
johndoe_servers_client_B = johndoe_clients.compute.ServersClient(


    region='B')
johndoe_servers = johndoe_servers_client_B.list_servers()

from tempest import config
params = {}
for service_version in available_modules():


    service = service_version.split('.')[0]


    params[service] = config.service_client_config(service)
service_clients = ServiceClients(creds, identity_uri,


                                 client_parameters=params)

Compute service client modules

Compute Client Usage

Credential Providers

These library interfaces are used to deal with allocating credentials on demand either dynamically by calling keystone to allocate new credentials, or from a list of preprovisioned credentials. These 2 modules are implementations of the same abstract credential providers class and can be used interchangeably. However, each implementation has some additional parameters that are used to influence the behavior of the modules. The API reference at the bottom of this doc shows the interface definitions for both modules, however that may be a bit opaque. You can see some examples of how to leverage this interface below.

Initialization Example

This example is from Tempest itself (from tempest/common/credentials_factory.py just modified slightly) and is how it initializes the credential provider based on config:

from tempest import config
from tempest.lib.common import dynamic_creds
from tempest.lib.common import preprov_creds
CONF = config.CONF
def get_credentials_provider(name, network_resources=None,


                             force_tenant_isolation=False,


                             identity_version=None):


    # If a test requires a new account to work, it can have it via forcing


    # dynamic credentials. A new account will be produced only for that test.


    # In case admin credentials are not available for the account creation,


    # the test should be skipped else it would fail.


    identity_version = identity_version or CONF.identity.auth_version


    if CONF.auth.use_dynamic_credentials or force_tenant_isolation:


        admin_creds = get_configured_admin_credentials(


            fill_in=True, identity_version=identity_version)


        return dynamic_creds.DynamicCredentialProvider(


            name=name,


            network_resources=network_resources,


            identity_version=identity_version,


            admin_creds=admin_creds,


            identity_admin_domain_scope=CONF.identity.admin_domain_scope,


            identity_admin_role=CONF.identity.admin_role,


            extra_roles=CONF.auth.tempest_roles,


            neutron_available=CONF.service_available.neutron,


            project_network_cidr=CONF.network.project_network_cidr,


            project_network_mask_bits=CONF.network.project_network_mask_bits,


            public_network_id=CONF.network.public_network_id,


            create_networks=(CONF.auth.create_isolated_networks and not


                             CONF.network.shared_physical_network),


            resource_prefix='tempest',


            credentials_domain=CONF.auth.default_credentials_domain_name,


            admin_role=CONF.identity.admin_role,


            identity_uri=CONF.identity.uri_v3,


            identity_admin_endpoint_type=CONF.identity.v3_endpoint_type)


    else:


        if CONF.auth.test_accounts_file:


            # Most params are not relevant for pre-created accounts


            return preprov_creds.PreProvisionedCredentialProvider(


                name=name, identity_version=identity_version,


                accounts_lock_dir=lockutils.get_lock_path(CONF),


                test_accounts_file=CONF.auth.test_accounts_file,


                object_storage_operator_role=CONF.object_storage.operator_role,


                object_storage_reseller_admin_role=reseller_admin_role,


                credentials_domain=CONF.auth.default_credentials_domain_name,


                admin_role=CONF.identity.admin_role,


                identity_uri=CONF.identity.uri_v3,


                identity_admin_endpoint_type=CONF.identity.v3_endpoint_type)


        else:


            raise exceptions.InvalidConfiguration(


                'A valid credential provider is needed')

This function just returns an initialized credential provider class based on the config file. The consumer of this function treats the output as the same regardless of whether it's a dynamic or preprovisioned provider object.

Dealing with Credentials

Once you have a credential provider object created the access patterns for allocating and removing credentials are the same across both the dynamic and preprovisioned credentials. These are defined in the abstract CredentialProvider class. At a high level the credentials provider enables you to get 3 basic types of credentials at once (per object): a primary, alt, and admin. You're also able to allocate a credential by role. These credentials are tracked by the provider object and delete must manually be called otherwise the created resources will not be deleted (or returned to the pool in the case of preprovisioned creds)

Examples

Continuing from the example above, to allocate credentials by the 3 basic types you can do the following:

provider = get_credentials_provider('my_tests')
primary_creds = provider.get_primary_creds()
alt_creds = provider.get_alt_creds()
admin_creds = provider.get_admin_creds()
# Make sure to delete the credentials when you're finished
provider.clear_creds()

To create and interact with credentials by role you can do the following:

provider = get_credentials_provider('my_tests')
my_role_creds = provider.get_creds_by_role({'roles': ['my_role']})
# provider.clear_creds() will clear all creds including those allocated by
# role
provider.clear_creds()

When multiple roles are specified a set of creds with all the roles assigned will be allocated:

provider = get_credentials_provider('my_tests')
my_role_creds = provider.get_creds_by_role({'roles': ['my_role',


                                                      'my_other_role']})
# provider.clear_creds() will clear all creds including those allocated by
# role
provider.clear_creds()

If you need multiple sets of credentials with the same roles you can also do this by leveraging the force_new kwarg:

provider = get_credentials_provider('my_tests')
my_role_creds = provider.get_creds_by_role({'roles': ['my_role']})
my_role_other_creds = provider.get_creds_by_role({'roles': ['my_role']},


                                                 force_new=True)
# provider.clear_creds() will clear all creds including those allocated by
# role
provider.clear_creds()

API Reference

The dynamic credentials module

The pre-provisioned credentials module

accounts_lock_dir = os.path.join(lockutils.get_lock_path(CONF),


                                 'test_accounts')

Validation Resources

The validation_resources module

from tempest.common import validation_resources as vr
from tempest.lib import auth
from tempest.lib.services import clients
creds = auth.get_credentials('http://mycloud/identity/v3',


                             username='me', project_name='me',


                             password='secret', domain_name='Default')
osclients = clients.ServiceClients(creds, 'http://mycloud/identity/v3')
# Request keypair and floating IP
resources = dict(keypair=True, security_group=False,


                 security_group_rules=False, floating_ip=True)
resources = vr.create_validation_resources(


    osclients, validation_resources=resources, use_neutron=True,


    floating_network_id='4240E68E-23DA-4C82-AC34-9FEFAA24521C')
# Now cleanup the resources
try:


    vr.clear_validation_resources(osclients, use_neutron=True,


                                  **resources)
except Exception as e:


    LOG.exception('Something went wrong during cleanup, ignoring')

from tempest.common import validation_resources as vr
from tempest.lib import auth
from tempest.lib.services import clients
creds = auth.get_credentials('http://mycloud/identity/v3',


                             username='me', project_name='me',


                             password='secret', domain_name='Default')
osclients = clients.ServiceClients(creds, 'http://mycloud/identity/v3')
# Security group for IPv4 tests
sg4 = vr.create_ssh_security_group(osclients, add_rule=True)
# Security group for IPv6 tests
sg6 = vr.create_ssh_security_group(osclients, ethertype='IPv6',


                                   add_rule=True)

from tempest.common import validation_resources as vr
from tempest.lib import auth
from tempest.lib.services import clients
creds = auth.get_credentials('http://mycloud/identity/v3',


                             username='me', project_name='me',


                             password='secret', domain_name='Default')
osclients = clients.ServiceClients(creds, 'http://mycloud/identity/v3')
# Request keypair and floating IP
resources = dict(keypair=True, security_group=False,


                 security_group_rules=False, floating_ip=True)
resources = vr.create_validation_resources(


    osclients, use_neutron=True,


    floating_network_id='4240E68E-23DA-4C82-AC34-9FEFAA24521C',


    **resources)
# The floating IP to be attached to the VM
floating_ip = resources['floating_ip']['ip']