manpages

osprofiler(1)

OSPROFILER(1) OSprofiler OSPROFILER(1)

NAME

osprofiler - OSprofiler 4.3.0

OSProfiler provides a tiny but powerful library that is used by most (soon to be all) OpenStack projects and their python clients. It provides functionality to generate 1 trace per request, that goes through all involved services. This trace can then be extracted and used to build a tree of calls which can be quite handy for a variety of reasons (for example in isolating cross-project performance issues).

USING OSPROFILER

OSProfiler provides a tiny but powerful library that is used by most (soon to be all) OpenStack projects and their python clients. It provides functionality to generate 1 trace per request, that goes through all involved services. This trace can then be extracted and used to build a tree of calls which can be quite handy for a variety of reasons (for example in isolating cross-project performance issues).

Background

OpenStack consists of multiple projects. Each project, in turn, is composed of multiple services. To process some request, e.g. to boot a virtual machine, OpenStack uses multiple services from different projects. In the case something works too slow, it's extremely complicated to understand what exactly goes wrong and to locate the bottleneck.

To resolve this issue, we introduce a tiny but powerful library, osprofiler, that is going to be used by all OpenStack projects and their python clients. It generates 1 trace per request, that goes through all involved services, and builds a tree of calls.

Why not cProfile and etc?

The scope of this library is quite different:

  • We are interested in getting one trace of points from different services, not tracing all Python calls inside one process.
  • This library should be easy integrable into OpenStack. This means that:
  • It shouldn't require too many changes in code bases of projects it's integrated with.
  • We should be able to fully turn it off.
  • We should be able to keep it turned on in lazy mode in production (e.g. admin should be able to "trace" on request).


API

There are few things that you should know about API before using it.

Five ways to add a new trace point.

from osprofiler import profiler
def some_func():


    profiler.start("point_name", {"any_key": "with_any_value"})


    # your code


    profiler.stop({"any_info_about_point": "in_this_dict"})
@profiler.trace("point_name",


                info={"any_info_about_point": "in_this_dict"},


                hide_args=False)
def some_func2(*args, **kwargs):


    # If you need to hide args in profile info, put hide_args=True


    pass
def some_func3():


    with profiler.Trace("point_name",


                        info={"any_key": "with_any_value"}):


        # some code here
@profiler.trace_cls("point_name", info={}, hide_args=False,


                    trace_private=False)
class TracedClass(object):


    def traced_method(self):


        pass


    def _traced_only_if_trace_private_true(self):


         pass
class RpcManagerClass(object, metaclass=profiler.TracedMeta):


    __trace_args__ = {'name': 'rpc',


                      'info': None,


                      'hide_args': False,


                      'trace_private': False}


     def my_method(self, some_args):


         pass


     def my_method2(self, some_arg1, some_arg2, kw=None, kw2=None)


         pass


How profiler works?

  • profiler.Trace() and @profiler.trace() are just syntax sugar, that just calls profiler.start() & profiler.stop() methods.
  • Every call of profiler.start() & profiler.stop() sends to collector 1 message. It means that every trace point creates 2 records in the collector. (more about collector & records later)
  • Nested trace points are supported. The sample below produces 2 trace points:

profiler.start("parent_point")
profiler.start("child_point")
profiler.stop()
profiler.stop()


The implementation is quite simple. Profiler has one stack that contains ids of all trace points. E.g.:

profiler.start("parent_point") # trace_stack.push(<new_uuid>)


                               # send to collector -> trace_stack[1]
profiler.start("child_point") # trace_stack.push(<new_uuid>)


                               # send to collector -> trace_stack[2]
profiler.stop()                # send to collector -> trace_stack[2]


                               # trace_stack.pop()
profiler.stop()                # send to collector -> trace_stack[1]


                               # trace_stack.pop()


It's simple to build a tree of nested trace points, having (parent_id, point_id) of all trace points.


Process of sending to collector.

Trace points contain 2 messages (start and stop). Messages like below are sent to a collector:

{


    "name": <point_name>-(start|stop)


    "base_id": <uuid>,


    "parent_id": <uuid>,


    "trace_id": <uuid>,


    "info": <dict>
}


The fields are defined as the following:

  • base_id - <uuid> that is equal for all trace points that belong to one trace, this is done to simplify the process of retrieving all trace points related to one trace from collector
  • parent_id - <uuid> of parent trace point
  • trace_id - <uuid> of current trace point
  • info - the dictionary that contains user information passed when calling profiler start() & stop() methods.

Setting up the collector.

Using OSProfiler notifier.

NOTE:

The following way of configuring OSProfiler is deprecated. The new version description is located below - Using OSProfiler initializer.. Don't use OSproliler notifier directly! Its support will be removed soon from OSProfiler.


The profiler doesn't include a trace point collector. The user/developer should instead provide a method that sends messages to a collector. Let's take a look at a trivial sample, where the collector is just a file:

import json
from osprofiler import notifier
def send_info_to_file_collector(info, context=None):


    with open("traces", "a") as f:


        f.write(json.dumps(info))
notifier.set(send_info_to_file_collector)


So now on every profiler.start() and profiler.stop() call we will write info about the trace point to the end of the traces file.

Using OSProfiler initializer.

OSProfiler now contains various storage drivers to collect tracing data. Information about what driver to use and what options to pass to OSProfiler are now stored in OpenStack services configuration files. Example of such configuration can be found below:

[profiler]
enabled = True
trace_sqlalchemy = True
hmac_keys = SECRET_KEY
connection_string = messaging://


If such configuration is provided, OSProfiler setting up can be processed in following way:

if CONF.profiler.enabled:


    osprofiler_initializer.init_from_conf(


        conf=CONF,


        context=context.get_admin_context().to_dict(),


        project="cinder",


        service=binary,


        host=host


    )


Initialization of profiler.

If profiler is not initialized, all calls to profiler.start() and profiler.stop() will be ignored.

Initialization is a quite simple procedure.

from osprofiler import profiler
profiler.init("SECRET_HMAC_KEY", base_id=<uuid>, parent_id=<uuid>)


SECRET_HMAC_KEY - will be discussed later, because it's related to the integration of OSprofiler & OpenStack.

base_id and trace_id will be used to initialize stack_trace in profiler, e.g. stack_trace = [base_id, trace_id].

OSProfiler CLI.

To make it easier for end users to work with profiler from CLI, OSProfiler has entry point that allows them to retrieve information about traces and present it in human readable form.

Available commands:

•
Help message with all available commands and their arguments:

$ osprofiler -h/--help


•
OSProfiler version:

$ osprofiler -v/--version


•
Results of profiling can be obtained in JSON (option: --json) and HTML (option: --html) formats:

$ osprofiler trace show <trace_id> --json/--html


hint: option --out will redirect result of osprofiler trace show in specified file:

$ osprofiler trace show <trace_id> --json/--html --out /path/to/file


•
In latest versions of OSProfiler with storage drivers (e.g. MongoDB (URI: mongodb://), Messaging (URI: messaging://)) --connection-string parameter should be set up:

$ osprofiler trace show <trace_id> --connection-string=<URI> --json/--html



Integration

There are 4 topics related to integration OSprofiler & OpenStack:

What we should use as a centralized collector?

We primarily decided to use Ceilometer, because:

  • It's already integrated in OpenStack, so it's quite simple to send notifications to it from all projects.
  • There is an OpenStack API in Ceilometer that allows us to retrieve all messages related to one trace. Take a look at osprofiler.drivers.ceilometer.Ceilometer:get_report

In OSProfiler starting with 1.4.0 version other options (MongoDB driver in 1.4.0 release, Elasticsearch driver added later, etc.) are also available.

How to setup profiler notifier?

We primarily decided to use oslo.messaging Notifier API, because:

  • oslo.messaging is integrated in all projects
  • It's the simplest way to send notification to Ceilometer, take a look at: osprofiler.drivers.messaging.Messaging:notify method
  • We don't need to add any new CONF options in projects

In OSProfiler starting with 1.4.0 version other options (MongoDB driver in 1.4.0 release, Elasticsearch driver added later, etc.) are also available.

How to initialize profiler, to get one trace across all services?

To enable cross service profiling we actually need to do send from caller to callee (base_id & trace_id). So callee will be able to init its profiler with these values.

In case of OpenStack there are 2 kinds of interaction between 2 services:

•
REST API

It's well known that there are python clients for every project, that generate proper HTTP requests, and parse responses to objects.

These python clients are used in 2 cases:

  • User access -> OpenStack
  • Service from Project 1 would like to access Service from Project 2

So what we need is to:

  • Put in python clients headers with trace info (if profiler is inited)
  • Add OSprofiler WSGI middleware to your service, this initializes the profiler, if and only if there are special trace headers, that are signed by one of the HMAC keys from api-paste.ini (if multiple keys exist the signing process will continue to use the key that was accepted during validation).
•
The common items that are used to configure the middleware are the following (these can be provided when initializing the middleware object or when setting up the api-paste.ini file):

hmac_keys = KEY1, KEY2 (can be a single key as well)




Actually the algorithm is a bit more complex. The Python client will also sign the trace info with a HMAC key (lets call that key A) passed to profiler.init, and on reception the WSGI middleware will check that it's signed with one of the HMAC keys (the wsgi server should have key A as well, but may also have keys B and C) that are specified in api-paste.ini. This ensures that only the user that knows the HMAC key A in api-paste.ini can init a profiler properly and send trace info that will be actually processed. This ensures that trace info that is sent in that does not pass the HMAC validation will be discarded. NOTE: The application of many possible validation keys makes it possible to roll out a key upgrade in a non-impactful manner (by adding a key into the list and rolling out that change and then removing the older key at some time in the future).

•
Optionally you can enable client tracing using requests, Currently only supported by OTLP driver, this will add client call tracing. see profiler/trace_requests's option.

•
RPC API

RPC calls are used for interaction between services of one project. It's well known that projects are using oslo.messaging to deal with RPC. It's very good, because projects deal with RPC in similar way.

So there are 2 required changes:

  • On callee side put in request context trace info (if profiler was initialized)
  • On caller side initialize profiler, if there is trace info in request context.
  • Trace all methods of callee API (can be done via profiler.trace_cls).


What points should be tracked by default?

I think that for all projects we should include by default 5 kinds of points:

  • All HTTP calls - helps to get information about: what HTTP requests were done, duration of calls (latency of service), information about projects involved in request.
  • All RPC calls - helps to understand duration of parts of request related to different services in one project. This information is essential to understand which service produce the bottleneck.
  • All DB API calls - in some cases slow DB query can produce bottleneck. So it's quite useful to track how much time request spend in DB layer.
  • All driver calls - in case of nova, cinder and others we have vendor drivers. Duration
  • ALL SQL requests (turned off by default, because it produce a lot of traffic)

Collectors

There are a number of drivers to support different collector backends:

Redis

  • Overview

    The Redis driver allows profiling data to be collected into a redis database instance. The traces are stored as key-value pairs where the key is a string built using trace ids and timestamps and the values are JSON strings containing the trace information. A second driver is included to use Redis Sentinel in addition to single node Redis.

  • Capabilities
  • Write trace data to the database.
  • Query Traces in database: This allows for pulling trace data querying on the keys used to save the data in the database.
  • Generate a report based on the traces stored in the database.
  • Supports use of Redis Sentinel for robustness.

  • Usage

    The driver is used by OSProfiler when using a connection-string URL of the form redis://[:password]@host[:port][/db]. To use the Sentinel version use a connection-string of the form redissentinel://[:password]@host[:port][/db]

  • Configuration
  • No config changes are required by for the base Redis driver.
  • There are two configuration options for the Redis Sentinel driver:
  • socket_timeout: specifies the sentinel connection socket timeout value. Defaults to: 0.1 seconds
  • sentinel_service_name: The name of the Sentinel service to use. Defaults to: "mymaster"



SQLAlchemy

The SQLAlchemy collector allows you to store profiling data into a database supported by SQLAlchemy.

Usage

To use the driver, the connection_string in the [osprofiler] config section needs to be set to a connection string that SQLAlchemy understands For example:

[osprofiler]
connection_string = mysql+pymysql://username:password@192.168.192.81/profiler?charset=utf8


where username is the database username, password is the database password, 192.168.192.81 is the database IP address and profiler is the database name.

The database (in this example called profiler) needs to be created manually and the database user (in this example called username) needs to have priviliges to create tables and select and insert rows.

NOTE:

SQLAlchemy collector requires database JSON data type support. This type of data is supported by versions listed below or higher:
  • MariaDB 10.2
  • MySQL 5.7.8



OTLP

Use OTLP exporter. Can be used with any comptable backend that support OTLP.

Usage

To use the driver, the connection_string in the [osprofiler] config section needs to be set:

[osprofiler]
connection_string = otlp://192.168.192.81:4318


Example: By default, jaeger is listening OTLP on 4318.

NOTE:

Curently the exporter is only supporting HTTP. In future some work may happen to support gRPC.


Similar projects

Other projects (some alive, some abandoned, some research prototypes) that are similar (in idea and ideal to OSprofiler).

  • Zipkin
  • Dapper
  • Tomograph
  • HTrace
  • Jaeger
  • OpenTracing

Release Notes

ChangeLog

CHANGES

4.3.0

  • tox: Remove basepython
  • Run pyupgrade to clean up Python 2 syntaxes
  • Drop redundant if-else block
  • pre-commit: Bump version
  • Drop remaining pin of elasticsearch
  • Remove Python 3.8 support
  • Unpin for newer elasticsearch versions
  • Replace deprecated constant_time_compare
  • Bump hacking to latest
  • Update bandit to 1.7.x

4.2.0

  • reno: Update master for unmaintained/victoria
  • Update python classifier in setup.cfg
  • List up extra requirements for drivers
  • Move oslo.config to normal requirements
  • Declare Python 3.10 support

4.1.0

•
profiler: add python requests profile

4.0.0

  • add support of otlp exporter
  • devstack: remove jaeger container on unstack
  • [profiler] hmac_key should be secret
  • jaeger: introduce process tags' option for tracer
  • jaeger: introduce service name prefix
  • jaeger: fix driver initialization for tests
  • setup.cfg: Replace dashes with underscores
  • Change StrictRedis usage to Redis
  • tox: Add functional-py38, functional-py39 envs
  • remove unicode prefix from code
  • Fix formattiing of release list

3.4.3

  • Make some revisions in the document
  • Update CI to use unversioned jobs template
  • Fix api index and module index

3.4.2

•
Set manila config opts in devstack

3.4.1

  • Uncap PrettyTable
  • Move flake8 as a pre-commit local target
  • Use TOX_CONSTRAINTS_FILE
  • Dropping lower constraints testing
  • Remove six
  • Use py3 as the default runtime for tox
  • Adding pre-commit
  • ignore reno generated artifacts
  • Add Python3 wallaby unit tests
  • Update master for stable/victoria

3.4.0

  • Fix StopIteration error on Ubuntu Focal
  • Bump bandit version

3.3.0

•
switch to importlib.metadata to find package version

3.2.2

•
Fix empty 'args' and 'kwargs' when using 'hide_args' with Jaeger

3.2.1

•
drop mock from lower-constraints

3.2.0

  • Fix pep8 failures
  • Switch to newer openstackdocstheme and reno versions
  • Bump default tox env from py37 to py38
  • Add py38 package metadata
  • Enforce order of import statements
  • Use unittest.mock instead of third party mock
  • Add Python3 victoria unit tests
  • Update master for stable/ussuri

3.1.0

  • Update hacking for Python3
  • Start README.rst with a better title

3.0.0

•
[ussuri][goal] Drop python 2.7 support and testing

2.9.0

  • Switch to Ussuri jobs
  • Update master for stable/train
  • Handle driver initialization errors to avoid service crash

2.8.2

•
Add Python 3 Train unit tests

2.8.1

  • Bring env OSPROFILER_CONNECTION_STRING into effect
  • Support standalone placement in the devstack

2.8.0

  • Automatic configuration of SQLAlchemy driver in DevStack
  • change function list_traces of mongodb module
  • Replace git.openstack.org URLs with opendev.org URLs
  • Fix elasticsearch version in python requirements

2.7.0

  • Minimum versions of databases with JSON data type support
  • OpenDev Migration Patch
  • Dropping the py35 testing
  • Optimize storage schema for Redis driver
  • Allow OSPROFILER_TRACE_SQLALCHEMY to be overridden
  • Rename OSProfiler-enabled Tempest job
  • Don't fail if sqlalchemy driver fails to initialize
  • Update master for stable/stein

2.6.0

  • Add sqlalchemy collector
  • Change python3.5 job to python3.7 job on Stein+
  • Use $STACK_USER variable in install_jaeger function
  • Add support for mongodb backend in devstack plugin
  • Reload keystone to apply osprofiler config
  • Do not insert osprofiler filter into Neutron api-paste pipeline
  • Allow test path to be overridden

2.5.2

  • In case of an error, always add message
  • Change http to https in reference link
  • [devstack] Add support for elasticsearch backend
  • Change openstack-dev to openstack-discuss
  • Configure Jaeger collector in DevStack

2.5.1

  • Update min tox version to 2.0
  • In DevStack install Redis client library via pip, not as system package
  • When shortening span-ids, check if they're already short
  • Don't quote {posargs} in tox.ini

2.5.0

  • build universal wheels
  • Collect traces from Tempest job
  • Add a job to run full Tempest with enabled profiler
  • Make tracing of SQL statements configurable in DevStack plugin

2.4.1

  • Use templates for cover and lower-constraints
  • add password for connecting redis-sentinel
  • add lib-forward-testing-python3 test job
  • add python 3.6 unit test job
  • import zuul job settings from project-config
  • Update reno for stable/rocky
  • Switch to stestr

2.3.0

  • Add release note link in README
  • Make profiler.clean() public method

2.2.0

  • opts: Fix invalid rST formatting
  • Put 'db' parameter back and provide a deprecation warning before remove
  • Add minimum version and fix dulwich issue
  • fix tox python3 overrides
  • Update documentation & usage for redis driver
  • Allow user to specify password for Redis connection
  • OSprofiler with Jaeger Tracing as backend

2.1.0

  • Fix typo
  • Remove lower bound from requirements
  • Trivial: Update pypi url to new url
  • Add lower-constraints job
  • set default python to python3

2.0.0

  • Filter traces that contain error/exception
  • Zuul: Remove project name
  • Update reno for stable/queens
  • Unify and fix `list_traces` function
  • Add initial 'trace list' command
  • Remove Ceilometer support
  • Check profiler instance before initialize
  • update sphinx-doc links
  • Initialize osprofiler in Nova Cell configs
  • Add oslo.serialization into requirements

1.15.1

•
Cleanup test-requirements

1.15.0

  • Update the invalid doc links to the right ones in osprofiler docs
  • Add filter for OSprofiler html output
  • Add kwargs to WsgiMiddleware __init__
  • Make collector configurable in DevStack plugin
  • Add functional test for Redis driver
  • Remove setting of version/release from releasenotes
  • Add Zuul job for functional testing

1.14.0

  • Extend messaging driver to support reporting
  • Handle and report SQLAlchemy errors

1.13.0

•
Remove dependency on oslo.log library

1.12.0

  • Do not require OpenStack authentication to run osprofiler CLI
  • Make dependency on oslo.messaging runtime only
  • Make test_notifier independent of test case execution order
  • Add function/sql results to trace info
  • Improve unit test coverage
  • Remove unused parameters from Profiler class
  • Add loading local static files option of template.html
  • Update reno for stable/pike

1.11.0

  • Update URLs in documents according to document migration
  • doc: Fix formatting
  • rearrange existing documentation to fit the new standard layout
  • Switch from oslosphinx to openstackdocstheme
  • Enable warning-is-error in doc build
  • Update .gitignore

1.10.1

•
Expose connection_string parameter into DevStack plugin

1.10.0

  • Cleanup code of DevStack plugin
  • Improve error reporting for Ceilometer driver
  • Replace oslo.messaging.get_transport with get_notification_transport

1.9.1

•
devstack: use project conf file env variables

1.9.0

  • Fix error message for invalid trace
  • Remove unused imports

1.8.0

  • Switch to "topics" keyword for messaging driver
  • Add zun to devstack config
  • Python 3.4 support is removed

1.7.0

•
Highlight last trace for OSprofiler html output

1.6.0

  • Add magnum to devstack config
  • Add Jaeger to list of similar projects
  • [Fix gate]Update test requirement
  • fix an outdated link for zipkin
  • Change some bindings to one-time bindings
  • Revert "Change list_opts to dictionary style"
  • Move implemeted specs to implemented directory
  • Remove extra white spaces in json output
  • Increase angular digest iteration limit
  • devstack: make option hmac_keys configurable
  • Update reno for stable/ocata
  • Change list_opts to dictionary style
  • Fix mistake in split meta string
  • Fix enabling order specify in README.rst

1.5.0

  • Add py35 tox virtualenv
  • Add functional test for notifier backend
  • Upgrade libraries, add highlight for JSON data
  • Fix syntax in JS, JSON indent with 4 spaces
  • Pass oslo.messaging kwargs only for "messaging://"
  • Organize unit tests under tests/unit folder
  • Use uuidutils instead of uuid.uuid4()
  • Use oslo_utils.uuidutils.is_uuid_like
  • Error out for invalid trace ID
  • Re-format html template
  • Replace six.iteritems() with .items()
  • Show team and repo badges on README
  • Fix import order
  • Move hacking checks outside tests
  • Visualize trace output as graph
  • Remove print statement
  • Pretty print json output
  • Add exception to trace point details
  • Add a redis driver
  • Replace logging with oslo_log
  • Add Log Insight driver
  • Add reno for release notes management
  • Use method constant_time_compare from oslo.utils
  • Update documentation to the latest state
  • Update dependencies' version from project requirements
  • Update devstack plugin readme to enable Panko
  • Enable devstack to configure OSProfiler for Senlin project
  • Add .idea folder to .gitignore
  • Heat and Cinder now use new style conf
  • Fix the issue that ChangeLog not found when building docs
  • Add AUTHORS and ChangeLog to .gitignore
  • Update the driver path in th doc
  • Use an env variable for connection string default
  • Fix a doc typo
  • Update homepage with developer documentation page
  • Trivial: Remove vim header from source files
  • [doc]Add description for multi-backend URI
  • Add Elasticsearch driver
  • Remove old notifiers

1.4.0

  • Add tests for mongodb driver
  • Add connection string usage to osprofiler-cli
  • Add overall profiler stats by operation
  • Fix typos on spec directory
  • Fix title of index page
  • Add MongoDB driver
  • OSprofiler initialization method
  • Add Ceilometer driver
  • Add backward compatible drivers structure
  • Expose osprofiler middleware as entrypoint
  • Remove discover from test-requirements
  • Fix typo: 'Olso' to 'Oslo'
  • Don't set html_last_updated_fmt without git
  • Add exception type to stop trace info

1.3.0

  • Add hepler to trace sessions
  • doc: Log warning when can't get informaiton from git
  • Add an error tip when trace_id is not found
  • Add a similar link with reference to similar projects/libraries
  • Continue work on standardizing osprofiler docs
  • Remove dead/broken link to example
  • Updates to doc conf.py to look the same as other projects
  • Clean thread local profiler object after usage
  • Improve unit test coverage
  • Avoid tracing class and static methods
  • Avoid multiple tracing when applying meta or class decorator
  • Remove outdated version
  • Dont claim copyright for future years
  • Use pkg_resources to get version
  • Enable bandit in gate
  • Fallback if git is absent
  • It's unnecessary set deprecate group for option 'enabled'
  • Add CONTRIBUTING.rst

1.2.0

•
Remove flake8 ignore list in tox.ini

1.1.0

  • run py34 tests before py27 to work around testr bug
  • stop making a copy of options discovered by config generator
  • Make class detection more accurate

1.0.1

•
Disable staticmethods tracing

1.0.0

  • Add fix for static and class methods in @trace_cls
  • Expose X-Trace-* constants
  • Add raw Ceilometer events support to DevStack plugin
  • Use raw data storage for events to collect more info
  • Use oslo.utils reflection and avoid refinding decorated name
  • Move osprofiler tests into osprofiler
  • Consolidate osprofiler options
  • Remove argparse from requirements
  • Add py34 to tox env list
  • Make profiler timestamp json.dumps friendly
  • Replace deprecated library function os.popen() with subprocess
  • Add DevStack plugin

0.4.0

  • Remove Py33 support
  • Make it possible to specify file path as a source for trace
  • Remove support for py26
  • Improve HTML reports performance
  • Fix TracedMeta class
  • Fix a couple of typos in doc strings
  • Fix Ceilometer parser to use events
  • remove python 2.6 trove classifier
  • Add TracedMeta class
  • Update requirements
  • Deprecated tox -downloadcache option removed
  • Fix enable/disable compatibility
  • Add hacking rules & fix hacking issues

0.3.1

  • Make api-paste.ini config optional
  • Fix minor typos in the multi-backend specification
  • Spec: Integration Testing
  • Spec: Better DevStack Integration
  • Spec: Multi Backend support
  • Spec: Optional options in api-paste.ini
  • Add specs base structure
  • Update .gitreview for new namespace
  • Fix date parsing when there's not milliseconds in the date
  • Various cleanups
  • Remove version from setup.cfg
  • Stop using intersphinx
  • Rename doc environment to docs
  • Imporve generated trace html
  • Adding a hits to notice operator when trace not found

0.3.0

  • Cut version 0.3.0
  • add more unit tests
  • Allow N-keys (one should apply)
  • Some minor fixes in README.rst
  • ReadMe updates with CLI commands
  • Add entry point for OSProfiler, that display traces
  • Remove dead code
  • Add OSprofiler docs
  • Fix wrong code duplication in utils.itersubclasses()
  • Use compare_digest or an equivalent when available

0.2.5

  • Imporve read me
  • Fix issue with trace_cls
  • Add @profiler.trace_cls decorator
  • Prevent Messaging to resend failed notifications
  • Update README.rst with some small adjustments
  • Some grammar-related imprevements

0.2.4

  • Add alternative way to dissable middleware
  • Improve tracing of sqlalchemy

0.2.3

•
Fix ceilometer parse notifications

0.2.2

  • Improve a bit README.rst
  • Fix & improve trace decorator
  • Fix some typos in README.rst

0.2.1

  • Update README.rst
  • Add @profiler.trace decorator
  • Add missing tests for messaging notifer plugin

0.2.0

  • Add notifier plugin based on Ceilometer
  • Add base for notifier plugins
  • Make profiler.utils private
  • Improve ceilometer notifications getter
  • Move public methods to top of sqlalchemy module
  • Refactor web.add_trace_id_header()
  • Make a cleaner API for osporfiler
  • Add "_" to names of private methods

0.1.1

•
Remove unused libs from requirments and fix info in setup.cfg

0.1.0

  • Add extra docs in sqlalchemy module
  • Make hmac required argument in profiler.Profiler.init
  • Refactor WSGI.middleware and imporve test coverage
  • Improve test coverage
  • Improve README
  • Base64 encode the 'X-Trace-Info' header
  • Fix text requirements
  • Edit notifier.notify()
  • Add sanity tests for profiler and hmac usage
  • Imporve ceilometer parser
  • Split code sugar and logic in Profiler class
  • Simplify notifer API
  • Add git review file
  • Add in hmac signing/verification
  • Make name also use a deque
  • Use a collections.deque which has thread safe pop/append

0.0.1

  • Add work around if not all messages were consumed by ceilometer
  • Remove information about service in profiler
  • Add parser of ceilometer notifications
  • Fix setup.cfg python 2.6 is supported as well
  • Add possibility to disable sqlalchemy tracing
  • Fix WSGI middleware and add unit tests
  • Remove from sqlachemy.after_execute notifcation resutls and add UTs
  • Imporove profiler and add UTs
  • Update global requirments
  • Remove unused dependency from requirments.txt
  • Fix licenses
  • Fix pep
  • Add tracer for sqlalchemy
  • Add WSGI Middleware
  • Add profiler class
  • Init Strucutre of lib
  • Initial commit

OSPROFILER

osprofiler package

Subpackages

osprofiler.cmd package

Submodules

osprofiler.cmd.cliutils module

osprofiler.cmd.cliutils.add_arg(func, *args, **kwargs)
Bind CLI arguments to a shell.py do_foo function.

osprofiler.cmd.cliutils.arg(*args, **kwargs)
Decorator for CLI args.

Example:

>>> @arg("name", help="Name of the new entity")


    ... def entity_create(args):


    ... pass

osprofiler.cmd.cliutils.env(*args, **kwargs)
Returns the first environment variable set.

If all are empty, defaults to '' or keyword arg default.


osprofiler.cmd.commands module

class osprofiler.cmd.commands.BaseCommand
Bases: object
group_name = None


class osprofiler.cmd.commands.TraceCommands
Bases: BaseCommand
group_name = 'trace'

list(args)
List all traces

show(args)
Display trace results in HTML, JSON or DOT format.


osprofiler.cmd.shell module

Command-line interface to the OpenStack Profiler.

class osprofiler.cmd.shell.OSProfilerShell(argv)
Bases: object

osprofiler.cmd.shell.main(args=None)

Module contents

osprofiler.drivers package

Submodules

osprofiler.drivers.base module

class osprofiler.drivers.base.Driver(connection_str, project=None, service=None, host=None, **kwargs)
Bases: object

Base Driver class.

This class provides protected common methods that do not rely on a specific storage backend. Public methods notify() and/or get_report(), which require using storage backend API, must be overridden and implemented by any class derived from this class.

default_trace_fields = {'base_id', 'timestamp'}

classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Forms and returns report composed from the stored notifications.
Parameters
base_id -- Base id of trace elements.


list_error_traces()
Query all error traces from the storage.
:return List of traces, where each trace is a dictionary containing
base_id and timestamp.


list_traces(fields=None)
Query all traces from the storage.
Parameters
fields -- Set of trace fields to return. Defaults to 'base_id' and 'timestamp'
Returns
List of traces, where each trace is a dictionary containing at least base_id and timestamp.


notify(info, **kwargs)
This method will be called on each notifier.notify() call.

To add new drivers you should, create new subclass of this class and implement notify method.

Parameters
info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. Used to simplify retrieving of all trace elements from the backend. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.



osprofiler.drivers.base.get_driver(connection_string, *args, **kwargs)
Create driver's instance according to specified connection string

osprofiler.drivers.elasticsearch_driver module

class osprofiler.drivers.elasticsearch_driver.ElasticsearchDriver(connection_str, index_name='osprofiler-notifications', project=None, service=None, host=None, conf=<oslo_config.cfg.ConfigOpts object>, **kwargs)
Bases: Driver
classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Retrieves and parses notification from Elasticsearch.
Parameters
base_id -- Base id of trace elements.


list_error_traces()
Returns all traces that have error/exception.

list_traces(fields=None)
Query all traces from the storage.
Parameters
fields -- Set of trace fields to return. Defaults to 'base_id' and 'timestamp'
Returns
List of traces, where each trace is a dictionary containing at least base_id and timestamp.


notify(info)
Send notifications to Elasticsearch.
Parameters
info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. Used to simplify retrieving of all trace elements from Elasticsearch. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.


notify_error_trace(info)
Store base_id and timestamp of error trace to a separate index.


osprofiler.drivers.jaeger module

class osprofiler.drivers.jaeger.Jaeger(connection_str, project=None, service=None, host=None, conf=<oslo_config.cfg.ConfigOpts object>, **kwargs)
Bases: Driver
create_span_tags(payload)
Create tags for OpenTracing span.
Parameters
info -- Information from OSProfiler trace.
Returns tags
A dictionary contains standard tags from OpenTracing sematic conventions, and some other custom tags related to http, db calls.


classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Please use Jaeger Tracing UI for this task.

list_error_traces()
Please use Jaeger Tracing UI for this task.

list_traces(fields=None)
Please use Jaeger Tracing UI for this task.

notify(payload)
This method will be called on each notifier.notify() call.

To add new drivers you should, create new subclass of this class and implement notify method.

Parameters
info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. Used to simplify retrieving of all trace elements from the backend. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.



osprofiler.drivers.loginsight module

Classes to use VMware vRealize Log Insight as the trace data store.

class osprofiler.drivers.loginsight.LogInsightClient(host, username, password, api_port=9000, api_ssl_port=9543, query_timeout=60000)
Bases: object

A minimal Log Insight client.

CURRENT_SESSIONS_PATH = 'api/v1/sessions/current'

EVENTS_INGEST_PATH = 'api/v1/events/ingest/F52D775B-6017-4787-8C8A-F21AE0AEC057'

LI_OSPROFILER_AGENT_ID = 'F52D775B-6017-4787-8C8A-F21AE0AEC057'

QUERY_EVENTS_BASE_PATH = 'api/v1/events'

SESSIONS_PATH = 'api/v1/sessions'

login()

query_events(params)

send_event(event)


class osprofiler.drivers.loginsight.LogInsightDriver(connection_str, project=None, service=None, host=None, **kwargs)
Bases: Driver

Driver for storing trace data in VMware vRealize Log Insight.

The driver uses Log Insight ingest service to store trace data and uses the query service to retrieve it. The minimum required Log Insight version is 3.3.

The connection string to initialize the driver should be of the format: loginsight://<username>:<password>@<loginsight-host>

If the username or password contains the character ':' or '@', it must be escaped using URL encoding. For example, the connection string to connect to Log Insight server at 10.1.2.3 using username "osprofiler" and password "p@ssword" is: loginsight://osprofiler:p%40ssword@10.1.2.3

classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Retrieves and parses trace data from Log Insight.
Parameters
base_id -- Trace base ID


notify(info)
Send trace to Log Insight server.


osprofiler.drivers.messaging module

class osprofiler.drivers.messaging.Messaging(connection_str, project=None, service=None, host=None, context=None, conf=None, transport_url=None, idle_timeout=1, **kwargs)
Bases: Driver
classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Forms and returns report composed from the stored notifications.
Parameters
base_id -- Base id of trace elements.


notify(info, context=None)
Send notifications to backend via oslo.messaging notifier API.
Parameters
  • info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.
  • context -- request context that is mostly used to specify current active user and tenant.




class osprofiler.drivers.messaging.NotifyEndpoint(oslo_messaging, base_id)
Bases: object
get_last_read_time()

get_messages()

info(ctxt, publisher_id, event_type, payload, metadata)


exception osprofiler.drivers.messaging.SignalExit
Bases: BaseException

osprofiler.drivers.messaging.signal_handler(signum, frame, state)

osprofiler.drivers.mongodb module

class osprofiler.drivers.mongodb.MongoDB(connection_str, db_name='osprofiler', project=None, service=None, host=None, **kwargs)
Bases: Driver
classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Retrieves and parses notification from MongoDB.
Parameters
base_id -- Base id of trace elements.


list_error_traces()
Returns all traces that have error/exception.

list_traces(fields=None)
Query all traces from the storage.
Parameters
fields -- Set of trace fields to return. Defaults to 'base_id' and 'timestamp'
Returns
List of traces, where each trace is a dictionary containing at least base_id and timestamp.


notify(info)
Send notifications to MongoDB.
Parameters
info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. Used to simplify retrieving of all trace elements from MongoDB. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.


notify_error_trace(data)
Store base_id and timestamp of error trace to a separate db.


osprofiler.drivers.otlp module

class osprofiler.drivers.otlp.OTLP(connection_str, project=None, service=None, host=None, conf=<oslo_config.cfg.ConfigOpts object>, **kwargs)
Bases: Driver
create_span_tags(payload)
Create tags an OpenTracing compatible span.
Parameters
info -- Information from OSProfiler trace.
Returns tags
A dictionary contains standard tags from OpenTracing sematic conventions, and some other custom tags related to http, db calls.


classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Forms and returns report composed from the stored notifications.
Parameters
base_id -- Base id of trace elements.


list_error_traces()
Query all error traces from the storage.
:return List of traces, where each trace is a dictionary containing
base_id and timestamp.


list_traces(fields=None)
Query all traces from the storage.
Parameters
fields -- Set of trace fields to return. Defaults to 'base_id' and 'timestamp'
Returns
List of traces, where each trace is a dictionary containing at least base_id and timestamp.


notify(payload)
This method will be called on each notifier.notify() call.

To add new drivers you should, create new subclass of this class and implement notify method.

Parameters
info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. Used to simplify retrieving of all trace elements from the backend. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.



osprofiler.drivers.redis_driver module

class osprofiler.drivers.redis_driver.Redis(connection_str, db=0, project=None, service=None, host=None, conf=<oslo_config.cfg.ConfigOpts object>, **kwargs)
Bases: Driver
classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Retrieves and parses notification from Redis.
Parameters
base_id -- Base id of trace elements.


list_error_traces()
Returns all traces that have error/exception.

list_traces(fields=None)
Query all traces from the storage.
Parameters
fields -- Set of trace fields to return. Defaults to 'base_id' and 'timestamp'
Returns
List of traces, where each trace is a dictionary containing at least base_id and timestamp.


notify(info)
Send notifications to Redis.
Parameters
info -- Contains information about trace element. In payload dict there are always 3 ids: "base_id" - uuid that is common for all notifications related to one trace. Used to simplify retrieving of all trace elements from Redis. "parent_id" - uuid of parent element in trace "trace_id" - uuid of current element in trace With parent_id and trace_id it's quite simple to build tree of trace elements, which simplify analyze of trace.


notify_error_trace(data)
Store base_id and timestamp of error trace to a separate key.


class osprofiler.drivers.redis_driver.RedisSentinel(connection_str, db=0, project=None, service=None, host=None, conf=<oslo_config.cfg.ConfigOpts object>, **kwargs)
Bases: Redis, Driver
classmethod get_name()
Returns backend specific name for the driver.


osprofiler.drivers.sqlalchemy_driver module

class osprofiler.drivers.sqlalchemy_driver.SQLAlchemyDriver(connection_str, project=None, service=None, host=None, **kwargs)
Bases: Driver
classmethod get_name()
Returns backend specific name for the driver.

get_report(base_id)
Forms and returns report composed from the stored notifications.
Parameters
base_id -- Base id of trace elements.


list_traces(fields=None)
Query all traces from the storage.
Parameters
fields -- Set of trace fields to return. Defaults to 'base_id' and 'timestamp'
Returns
List of traces, where each trace is a dictionary containing at least base_id and timestamp.


notify(info, context=None)
Write a notification the the database


Module contents

Submodules

osprofiler.exc module

exception osprofiler.exc.CommandError(message=None)
Bases: Exception

Invalid usage of CLI.


exception osprofiler.exc.LogInsightAPIError
Bases: Exception

exception osprofiler.exc.LogInsightLoginTimeout
Bases: Exception

osprofiler.initializer module

osprofiler.initializer.init_from_conf(conf, context, project, service, host, **kwargs)
Initialize notifier from service configuration
Parameters
  • conf -- service configuration
  • context -- request context
  • project -- project name (keystone, cinder etc.)
  • service -- service name that will be profiled
  • host -- hostname or host IP address that the service will be running on.
  • kwargs -- other arguments for notifier creation



osprofiler.notifier module

osprofiler.notifier.clear_notifier_cache()

osprofiler.notifier.create(connection_string, *args, **kwargs)
Create notifier based on specified plugin_name
Parameters
  • connection_string -- connection string which specifies the storage driver for notifier
  • args -- args that will be passed to the driver's __init__ method
  • kwargs -- kwargs that will be passed to the driver's __init__ method

Returns
Callable notifier method


osprofiler.notifier.get()
Returns notifier callable.

osprofiler.notifier.notify(info)
Passes the profiling info to the notifier callable.
Parameters
info -- dictionary with profiling information


osprofiler.notifier.set(notifier)
Service that are going to use profiler should set callable notifier.

Callable notifier is instance of callable object, that accept exactly one argument "info". "info" - is dictionary of values that contains profiling information.


osprofiler.opts module

osprofiler.opts.list_opts()

osprofiler.opts.set_defaults(conf, enabled=None, trace_sqlalchemy=None, hmac_keys=None, connection_string=None, es_doc_type=None, es_scroll_time=None, es_scroll_size=None, socket_timeout=None, sentinel_service_name=None)

osprofiler.profiler module

class osprofiler.profiler.Trace(name, info=None)
Bases: object

class osprofiler.profiler.TracedMeta(cls_name, bases, attrs)
Bases: type

Metaclass to comfortably trace all children of a specific class.

Possible usage:

>>>  class RpcManagerClass(object, metaclass=profiler.TracedMeta):
>>>      __trace_args__ = {'name': 'rpc',
>>>                        'info': None,
>>>                        'hide_args': False,
>>>                        'hide_result': True,
>>>                        'trace_private': False}
>>>
>>>      def my_method(self, some_args):
>>>          pass
>>>
>>>      def my_method2(self, some_arg1, some_arg2, kw=None, kw2=None)
>>>          pass

Adding of this metaclass requires to set __trace_args__ attribute to the class we want to modify. __trace_args__ is the dictionary with one mandatory key included - "name", that will define name of action to be traced - E.g. wsgi, rpc, db, etc...


osprofiler.profiler.clean()

osprofiler.profiler.get()
Get profiler instance.
Returns
Profiler instance or None if profiler wasn't inited.


osprofiler.profiler.init(hmac_key, base_id=None, parent_id=None)
Init profiler instance for current thread.

You should call profiler.init() before using osprofiler. Otherwise profiler.start() and profiler.stop() methods won't do anything.

Parameters
  • hmac_key -- secret key to sign trace information.
  • base_id -- Used to bind all related traces.
  • parent_id -- Used to build tree of traces.

Returns
Profiler instance


osprofiler.profiler.start(name, info=None)
Send new start notification if profiler instance is presented.
Parameters
  • name -- The name of action. E.g. wsgi, rpc, db, etc..
  • info -- Dictionary with extra trace information. For example in wsgi it can be url, in rpc - message or in db sql - request.



osprofiler.profiler.stop(info=None)
Send new stop notification if profiler instance is presented.

osprofiler.profiler.trace(name, info=None, hide_args=False, hide_result=True, allow_multiple_trace=True)
Trace decorator for functions.

Very useful if you would like to add trace point on existing function:

>> @profiler.trace("my_point") >> def my_func(self, some_args): >> #code

Parameters
  • name -- The name of action. E.g. wsgi, rpc, db, etc..
  • info -- Dictionary with extra trace information. For example in wsgi it can be url, in rpc - message or in db sql - request.
  • hide_args -- Don't push to trace info args and kwargs. Quite useful if you have some info in args that you wont to share, e.g. passwords.
  • hide_result -- Boolean value to hide/show function result in trace. True - hide function result (default). False - show function result in trace.
  • allow_multiple_trace -- If the wrapped function has already been traced either allow the new trace to occur or raise a value error denoting that multiple tracing is not allowed (by default allow).



osprofiler.profiler.trace_cls(name, info=None, hide_args=False, hide_result=True, trace_private=False, allow_multiple_trace=True, trace_class_methods=False, trace_static_methods=False)
Trace decorator for instances of class .

Very useful if you would like to add trace point on existing method:

>> @profiler.trace_cls("rpc") >> RpcManagerClass(object): >> >> def my_method(self, some_args): >> pass >> >> def my_method2(self, some_arg1, some_arg2, kw=None, kw2=None) >> pass >>

Parameters
  • name -- The name of action. E.g. wsgi, rpc, db, etc..
  • info -- Dictionary with extra trace information. For example in wsgi it can be url, in rpc - message or in db sql - request.
  • hide_args -- Don't push to trace info args and kwargs. Quite useful if you have some info in args that you wont to share, e.g. passwords.
  • hide_result -- Boolean value to hide/show function result in trace. True - hide function result (default). False - show function result in trace.
  • trace_private -- Trace methods that starts with "_". It wont trace methods that starts "__" even if it is turned on.
  • trace_static_methods -- Trace staticmethods. This may be prone to issues so careful usage is recommended (this is also why this defaults to false).
  • trace_class_methods -- Trace classmethods. This may be prone to issues so careful usage is recommended (this is also why this defaults to false).
  • allow_multiple_trace -- If wrapped attributes have already been traced either allow the new trace to occur or raise a value error denoting that multiple tracing is not allowed (by default allow).



osprofiler.requests module

osprofiler.requests.enable()

osprofiler.requests.send(self, request, *args, **kwargs)

osprofiler.sqlalchemy module

osprofiler.sqlalchemy.add_tracing(sqlalchemy, engine, name, hide_result=True)
Add tracing to all sqlalchemy calls.

osprofiler.sqlalchemy.disable()
Disable tracing of all DB queries. Reduce a lot size of profiles.

osprofiler.sqlalchemy.enable()
add_tracing adds event listeners for sqlalchemy.

osprofiler.sqlalchemy.handle_error(exception_context)
Handle SQLAlchemy errors

osprofiler.sqlalchemy.wrap_session(sqlalchemy, sess)

osprofiler.web module

class osprofiler.web.WsgiMiddleware(application, hmac_keys=None, enabled=False, **kwargs)
Bases: object

WSGI Middleware that enables tracing for an application.

classmethod factory(global_conf, **local_conf)


osprofiler.web.X_TRACE_HMAC = 'X-Trace-HMAC'
Http header that will contain the traces data hmac (that will be validated).

osprofiler.web.X_TRACE_INFO = 'X-Trace-Info'
Http header that will contain the needed traces data.

osprofiler.web.disable()
Disable middleware.

This is the alternative way to disable middleware. It will be used to be able to disable middleware via oslo.config.


osprofiler.web.enable(hmac_keys=None)
Enable middleware.

osprofiler.web.get_trace_id_headers()
Adds the trace id headers (and any hmac) into provided dictionary.

Module contents

Indices and tables

  • Index
  • Module Index
  • Search Page

AUTHOR

Author name not set

COPYRIGHT

2016, OpenStack Foundation

June 29, 2025 4.3.0