stevedore(1)
| STEVEDORE(1) | stevedore | STEVEDORE(1) |
NAME
stevedore - stevedore 5.5.0
Python makes loading code dynamically easy, allowing you to configure and extend your application by discovering and loading extensions ("plugins") at runtime. Many applications implement their own library for doing this, using __import__ or importlib. stevedore avoids creating yet another extension mechanism by building on top of entry points. The code for managing entry points tends to be repetitive, though, so stevedore provides manager classes for implementing common patterns for using dynamically loaded extensions.
STEVEDORE USER GUIDE
Patterns for Loading
Setuptools entry points are registered as within a namespace that defines the API expected by the plugin code. Each entry point has a name, which does not have to be unique within a given namespace. The flexibility of this name management system makes it possible to use plugins in a variety of ways. The manager classes in stevedore wrap importlib.metadata to apply different rules matching the patterns described here.
Drivers -- Single Name, Single Entry Point
Specifying a driver for communicating with an external resource (database, device, or remote application) is perhaps the most common use of dynamically loaded libraries. Drivers support the abstracted view of the resource so an application can work with different types of resources. For example, drivers may connect to database engines, load different file formats, or communicate with similar web services from different providers. Many drivers may be available for a given application, but it is implied in the interface between the application and the driver that only one driver will be used to manage a given resource. [graph]
Examples of the drivers pattern include:
- database client libraries used by SQLAlchemy
- cloud vendor API clients used by libcloud
SEE ALSO:
Hooks -- Single Name, Many Entry Points
Hooks, signals, or callbacks are invoked based on an event occurring within an application. All of the hooks for an application may share a single namespace (e.g., my.application.hooks) and use a different name for the triggered event (e.g., startup and precommit). Multiple entry points can share the same name within the namespace, so that multiple hooks can be invoked when an event occurs. [graph]
Examples of the hooks pattern include:
- Emacs mode hook functions
- Django signals
SEE ALSO:
Extensions -- Many Names, Many Entry Points
The more general form of extending an application is to load additional functionality by discovering add-on modules that use a minimal API to inject themselves at runtime. Extensions typically want to be notified that they have been loaded and are being used so they can perform initialization or setup steps. An extension may replace core functionality or add to it. [graph]
Examples of the extensions pattern include:
- Django apps
- Sphinx extensions
- Trac Plugins
SEE ALSO:
Patterns for Enabling
The entry point registry maintained by setuptools lists the available plugins, but does not provide a way for the end-user to control which are used or enabled. The common patterns for managing the set of extensions to be used are described below.
Enabled Through Installation
For many applications, simply installing an extension is enough of an indication that the extension should be used. No explicit configuration is required on the part of the user to either discover or enable the extension, since its entry point can be discovered when all of the plugins are loaded at runtime.
Examples of enabling through installation include:
- python-openstackclient
- virtualenvwrapper
Enabled Explicitly
In other cases, the extensions may be installed system-wide but should not all be enabled for a given application or instance of an application. In these situations, the person deploying or using the application will want to select the extensions to be used through an explicit configuration step.
Examples of explicitly enabled extensions include:
- Django apps
- Sphinx extensions
- Trac Plugins
SEE ALSO:
Self-Enabled
Finally, some applications ask their extensions whether they should be enabled. The extension may look at other libraries installed on the system, check an external configuration setting, or examine a resource to see if it can be managed by the plugin. These checks are usually at runtime, either when the extension is loaded or when the user tries to access a specific resource.
Examples of self-enabled extensions include:
- anydbm
- PIL
SEE ALSO:
Using Stevedore in Your Application
This tutorial is a step-by-step walk-through demonstrating how to define plugins and then use stevedore to load and use them in your application.
Guidelines for Naming Plugins
Stevedore uses setuptools entry points to define and load plugins. An entry point is standard way to refer to a named object defined inside a Python module or package. The name can be a reference to any class, function, or instance, as long as it is created when the containing module is imported (i.e., it needs to be a module-level global).
Names and Namespaces
Entry points are registered using a name in a namespace.
Entry point names are usually considered user-visible. For example, they frequently appear in configuration files where a driver is being enabled. Because they are public, names are typically as short as possible while remaining descriptive. For example, database driver plugin names might be "mysql", "postgresql", "sqlite", etc.
Namespaces, on the other hand, are an implementation detail, and while they are known to developers they are not usually exposed to users. The namespace naming syntax looks a lot like Python's package syntax (a.b.c) but namespaces do not correspond to Python packages. Using a Python package name for an entry point namespace is an easy way to ensure a unique name, but it's not required at all. The main feature of entry points is that they can be discovered across packages. That means that a plugin can be developed and installed completely separately from the application that uses it, as long as they agree on the namespace and API.
Each namespace is owned by the code that consumes the plugins and is used to search for entry points. The entry point names are typically owned by the plugin, but they can also be defined by the consuming code for named hooks (see HookManager). The names of entry points must be unique within a given distribution, but are not necessarily unique in a namespace.
Creating Plugins
After a lot of trial and error, the easiest way I have found to define an API is to follow these steps:
- 1.
- Use the abc module to create a base abstract class to define the behaviors required of plugins of the API. Developers don't have to subclass from the base class, but it provides a convenient way to document the API, and using an abstract base class keeps you honest.
- 2.
- Create plugins by subclassing the base class and implementing the required methods.
- 3.
- Define a unique namespace for each API by combining the name of the application (or library) and a name of the API. Keep it shallow. For example, "cliff.formatters" or "ceilometer.pollsters.compute".
Example Plugin Set
The example program in this tutorial will create a plugin set with several data formatters, like what might be used by a command line program to prepare data to be printed to the console. Each formatter will take as input a dictionary with string keys and built-in data types as values. It will return as output an iterator that produces the string with the data structure formatted based on the rules of the specific formatter being used. The formatter's constructor lets the caller specify the maximum width the output should have.
A Plugin Base Class
Step 1 above is to define an abstract base class for the API that needs to be implemented by each plugin.
# stevedore/example/base.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import abc class FormatterBase(metaclass=abc.ABCMeta):
"""Base class for example plugin used in the tutorial.
"""
def __init__(self, max_width=60):
self.max_width = max_width
@abc.abstractmethod
def format(self, data):
"""Format the data and return unicode text.
:param data: A dictionary with string keys and simple types as
values.
:type data: dict(str:?)
:returns: Iterable producing the formatted text.
"""
The constructor is a concrete method because subclasses do not need to override it, but the format() method does not do anything useful because there is no "default" implementation available.
Concrete Plugins
The next step is to create a couple of plugin classes with concrete implementations of format(). A simple example formatter produces output with each variable name and value on a single line.
# stevedore/example/simple.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); you may # not use this file except in compliance with the License. You may obtain # a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. from stevedore.example import base class Simple(base.FormatterBase):
"""A very basic formatter."""
def format(self, data):
"""Format the data and return unicode text.
:param data: A dictionary with string keys and simple types as
values.
:type data: dict(str:?)
"""
for name, value in sorted(data.items()):
line = '{name} = {value}\n'.format(
name=name,
value=value,
)
yield line
There are plenty of other formatting options, but this example will give us enough to work with to demonstrate registering and using plugins.
Registering the Plugins
To use setuptools entry points, you must package your application or library using setuptools. The build and packaging process generates metadata which is available after installation to find the plugins provided by each python distribution.
The entry points must be declared as belonging to a specific namespace, so we need to pick one before going any further. These plugins are formatters from the stevedore examples, so I will use the namespace "stevedore.example.formatter". Now it is possible to provide all of the necessary information in the packaging instructions:
# stevedore/example/setup.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from setuptools import find_packages from setuptools import setup setup(
name='stevedore-examples',
version='1.0',
description='Demonstration package for stevedore',
author='Doug Hellmann',
author_email='doug@doughellmann.com',
url='http://opendev.org/openstack/stevedore',
classifiers=['Development Status :: 3 - Alpha',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Intended Audience :: Developers',
'Environment :: Console',
],
platforms=['Any'],
scripts=[],
provides=['stevedore.examples',
],
packages=find_packages(),
include_package_data=True,
entry_points={
'stevedore.example.formatter': [
'simple = stevedore.example.simple:Simple',
'plain = stevedore.example.simple:Simple',
],
},
zip_safe=False, )
The important lines are near the bottom where the entry_points argument to setup() is set. The value is a dictionary mapping the namespace for the plugins to a list of their definitions. Each item in the list should be a string with name = module:importable where name is the user-visible name for the plugin, module is the Python import reference for the module, and importable is the name of something that can be imported from inside the module.
'Environment :: Console',
],
platforms=['Any'],
scripts=[],
In this case, there are two plugins registered. The "simple" plugin defined above, and a "plain" plugin, which is just an alias for the simple plugin.
setuptools Metadata
During the build, setuptools copies entry point definitions to a file in the ".egg-info" directory for the package. For example, the file for stevedore is located in stevedore.egg-info/entry_points.txt:
[stevedore.example.formatter] simple = stevedore.example.simple:Simple plain = stevedore.example.simple:Simple [stevedore.test.extension] t2 = stevedore.tests.test_extension:FauxExtension t1 = stevedore.tests.test_extension:FauxExtension
importlib.metadata uses the entry_points.txt file from all of the installed packages on the import path to find plugins. You should not modify these files, except by changing the list of entry points in setup.py.
Adding Plugins in Other Packages
Part of the appeal of using entry points for plugins is that they can be distributed independently of an application. The namespace setuptools uses to find the plugins is different from the Python source code namespace. It is common to use a plugin namespace prefixed with the name of the application or library that loads the plugins, to ensure it is unique, but that name has no bearing on what Python package the code for the plugin should live in.
For example, we can add an alternate implementation of a formatter plugin that produces a reStructuredText field list.
# stevedore/example2/fields.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import textwrap from stevedore.example import base class FieldList(base.FormatterBase):
"""Format values as a reStructuredText field list.
For example::
: name1 : value
: name2 : value
: name3 : a long value
will be wrapped with
a hanging indent
"""
def format(self, data):
"""Format the data and return unicode text.
:param data: A dictionary with string keys and simple types as
values.
:type data: dict(str:?)
"""
for name, value in sorted(data.items()):
full_text = ': {name} : {value}'.format(
name=name,
value=value,
)
wrapped_text = textwrap.fill(
full_text,
initial_indent='',
subsequent_indent=' ',
width=self.max_width,
)
yield wrapped_text + '\n'
The new plugin can then be packaged using a setup.py containing
# stevedore/example2/setup.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. from setuptools import find_packages from setuptools import setup setup(
name='stevedore-examples2',
version='1.0',
description='Demonstration package for stevedore',
author='Doug Hellmann',
author_email='doug@doughellmann.com',
url='http://opendev.org/openstack/stevedore',
classifiers=['Development Status :: 3 - Alpha',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Intended Audience :: Developers',
'Environment :: Console',
],
platforms=['Any'],
scripts=[],
provides=['stevedore.examples2',
],
packages=find_packages(),
include_package_data=True,
entry_points={
'stevedore.example.formatter': [
'field = stevedore.example2.fields:FieldList',
],
},
zip_safe=False, )
The new plugin is in a separate stevedore-examples2 package.
# Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License.
However, the plugin is registered as part of the stevedore.example.formatter namespace.
'Intended Audience :: Developers',
'Environment :: Console',
],
platforms=['Any'],
When the plugin namespace is scanned, all packages on the current PYTHONPATH are examined and the entry point from the second package is found and can be loaded without the application having to know where the plugin is actually installed.
Loading the Plugins
There are several different enabling and invocation patterns for consumers of plugins, depending on your needs.
Loading Drivers
The most common way plugins are used is as individual drivers. In this case, there may be many plugin options to choose from, but only one needs to be loaded and called. The DriverManager class supports this pattern.
This example program uses a DriverManager to load a formatter defined in the examples for stevedore. It then uses the formatter to convert a data structure to a text format, which it can print.
# stevedore/example/load_as_driver.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import argparse from stevedore import driver if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument(
'format',
nargs='?',
default='simple',
help='the output format',
)
parser.add_argument(
'--width',
default=60,
type=int,
help='maximum output width for text',
)
parsed_args = parser.parse_args()
data = {
'a': 'A',
'b': 'B',
'long': 'word ' * 80,
}
mgr = driver.DriverManager(
namespace='stevedore.example.formatter',
name=parsed_args.format,
invoke_on_load=True,
invoke_args=(parsed_args.width,),
)
for chunk in mgr.driver.format(data):
print(chunk, end='')
The manager takes the plugin namespace and name as arguments, and uses them to find the plugin. Then, because invoke_on_load is true, it calls the object loaded. In this case that object is the plugin class registered as a formatter. The invoke_args are positional arguments passed to the class constructor, and are used to set the maximum width parameter.
default=60,
type=int,
help='maximum output width for text',
)
parsed_args = parser.parse_args()
After the manager is created, it holds a reference to a single object returned by calling the code registered for the plugin. That object is the actual driver, in this case an instance of the formatter class from the plugin. The single driver can be accessed via the driver property of the manager, and then its methods can be called directly.
parsed_args = parser.parse_args()
Running the example program produces this output:
$ python -m stevedore.example.load_as_driver a = A b = B long = word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word $ python -m stevedore.example.load_as_driver field : a : A : b : B : long : word word word word word word word word word word
word word word word word word word word word word word
word word word word word word word word word word word
word word word word word word word word word word word
word word word word word word word word word word word
word word word word word word word word word word word
word word word word word word word word word word word
word word word word $ python -m stevedore.example.load_as_driver field --width 30 : a : A : b : B : long : word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word
Loading Extensions
Another common use case is to load several extensions at one time, and do something with all of them. Several of the other manager classes support this invocation pattern, including ExtensionManager, NamedExtensionManager, and EnabledExtensionManager.
# stevedore/example/load_as_extension.py # Copyright (C) 2020 Red Hat, Inc. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or # implied. # See the License for the specific language governing permissions and # limitations under the License. import argparse from stevedore import extension if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument(
'--width',
default=60,
type=int,
help='maximum output width for text',
)
parsed_args = parser.parse_args()
data = {
'a': 'A',
'b': 'B',
'long': 'word ' * 80,
}
mgr = extension.ExtensionManager(
namespace='stevedore.example.formatter',
invoke_on_load=True,
invoke_args=(parsed_args.width,),
)
def format_data(ext, data):
return (ext.name, ext.obj.format(data))
results = mgr.map(format_data, data)
for name, result in results:
print('Formatter: {}'.format(name))
for chunk in result:
print(chunk, end='')
print('')
The ExtensionManager is created slightly differently from the DriverManager because it does not need to know in advance which plugin to load. It loads all of the plugins it finds.
default=60,
type=int,
help='maximum output width for text',
)
parsed_args = parser.parse_args()
To call the plugins, use the map() method, passing a callable to be invoked for each extension. The format_data() function used with map() in this example takes two arguments, the Extension and the data argument given to map().
data = {
'a': 'A',
'b': 'B',
'long': 'word ' * 80,
The Extension passed format_data() is a class defined by stevedore that wraps the plugin. It includes the name of the plugin, the EntryPoint returned by importlib.metadata, and the plugin itself (the named object referenced by the plugin definition). When invoke_on_load is true, the Extension will also have an obj attribute containing the value returned when the plugin was invoked.
map() returns a sequence of the values returned by the callback function. In this case, format_data() returns a tuple containing the extension name and the iterable that produces the text to print. As the results are processed, the name of each plugin is printed and then the formatted data.
'long': 'word ' * 80,
}
mgr = extension.ExtensionManager(
namespace='stevedore.example.formatter',
The order the plugins are loaded is undefined, and depends on the order packages are found on the import path as well as the way the metadata files are read. If the order extensions are used matters, try the NamedExtensionManager.
$ python -m stevedore.example.load_as_extension --width 30 Formatter: simple a = A b = B long = word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word Formatter: field : a : A : b : B : long : word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word word word word word
word Formatter: plain a = A b = B long = word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word word
Why Not Call Plugins Directly?
Using a separate callable argument to map(), rather than just invoking the plugin directly introduces a separation between your application code and the plugins. The benefits of this separation manifest in the application code design and in the plugin API design.
If map() called the plugin directly, each plugin would have to be a callable. That would mean a separate namespace for what is really just a method of the plugin. By using a separate callable argument, the plugin API does not need to match exactly any particular use case in the application. This frees you to create a finer-grained API, with more individual methods that can be called in different ways to achieve different goals.
SEE ALSO:
- Patterns for Loading
- Patterns for Enabling
Testing
SEE ALSO:
- •
- TestExtensionManager
SEE ALSO:
- Dynamic Code Patterns: Extending Your Applications with Plugins
- Using setuptools entry points
- Package Discovery and Resource Access using pkg_resources
- Using Entry Points to Write Plugins | Pylons
- importlib.metadata
Sphinx Integration
Stevedore includes an extension for integrating with Sphinx to automatically produce documentation about the supported plugins. To activate the plugin add stevedore.sphinxext to the list of extensions in your conf.py.
- .. list-plugins:: namespace
- List the plugins in a namespace.
Options:
- detailed
- Flag to switch between simple and detailed output (see below).
- overline-style
- Character to use to draw line above header, defaults to none.
- underline-style
- Character to use to draw line below header, defaults to =.
Simple List
By default, the list-plugins directive produces a simple list of plugins in a given namespace including the name and the first line of the docstring. For example:
.. list-plugins:: stevedore.example.formatter
produces
----
- field -- Format values as a reStructuredText field list.
- plain -- A very basic formatter.
- simple -- A very basic formatter.
----
Detailed Lists
Adding the detailed flag to the directive causes the output to include a separate subsection for each plugin, with the full docstring rendered. The section heading level can be controlled using the underline-style and overline-style options to fit the results into the structure of your existing document.
.. list-plugins:: stevedore.example.formatter
:detailed:
produces
----
field
Format values as a reStructuredText field list.
For example:
: name1 : value : name2 : value : name3 : a long value
will be wrapped with
a hanging indent
plain
A very basic formatter.
simple
A very basic formatter.
----
NOTE:
Dynamic Code Patterns: Extending Your Applications with Plugins
Over the past few years I have been doing a lot of work on applications that make heavy use of Python’s ability to load code dynamically at runtime. This essay includes the results of some focused research that I did into patterns for using dynamic code, and the impact that research had on the design of stevedore and ceilometer, an application that uses it
For my analysis, I counted any application or framework that loads code dynamically at runtime to be using plugins. I did not consider delayed execution of hard-coded import statements as plugins, but restricted the research to true dynamic loading. In most cases, the name or location of the code is given through some external mechanism like a configuration file.
Why Use Plugins?
Before examining patterns for using plugins, we should talk about why to use plugins in an application at all.
One important benefit is improved design. Keeping a separation between core and extension code encourages you to think more about abstractions in your design. Building an extensible system can take more work than hardwiring everything, but the results tend to be more flexible and maintainable over the long term.
Plugins are a good way to implement device drivers and other versions of the Strategy pattern. The application can maintain generic core logic, and the plugin can handle the details for interfacing with an outside system or device.
Packaging extensions separately reduces dependency bloat and makes deployment easier to manage and install. Users who do not need some drivers or features can avoid deploying dependencies that are only used by those plugins.
Plugins also provide a convenient way to extend the feature set of an application by hooking new code into well-defined extension points. And having such an extensible system makes it easier for other developers to contribute to your project indirectly by providing add-on packages that are released separately.
Requirements for Ceilometer
I have spent a fair bit of time studying plugin-based architectures over the last year while helping to create ceilometer, the new metering component in OpenStack. Ceilometer measures the resources being used in a cloud deployment so we can bill the tenants for those resources. We collect data like the lifetimes of servers, along with bandwidth and storage consumed.
However, the type and number of things a given cloud deployer will want to charge for will vary, so we wanted a flexible system for taking those measurements. We need to allow deployers to write their own plugins to measure things we haven’t thought of, or that may need to be measured in a way that is private to their configuration (a case we have at DreamHost).
We are expecting a lot of developers who don’t interact with us directly to be trying to write extensions to ceilometer for use in private cloud deployments, so it is also important to clearly document how to create new plugins.
With these things in mind, we designed ceilometer to be flexible in several different areas. [image]
OpenStack is a collection of components that cooperate to provide Infrastructure as a Service features. Each component manages a different aspect of the cloud and uses a message bus to communicate with the other components.
All of the components generate notification messages when events happen (like instances being created or destroyed). Capturing those messages was the first source of data for ceilometer. The notifications contain different metadata depending on the resource that triggered the event, so we needed plugins to translate the notification messages into a standard format for metering.
There aren’t events for all of the things we want to measure for billing, so we also had to create some agents to poll for data. For example, we want to check periodically to see how much CPU capacity each instance has consumed. Some pollsters run on the hypervisor server, and others run on a management server where they can communicate with the other OpenStack components via their APIs.
All of the ceilometer services use another message bus to deliver data to a collector process which uses a storage driver to write data to a database. We support relational and non-relational databases, depending on the choice of the deployer.
This architecture resulted in five sets of plugins for ceilometer. OpenStack includes a message bus abstraction layer and a set of drivers for using RabbitMQ, Qpid, and ZMQ. Since that was already implemented for us, we didn’t have to touch it.
The other 4 sets we created from scratch:
- The plugins for processing notification messages
- The pollsters for the compute nodes
- The central pollsters
- And the storage driver
The resulting designs use patterns found in other applications and frameworks that use plugins.
Other Plugin-based Applications
During my research, I looked at a few projects that I was already familiar with, either as a user or a developer, and some I had not used before. There are plenty of other examples, but this list was long enough to identify some common patterns and help us with our design.
Blogofile and Sphinx are two apps for working with different forms of text for publishing. They use extensions to add new content processing features.
Mercurial is a command line app that can be extended with new subcommands. Cliff is a library I created for building apps like Mercurial.
Virtualenvwrapper is a command line tool that uses hooks in a different way, to extend existing commands but not necessarily add new ones.
Nose and Trac are common developer tools. You’re more likely to have used them than written extensions for them, but they do both use plugins.
Django, Pyramid, and SQLAlchemy are developer libraries that use plugins.
Diamond is a monitoring app with an extensive plugin set, similar to the system we were planning to build for OpenStack.
Nova is the primary component of the OpenStack cloud system. It relies on a large number of drivers for managing different aspects of the computing environment.
I looked at all of this code in an effort to derive ideas about the "right way" to handle plugins for ceilometer. While some of what I will say today may sound critical of the choices made by the developers of the other code, I do have the benefit of hindsight and a different perspective based on looking at the examples all together, as well as different requirements for our project.
Discovery
The first thing an application has to do with a plugin is find it. The tools I looked at are split between some form of explicit definition of plugins and a scanner that looked for the plugins. [image]
Each of those sets is then further divided between what was being listed or scanned -- files on the filesystem, or python import references (either a module, or something inside of a module).
The "Explicit import reference" category means there is a configuration file somewhere and a user lists an importable object in that file.
The "Scan import reference" category means a registry of import strings is being scanned. All of these examples use setuptools and pkg_resources to manage entry points.
Enabling
After the app finds a plugin, the next step is to decide whether to load it and use it. Most applications require an explicit step to configure extensions. There are times when this makes sense. Developer tools like Django are right to ask the developer to list the desired extensions explicitly, since you’re really bringing that code in statically. The extensions to SQLAlchemy are all enabled, but only one is really used at a time and that is chosen by the database connection string. [image]
However, some of the user applications like Blogofile, Mercurial, and Trac ask the user to explicitly enable extensions through a configuration step that seems like it could be skipped. When I created virtualenvwrapper and cliff, I decided to use installation as a trigger for activation because I wanted to avoid an extra opportunity for misconfiguration. In both of those cases, installing an extension makes it available, so the user can start taking advantage of it immediately.
That’s also true for Nose extensions, although whether or not they are used for a given test suite or test run depends on the options you give nose.
Importing
After the app decides whether to load a plugin, the next step is to actually get the code. All of the examples I looked at used two techniques, either calling import explicitly (by using the builtin function, the imp module, or some other variation), or using pkg_resources. [image]
Nose, SQLAlchemy, and Blogofile all use both techniques. Nose falls back to a custom importer if pkg_resources is not installed. SQLAlchemy uses a custom importer for "extensions" distributed with the core but pkg_resources to find separate packages. Blogofile uses pkg_resources to find plugins, coupled with manual scanning and importing of the directories containing those plugins to load their parts.
If I discount the packages I created myself, shown here in italics, there seems to be a clear bias towards creating custom wrappers around import. That route seems easy at first, but all of the implementations I found exhibited some problems with tricky corner cases.
Application/Plugin Integration
After the code for the extension is imported, the next step is to integrate it with the rest of the app. That is, to configure any hooks that need to call into the plugin, pass the plugin any state it needs, etc. I looked at this step along two axes. [image]
First, I considered the granularity of the plugin interface. For "fine" grained plugins, the extension is treated as a standalone object to be called on as needed. In these cases, the code object being loaded is usually a function or a class.
For more "coarse" grained cases, a single plugin will include hooks that are referenced from multiple places in the application. There may be several classes inside the plugin, for example, or templates that are accessed directly by the application, not through a plugin API.
The other axis related to integration looks at how the code provided by the plugin is brought into the application. I found two techniques for doing that.
First, the application can instruct the plugin to integrate itself. That prompting usually takes the form of a setup() or initialization function implemented by the plugin author that calls back to an application context object, registering parts of the plugin explicitly. That registration could also be handled implicitly using an interface library such as the way Trac uses zope.interface.
Second, the application itself can interrogate or inspect the plugin, and make decisions based on the result. This usually means that part of the plugin API is responsible for providing metadata about the plugin itself, not just taking action.
API Enforcement
One common issue with dynamically loaded code is enforcing the plugin API at runtime. This is always a potential issue in dynamic languages, but it comes up frequently with plugins because the code is often written by someone other than the core developer for the application. I saw two basic techniques to help developers get their plugins right: convention and interfaces. [image]
Many of the applications that used convention also had coarse-grained plugin APIs, and so while they may use classes to provide their features, the plugin relies on convention for discovering its configuration.
On the right are applications for which the plugin uses a class hierarchy. In the case of Nose, using the base class is optional, so that’s a quasi-interface. Trac, on the other hand, uses formal interfaces through zope.interface.
Diamond enforces a strict subclassing of its Collector base class.
For cliff I chose to use the abc module to define an abstract base class, but stick with "duck typing" in the actual application. The developer doesn’t have to inherit from the base class, but doing so helps ensure that the implementation is complete.
Invocation
And the final dimension I looked at was how the plugin code was used at runtime. There were three primary patterns here. [image]
- 1.
- "Drivers" are loaded one at a time, and used directly.
- 2.
- The apps using the "Dispatcher" pattern load all of the extensions, and then make calls to the appropriate one based on name or some other selection criteria when an event happens.
- 3.
- The apps that use the "Iterator" pattern call each extension in turn, so that all of the plugins have a chance to participate in the processing.
Ceilometer Design
This analysis had a direct influence on the choices we made while implementing ceilometer.
Discovery and Import
For finding and loading, we chose to use entry points because they were the simplest solution. All of the apps that work on files instead of import references had issues, ranging from poorly implemented import path munging to packaging and distribution challenges. Even some of the code for working with import references directly was a little hairy. Leaving that to a library that handles the different cases transparently made our life a lot easier.
They are easier for users to install and configure because they don’t have to understand how your code is laid out. That also makes them more resilient in the face of code changes.
Entry points also support different package formats (egg, sdist, operating system packages), so it doesn’t matter how extensions are distributed.
They also make it easier for code to be packaged by the Linux distributions, since the packages don’t have to share overlapping installation directories.
There are alternate implementations of entry-point like systems, but none are so widely used or tested as pkg_resources.
And to further simplify, we always use entry points, even for the plugins we distribute with our core. That eliminates any special cases.
Enabling
We came up with a somewhat novel solution to manage which plugins are enabled. For Ceilometer we wanted to default to collecting data, but allow deployers to disable certain meters to save storage space if they knew they did not need the data. The solution was to use explicit configuration, but invert it from the normal implementation.
We assume that all of the extensions found should be loaded and used, unless they are explicitly disabled in the configuration file. We did that in the first version to simplify the configuration process, because we assumed most users would want most of the plugins to be used. Defaulting to enabled means users only have to provide a short list of the meters to turn off.
Ceilometer plugins also have a chance, when they are being loaded, to disable themselves automatically. This is especially useful in the polling plugins, which can tell the app to ignore them if the resource they use for collecting measurements is not present (like they work with a different hypervisor, or external service that is not configured).
Letting the plugin disable itself avoids repeated warning messages in the log file as a plugin is asked to poll for data that it cannot retrieve.
Integration
For our integration pattern we went with a fine-grained API using inspection. There is a separate namespace for each type of plugin, and each plugin instance refers to a single class. The application loads and instantiates that class, then calls methods on it it to determine what it provides and wants (which notifications to subscribe to and which meters are produced).
This design lets us avoid having repetitious setup or configuration code in each plugin, since they provide data to the application on demand and the application configures itself. The instances don’t know about the application, or each other. They only run when the application calls them, never independently.
API Enforcement
To define the API for each set of plugins we created a separate abstract base class using the abc module. This gives us a way to document each plugin API and Developers who use the base class get some help for free.
Since we don’t enforce the class hierarchy we also watch for unexpected errors from the plugins any time we call into them.
Invocation
We used all three invocation patterns, in different places.
- 1.
- We only use one storage system at a time, so we treat the storage plugin like a driver.
- 2.
- We load all of the notification plugins, and then dispatch incoming messages to them based on the message content.
- 3.
- We load all of the polling plugins and iterate through them on a regular schedule.
Conclusions
After we had all of this working in ceilometer, I extracted some of the code into stevedore. It wraps pkg_resources with a series of manager classes that implement the loading, enabling, and invocation patterns.
SEE ALSO:
- PyCon 2013 Video: http://pyvideo.org/video/1789/dynamic-code-patterns-extending-your-application
- Slides: http://www.slideshare.net/doughellmann/dynamic-codepatterns
- Patterns for Loading
- Patterns for Enabling
ChangeLog
CHANGES
5.5.0
- add pyproject.toml to support pip 23.1
- tox: Remove basepython
- Update master for stable/2025.1
5.4.1
- Skip installation to speed up pep8
- reno: Update master for unmaintained/2023.1
5.4.0
- Remove explicit pbr dependency
- Add note about requirements lower bounds
- Remove Python 3.8 support
- Run pyupgrade to clean up Python 2 syntaxes
- Declare Python 3.12 support
- Update master for stable/2024.2
5.3.0
- reno: Update master for unmaintained/zed
- Remove old excludes
- Update master for stable/2024.1
- reno: Update master for unmaintained/xena
- reno: Update master for unmaintained/wallaby
- reno: Update master for unmaintained/victoria
5.2.0
- pre-commit: Integrate bandit
- pre-commit: Bump versions
- reno: Update master for unmaintained/yoga
- Bump hacking
- Update python classifier in setup.cfg
- Update master for stable/2023.2
5.1.0
- Revert "Moves supported python runtimes from version 3.8 to 3.10"
- Moves supported python runtimes from version 3.8 to 3.10
- Update master for stable/2023.1
5.0.0
4.1.1
- •
- Order old importlib-metadata results by group
4.1.0
- Remove Extension.extras
- Fix compatibility with Python 3.12, importlib-metadata 5.0
- Fix compatibility with Python 3.10, 3.9.11
- Catch NotADirectoryError error
- Add Python3 antelope unit tests
- Update master for stable/zed
- remove unicode from code
4.0.0
- Fix remaining logic to support Python 3.6/7
- Drop python3.6/3.7 support in testing runtime
- Add Python3 zed unit tests
- Update master for stable/yoga
3.5.0
- Add Python3 yoga unit tests
- Update master for stable/xena
- Rely on member access, the preferred access since importlib_metadata 4.8
3.4.0
- setup.cfg: Replace dashes with underscores
- Fix formatting of release list
- Remove lower-constraints remnants
- Move flake8 as a pre-commit local target
- Add Python3 xena unit tests
- Update master for stable/wallaby
- Dropping lower constraints testing
3.3.0
- Use TOX_CONSTRAINTS_FILE
- Use py3 as the default runtime for tox
- Adding pre-commit
- Fix cache dir flooding when running from /tmp
- Add Python3 wallaby unit tests
- Update master for stable/victoria
3.2.2
- •
- fix supported python versions in documentation
3.2.1
- •
- Fix the bug 1892610. There're some syntax errors in the comment of stevedore code
3.2.0
- •
- add property methods to extension for more entry point values
3.1.0
- •
- sphinxext: fix warning message for detailed list
3.0.0
- add release note before major version update
- switch to importlib.metadata package
2.0.1
- Remove Travis CI config
- Replace external mock with built-in unittest.mock
2.0.0
- Stop to use the __future__ module
- Switch to newer openstackdocstheme and reno versions
- Add Python3 victoria unit tests
- Mark sphinx extensions thread safe
- Remove dead files
- Drop Python 2.7 support
- Update master for stable/ussuri
1.32.0
- Switch to Ussuri jobs
- Blacklist sphinx 2.1.0 (autodoc bug)
- Update the constraints url
- Update master for stable/train
1.31.0
- Add Python 3 Train unit tests
- Add local bindep.txt
- Cap Bandit below 1.6.0 and update Sphinx requirement
- update git.openstack.org to opendev
- OpenDev Migration Patch
- Dropping the py35 testing
- Update master for stable/stein
- Delete repeated param description
- add python 3.7 unit test job
1.30.1
- Use template for lower-constraints
- Change openstack-dev to openstack-discuss
1.30.0
- Update sphinx logging to not use app object
- Removed older version of python added 3.5
- Update doc/conf.py to avoid warnings with sphinx 1.8
- add lib-forward-testing-python3 test job
- fix wrong link
- add python 3.6 unit test job
- import zuul job settings from project-config
- Update reno for stable/rocky
1.29.0
- Remove unnecessary py27 testenv
- Switch to stestr
- fix tox python3 overrides
- Trivial: Update pypi url to new url
- Trivial: Update pypi url to new url
- set default python to python3
- add lower-constraints job
- Updated from global requirements
- Update links in README
- Update reno for stable/queens
- Updated from global requirements
- Updated from global requirements
- Follow the new PTI for document build
1.28.0
- Updated from global requirements
- Remove -U from pip install
- Avoid tox_install.sh for constraints support
- add bandit to pep8 job
- move doc requirements to doc/requirements.txt
- Remove setting of version/release from releasenotes
- Updated from global requirements
1.27.1
- Move reno to optional docs requirements
- Remove duplicate optional requirement
1.27.0
- •
- Updated from global requirements
1.26.0
- Updated from global requirements
- Remove Pillow from test-requirements
- Make openstackdocstheme an optional doc dependency
- Updated from global requirements
- Add an ExtensionManager.items() method
- Update reno for stable/pike
- Updated from global requirements
1.25.0
- •
- Update URLs in documents according to document migration
1.24.0
- switch from oslosphinx to openstackdocstheme
- turn on warning-is-error for doc build
- move documentation into the new standard layout
- Updated from global requirements
- fix setuptools url
1.23.0
- Updated from global requirements
- Remove 'run_sphinx' script
- Remove unused doc/requirements.txt
- Mark as Production/Stable instead of Alpha
1.22.0
- Remove oslotest from test-requirements
- Updated from global requirements
- Updated from global requirements
- Updated from global requirements
- Update reno for stable/ocata
1.21.0
- Updated from global requirements
- Updated from global requirements
- Updated from global requirements
- Remove support for py34
1.20.0
- extension: add entry_point_names method
- extension: expose _find_entry_points as list_entry_points
- Allow suppression of warnings from DriverManager
- Show team and repo badges on README
- Add Constraints support
1.19.1
- •
- Broken link at stevedore developer documentation
1.19.0
- Add Apache 2.0 license to source file
- Updated from global requirements
- Add reno for release notes management
- Updated from global requirements
- Remove reference to non-existing page
1.18.0
- Updated from global requirements
- Fix typos in exception.py
1.17.1
- •
- do not emit warnings for missing hooks
1.17.0
- Remove discover from test-requirements
- make error reporting for extension loading quieter
- Add Python 3.5 classifier and venv
- Replace assertEquals() with assertEqual()
1.16.0
- Fix NamedExtensionManager fails when loading failing extension in order
- Remove irrelated output item
- Fix broken link about setuptools entry points
- NamedExtensionManager: call a callback when some names cannot be found
- Updated from global requirements
1.15.0
- •
- Updated from global requirements
1.14.0
- •
- Trivial: ignore openstack/common in flake8 exclude list
1.13.0
- •
- dont claim copyright for future years
1.12.0
- •
- Add a reference to entry_point_inspector
1.11.0
- Updated from global requirements
- Trival:Remove unused logging import
- Remove work around for NullHandler
- remove unnecessary dependency on argparse
1.10.0
- Use Stevedore exceptions for finding extensions
- Clean up Python 2.6 related stuff
- Updated from global requirements
- Remove Python 2.6 classifier
- cleanup tox.ini
1.9.0
- Updated from global requirements
- docs - Set pbr 'warnerrors' option for doc build
- Add clarifying language to description of scanning for plugins
- clean up default tox environment list
- Show how to add a plugin in a separate package
- replace the hard-coded history list with an auto-generated one
- Fix spelling typo for maunal
- Updated from global requirements
- Examples typo fix
1.8.0
- Updated from global requirements
- Updated from global requirements
- Updated from global requirements
1.7.0
- Updated from global requirements
- Updated from global requirements
- Titlecase looks nicer sometimes in detailed mode
- Updated from global requirements
- Updated from global requirements
- Updated from global requirements
- Update homepage to openstack hosted docs page
1.6.0
- Document the signature for check_func
- Updated from global requirements
- Switch badges from 'pypip.in' to 'shields.io'
- Remove unnecessary openstack-common.conf
1.5.0
- Removed non-free color profile from .jpg
- Add sphinx integration
- Updated from global requirements
- Fix Python versions supported
- Remove run_cross_tests.sh
- fix author contact details
- re-raise exception with full traceback
1.4.0
- Uncap library requirements for liberty
- Add pypi download + version badges
- Updated from global requirements
1.3.0
- Updated from global requirements
- Fix test for finding multiple drivers
- ignore .testrepository directory created by testr
- clean up default environments run by tox
1.2.0
- Use pkg_resources resolve() and require() instead of load()
- Fix the README.rst file format for pypi
- Workflow documentation is now in infra-manual
- Implement a __contains__ override for extension manager
- Update link to docs in README
- Bring doc build up to standard
1.1.0
- Add pbr to dependency list
- Updated from global requirements
- Add more detail to the README
- Migrate tox to use testr
- Update repository location in docs
1.0.0
- Build universal wheels
- Work toward Python 3.4 support and testing
- warn against sorting requirements
1.0.0.0a1
- Updated from global requirements
- Fix incorrect image reference in documentation
- Fix requirement handling in tox
- Updated from global requirements
- use six.add_metaclass
- Updated from global requirements
- driver: raise by default on import failure
- Add doc requirements to venv environ
- Import run_cross_tests.sh from oslo-incubator
- fix link to entry point docs
0.15
- Only log error when no load handler is set
- Update readme with links to bug tracker and source
- Update .gitreview after moving the repository
0.14.1
- •
- Fix the test manager implementation
0.14
- prep history for 0.14 release
- Make requirements checking optional
- Update docstrings
- Remove requirements checking for dependencies
- fix typo in contrib docs
- Add contributing instructions for github mirrors
- Allow a on_load_failure_callback to be provided
- Add venv environment to tox
- driver: remove useless arg propagate_map_exceptions
- Move to stackforge
- update release announcement file
- update release number in history
0.13
- update history file before release
- deprecate TestExtensionManager
- clean up docs for DriverManager
- simplify test instance factory contract
- driver extension manager test instance factory
- enabled extension manager test instance factory
- make Extension responsible for formatting its target name
- add test instance factory for the base, named, and hook managers
- use item access instead of temporary dict when ordering extensions
- DOC: Updated index.rst: distribute -> setuptools
- a work-around to avoid cpython bug (15881)
- add pypy to travis config
- add pypy to default test env list
- Fix pip call in tox config
- fix version number in blog announcement file
0.12
- update history and announce file for release
- remove version from setup.cfg and rely on git tag
- Switch to pbr
- Fix flake8 failures from pull/27
- Add map_method function to managers
- Fixes reporting the error when drivers have the same name
0.11
- prep for release 0.11
- Update null log handling for py26
0.10
- prep for 0.10 release
- fixed a bug in the test for propagating map exceptions
- Fix doc build with Python 2.6.x
- Update docstrings for map exception propagation
- Formatting fix
- Adds ability to propogate exceptions within map
0.9.1
- prep for 0.9.1 release
- Include all images from docs in sdist
0.9
- prep for 0.9 release
- add docs to default tox suite
- Remove reliance on distribute
- doc cleanup
- add reference to test class
- finish tutorial section on loading plugins
- remove line number references
- Add example of loading as a driver
- Add tutorial section on creating plugins
- add docs about names and namespaces
- touch up ceilometer design diagram
- add tox env for building docs
- Add structure for tutorial
- Add PyCon 2013 essay
- Update docs for NameDispatchExtensionManager
- Clean up autodoc for manager classes
- Add ExtensionManager.__getitem__
- Change sort for NamedExtensionManager
- Ignore missing extensions in named dispatch
- Set up extlinks extension
- document new name_order param in history
- Correct argument types in name sort tests
- fix type definition of names parameter
- optionally sort named extensions
- flake8 fixes
- Add travis-ci configuration file
- add python 3.3 support tag
- add python 3.3 setup to tox
- link to docs from README
0.8
- update settings for 0.8 release
- Check the names of plugins before importing them
- fix typo in docstring
- Let AssertionErrors bubble up
0.7.2
- prep for release 0.7.2
- fix logging support under python 2.6
- Run tests under python 2.6
0.7.1
- •
- Fix logging configuration
0.7
- prepare release 0.7
- Cache the entry points discovered within a namespace
0.6
- Bump version to 0.6
- Load extensions before checking enabled status
- Fix line lengths for pep8
0.5
- Prepare for 0.5 release
- Add TestExtensionManager
0.4
- Add driver property to DriverManager
- Prepare release 0.4
- Remove the name argument to extension constructors
- fix inheritence hierarchy of DriverManager
- Set up logging in enabled module
- Log the full exception when plugin load fails
- Optimize implementation of NameDispatchExtensionManager
- Add response callback to _invoke_one_plugin()
- Refactor code for invoking plugins from map()
- clean up formatting
0.3
- make DriverManager callable
- add download link
- clean up announcement text
- add installation instructions
- update history for 0.3 release
- add dispatch managers
- documentation touch-up
0.2
- release 0.2 with docs
- finish first draft of documentation
- add API documentation
- rename loading; add enabling patterns
- add diagrams to illustrate the loading patterns
- Add descriptions of loading patterns
- add script for running sphinx as I edit
- add history file
0.1
- get the version from setup.py and always use today's date
- doc files created by sphinx-quickstart
- logging tweak
- add DriverManager
- add hook manager
- break up monolithic module
- add EnabledExtensionManager and NamedExtensionManager
- add docstring gs
- make ExtensionManager iterable
- error when no extensions to map()
- add map method
- basic ExtensionManager implementation
- add license
- set up tox and fix packaging
- create setup.py
- Initial commit
EXTENSION MANAGER CLASSES
DriverManager
- class stevedore.driver.DriverManager(namespace, name, invoke_on_load=False, invoke_args=(), invoke_kwds={}, on_load_failure_callback=None, verify_requirements=False, warn_on_missing_entrypoint=True)
- Bases: NamedExtensionManager
Load a single plugin with a given name from the namespace.
- Parameters
- namespace (str) -- The namespace for the entry points.
- name (str) -- The name of the driver to load.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- __call__(func, *args, **kwds)
- Invokes func() for the single loaded extension.
The signature for func() should be:
def func(ext, *args, **kwds):
pass
The first argument to func(), 'ext', is the Extension instance.
Exceptions raised from within func() are logged and ignored.
- Parameters
- func -- Callable to invoke for each extension.
- args -- Variable arguments to pass to func()
- kwds -- Keyword arguments to pass to func()
- Returns
- List of values returned from func()
- __init__(namespace, name, invoke_on_load=False, invoke_args=(), invoke_kwds={}, on_load_failure_callback=None, verify_requirements=False, warn_on_missing_entrypoint=True)
- property driver
- Returns the driver being used by this manager.
- classmethod make_test_instance(extension, namespace='TESTING', propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Construct a test DriverManager
Test instances are passed a list of extensions to work from rather than loading them from entry points.
- Parameters
- extension (Extension) -- Pre-configured Extension instance
- namespace (str) -- The namespace for the manager; used only for identification since the extensions are passed in.
- propagate_map_exceptions (bool) -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- Returns
- The manager instance, initialized for testing
HookManager
- class stevedore.hook.HookManager(namespace, name, invoke_on_load=False, invoke_args=(), invoke_kwds={}, on_load_failure_callback=None, verify_requirements=False, on_missing_entrypoints_callback=None, warn_on_missing_entrypoint=False)
- Bases: NamedExtensionManager
Coordinate execution of multiple extensions using a common name.
- Parameters
- namespace (str) -- The namespace for the entry points.
- name (str) -- The name of the hooks to load.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- warn_on_missing_entrypoint (bool) -- Flag to control whether failing to load a plugin is reported via a log mess. Only applies if on_missing_entrypoints_callback is None.
- __getitem__(name)
- Return the named extensions.
Accessing a HookManager as a dictionary (em['name']) produces a list of the Extension instance(s) with the specified name, in the order they would be invoked by map().
- __init__(namespace, name, invoke_on_load=False, invoke_args=(), invoke_kwds={}, on_load_failure_callback=None, verify_requirements=False, on_missing_entrypoints_callback=None, warn_on_missing_entrypoint=False)
NamedExtensionManager
- class stevedore.named.NamedExtensionManager(namespace, names, invoke_on_load=False, invoke_args=(), invoke_kwds={}, name_order=False, propagate_map_exceptions=False, on_load_failure_callback=None, on_missing_entrypoints_callback=None, verify_requirements=False, warn_on_missing_entrypoint=True)
- Bases: ExtensionManager
Loads only the named extensions.
This is useful for explicitly enabling extensions in a configuration file, for example.
- Parameters
- namespace (str) -- The namespace for the entry points.
- names (list(str)) -- The names of the extensions to load.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- name_order (bool) -- If true, sort the loaded extensions to match the order used in names.
- propagate_map_exceptions (bool) -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- on_missing_entrypoints_callback (function) -- Callback function that will be called when one or more names cannot be found. The provided argument will be a subset of the 'names' parameter.
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- warn_on_missing_entrypoint (bool) -- Flag to control whether failing to load a plugin is reported via a log mess. Only applies if on_missing_entrypoints_callback is None.
- __init__(namespace, names, invoke_on_load=False, invoke_args=(), invoke_kwds={}, name_order=False, propagate_map_exceptions=False, on_load_failure_callback=None, on_missing_entrypoints_callback=None, verify_requirements=False, warn_on_missing_entrypoint=True)
- classmethod make_test_instance(extensions, namespace='TESTING', propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Construct a test NamedExtensionManager
Test instances are passed a list of extensions to use rather than loading them from entry points.
- Parameters
- extensions (list of Extension) -- Pre-configured Extension instances
- namespace (str) -- The namespace for the manager; used only for identification since the extensions are passed in.
- propagate_map_exceptions (bool) -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- Returns
- The manager instance, initialized for testing
EnabledExtensionManager
- class stevedore.enabled.EnabledExtensionManager(namespace, check_func, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Bases: ExtensionManager
Loads only plugins that pass a check function.
The check_func argument should return a boolean, with True indicating that the extension should be loaded and made available and False indicating that the extension should be ignored.
- Parameters
- namespace (str) -- The namespace for the entry points.
- check_func (callable, taking an Extension instance as argument) -- Function to determine which extensions to load.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- propagate_map_exceptions (bool) -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- __init__(namespace, check_func, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
DispatchExtensionManager
- class stevedore.dispatch.DispatchExtensionManager(namespace, check_func, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Bases: EnabledExtensionManager
Loads all plugins and filters on execution.
This is useful for long-running processes that need to pass different inputs to different extensions.
- Parameters
- namespace (str) -- The namespace for the entry points.
- check_func (callable) -- Function to determine which extensions to load.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- propagate_map_exceptions -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- map(filter_func, func, *args, **kwds)
- Iterate over the extensions invoking func() for any where filter_func()
returns True.
The signature of filter_func() should be:
def filter_func(ext, *args, **kwds):
pass
The first argument to filter_func(), 'ext', is the Extension instance. filter_func() should return True if the extension should be invoked for the input arguments.
The signature for func() should be:
def func(ext, *args, **kwds):
pass
The first argument to func(), 'ext', is the Extension instance.
Exceptions raised from within func() are propagated up and processing stopped if self.propagate_map_exceptions is True, otherwise they are logged and ignored.
- Parameters
- filter_func -- Callable to test each extension.
- func -- Callable to invoke for each extension.
- args -- Variable arguments to pass to func()
- kwds -- Keyword arguments to pass to func()
- Returns
- List of values returned from func()
- map_method(filter_func, method_name, *args, **kwds)
- Iterate over the extensions invoking each one's object method called
method_name for any where filter_func() returns True.
This is equivalent of using map() with func set to lambda x: x.obj.method_name() while being more convenient.
Exceptions raised from within the called method are propagated up and processing stopped if self.propagate_map_exceptions is True, otherwise they are logged and ignored.
Added in version 0.12.
- Parameters
- filter_func -- Callable to test each extension.
- method_name -- The extension method name to call for each extension.
- args -- Variable arguments to pass to method
- kwds -- Keyword arguments to pass to method
- Returns
- List of values returned from methods
NameDispatchExtensionManager
- class stevedore.dispatch.NameDispatchExtensionManager(namespace, check_func, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Bases: DispatchExtensionManager
Loads all plugins and filters on execution.
This is useful for long-running processes that need to pass different inputs to different extensions and can predict the name of the extensions before calling them.
The check_func argument should return a boolean, with True indicating that the extension should be loaded and made available and False indicating that the extension should be ignored.
- Parameters
- namespace (str) -- The namespace for the entry points.
- check_func (callable) -- Function to determine which extensions to load.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- propagate_map_exceptions -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- __init__(namespace, check_func, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- map(names, func, *args, **kwds)
- Iterate over the extensions invoking func() for any where the name is in
the given list of names.
The signature for func() should be:
def func(ext, *args, **kwds):
pass
The first argument to func(), 'ext', is the Extension instance.
Exceptions raised from within func() are propagated up and processing stopped if self.propagate_map_exceptions is True, otherwise they are logged and ignored.
- Parameters
- names -- List or set of name(s) of extension(s) to invoke.
- func -- Callable to invoke for each extension.
- args -- Variable arguments to pass to func()
- kwds -- Keyword arguments to pass to func()
- Returns
- List of values returned from func()
- map_method(names, method_name, *args, **kwds)
- Iterate over the extensions invoking each one's object method called
method_name for any where the name is in the given list of names.
This is equivalent of using map() with func set to lambda x: x.obj.method_name() while being more convenient.
Exceptions raised from within the called method are propagated up and processing stopped if self.propagate_map_exceptions is True, otherwise they are logged and ignored.
Added in version 0.12.
- Parameters
- names -- List or set of name(s) of extension(s) to invoke.
- method_name -- The extension method name to call for each extension.
- args -- Variable arguments to pass to method
- kwds -- Keyword arguments to pass to method
- Returns
- List of values returned from methods
ExtensionManager
- class stevedore.extension.ExtensionManager(namespace, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Bases: object
Base class for all of the other managers.
- Parameters
- namespace (str) -- The namespace for the entry points.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- propagate_map_exceptions (bool) -- Boolean controlling whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- __contains__(name)
- Return true if name is in list of enabled extensions.
- __getitem__(name)
- Return the named extension.
Accessing an ExtensionManager as a dictionary (em['name']) produces the Extension instance with the specified name.
- __init__(namespace, invoke_on_load=False, invoke_args=(), invoke_kwds={}, propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- __iter__()
- Produce iterator for the manager.
Iterating over an ExtensionManager produces the Extension instances in the order they would be invoked.
- __weakref__
- list of weak references to the object
- entry_points_names()
- Return the list of entry points names for this namespace.
- items()
- Return an iterator of tuples of the form (name, extension).
This is analogous to the Mapping.items() method.
- list_entry_points()
- Return the list of entry points for this namespace.
The entry points are not actually loaded, their list is just read and returned.
- classmethod make_test_instance(extensions, namespace='TESTING', propagate_map_exceptions=False, on_load_failure_callback=None, verify_requirements=False)
- Construct a test ExtensionManager
Test instances are passed a list of extensions to work from rather than loading them from entry points.
- Parameters
- extensions (list of Extension) -- Pre-configured Extension instances to use
- namespace (str) -- The namespace for the manager; used only for identification since the extensions are passed in.
- propagate_map_exceptions (bool) -- When calling map, controls whether exceptions are propagated up through the map call or whether they are logged and then ignored
- on_load_failure_callback (function) -- Callback function that will be called when an entrypoint can not be loaded. The arguments that will be provided when this is called (when an entrypoint fails to load) are (manager, entrypoint, exception)
- verify_requirements (bool) -- Use setuptools to enforce the dependencies of the plugin(s) being loaded. Defaults to False.
- Returns
- The manager instance, initialized for testing
- map(func, *args, **kwds)
- Iterate over the extensions invoking func() for each.
The signature for func() should be:
def func(ext, *args, **kwds):
pass
The first argument to func(), 'ext', is the Extension instance.
Exceptions raised from within func() are propagated up and processing stopped if self.propagate_map_exceptions is True, otherwise they are logged and ignored.
- Parameters
- func -- Callable to invoke for each extension.
- args -- Variable arguments to pass to func()
- kwds -- Keyword arguments to pass to func()
- Returns
- List of values returned from func()
- map_method(method_name, *args, **kwds)
- Iterate over the extensions invoking a method by name.
This is equivalent of using map() with func set to lambda x: x.obj.method_name() while being more convenient.
Exceptions raised from within the called method are propagated up and processing stopped if self.propagate_map_exceptions is True, otherwise they are logged and ignored.
Added in version 0.12.
- Parameters
- method_name -- The extension method name to call for each extension.
- args -- Variable arguments to pass to method
- kwds -- Keyword arguments to pass to method
- Returns
- List of values returned from methods
- names()
- Returns the names of the discovered extensions
Extension
- class stevedore.extension.Extension(name, entry_point, plugin, obj)
- Bases: object
Book-keeping object for tracking extensions.
The arguments passed to the constructor are saved as attributes of the instance using the same names, and can be accessed by the callables passed to map() or when iterating over an ExtensionManager directly.
- Parameters
- name (str) -- The entry point name.
- entry_point (EntryPoint) -- The EntryPoint instance returned by entrypoints.
- plugin -- The value returned by entry_point.load()
- obj -- The object returned by plugin(*args, **kwds) if the manager invoked the extension on load.
- property attr
- The attribute of the module to be loaded.
- property entry_point_target
- The module and attribute referenced by this extension's entry_point.
- Returns
- A string representation of the target of the entry point in 'dotted.module:object' format.
- property module_name
- The name of the module from which the entry point is loaded.
- Returns
- A string in 'dotted.module' format.
TestExtensionManager
- class stevedore.tests.manager.TestExtensionManager(extensions, namespace='test', invoke_on_load=False, invoke_args=(), invoke_kwds={})
- Bases: ExtensionManager
ExtensionManager that is explicitly initialized for tests.
Deprecated since version 0.13: Use the make_test_instance() class method of the class being replaced by the test instance instead of using this class directly.
- Parameters
- extensions (list of Extension) -- Pre-configured Extension instances to use instead of loading them from entry points.
- namespace (str) -- The namespace for the entry points.
- invoke_on_load (bool) -- Boolean controlling whether to invoke the object returned by the entry point after the driver is loaded.
- invoke_args (tuple) -- Positional arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- invoke_kwds (dict) -- Named arguments to pass when invoking the object returned by the entry point. Only used if invoke_on_load is True.
- __init__(extensions, namespace='test', invoke_on_load=False, invoke_args=(), invoke_kwds={})
INSTALLATION
Python Versions
stevedore is tested under Python 3.6 and 3.7.
Basic Installation
stevedore should be installed into the same site-packages area where the application and extensions are installed (either a virtualenv or the global site-packages). You may need administrative privileges to do that. The easiest way to install it is using pip:
$ pip install stevedore
or:
$ sudo pip install stevedore
Download
stevedore releases are hosted on PyPI and can be downloaded from: http://pypi.org/project/stevedore
Source Code
The source is hosted on the OpenStack infrastructure: https://opendev.org/openstack/stevedore/
Entry point inspector
To list entrypoints and registered plugins this tool can be also very useful: https://pypi.org/project/entry_point_inspector
Reporting Bugs
Please report bugs through the launchpad project: https://launchpad.net/python-stevedore
Indices and tables
- Index
- Search Page
AUTHOR
Author name not set
COPYRIGHT
2016, DreamHost
| October 24, 2025 | 5.5.0 |