osloserialization(1)

OSLOSERIALIZATION(1) oslo.serialization OSLOSERIALIZATION(1)

NAME

osloserialization - oslo.serialization 5.1.1

The oslo serialization library provides support for representing objects in transmittable and storable formats, such as JSON and MessagePack.

CONTENTS

Installation

At the command line:

$ pip install oslo.serialization


Or, if you have virtualenvwrapper installed:

$ mkvirtualenv oslo.serialization
$ pip install oslo.serialization


Using oslo.serialization

CHANGES

5.1.1

Fix issues related to tox4

5.1.0

  • Add Python3 antelope unit tests
  • Update master for stable/zed

5.0.0

  • Drop python3.6/3.7 support in testing runtime
  • Remove unnecessary unicode prefixes
  • Add Python3 zed unit tests
  • Update master for stable/yoga

4.3.0

  • Add Python3 yoga unit tests
  • Update master for stable/xena

4.2.0

  • setup.cfg: Replace dashes with underscores
  • setup.cfg: Replace dashes with underscores
  • Move flake8 as a pre-commit local target
  • Add Python3 xena unit tests
  • Update master for stable/wallaby
  • Remove lower-constraints remnants
  • Drop use of deprecated collections classes

4.1.0

  • Fix json to_primitive when using IO OBjects
  • Use TOX_CONSTRAINTS_FILE
  • Dropping lower constraints testing
  • Use py3 as the default runtime for tox
  • Fix hacking min version to 3.0.1
  • Add Python3 wallaby unit tests
  • Update master for stable/victoria
  • ignore reno generated artifacts
  • Adding pre-commit

4.0.1

  • Bump bandit version
  • Update openstackdocstheme

4.0.0

  • Remove the yamlutils module
  • Fix pygments style

3.2.0

  • Stop to use the __future__ module
  • Switch to newer openstackdocstheme and reno versions
  • Remove the unused coding style modules
  • Remove translation sections from setup.cfg
  • Align contributing doc with oslo's policy
  • Bump default tox env from py37 to py38
  • Add py38 package metadata
  • Add release notes links to doc index
  • Add Python3 victoria unit tests
  • Update master for stable/ussuri

3.1.1

  • Update hacking for Python3
  • Use unittest.mock instead of third party mock
  • Remove Python 3-specific conditional in dump_as_bytes
  • Drop use of six

3.1.0

  • Remove long-dead workaround
  • Add releasenote to deprecate the yamlutils module
  • Deprecated the yamlutils module
  • Update the minversion parameter
  • remove outdated header
  • reword releasenote for py27 support dropping

3.0.0

  • [ussuri][goal] Drop python 2.7 support and testing
  • tox: Trivial cleanup
  • Switch to Ussuri jobs
  • tox: Keeping going with docs
  • Update master for stable/train

2.29.2

  • Add Python 3 Train unit tests
  • Cap sphinx for py2 to match global requirements
  • Sync Sphinx requirement
  • Resolve some issue with tox.ini, setup.cfg
  • Dropping the py35 testing

2.29.1

  • Replace git.openstack.org URLs with opendev.org URLs
  • Explicitly set default_flow_style to False

2.29.0

  • OpenDev Migration Patch
  • Update master for stable/stein
  • Introduce a base yaml parser for all openstack components

2.28.2

  • add python 3.7 unit test job
  • Update hacking version
  • Fix README
  • Use template for lower-constraints
  • Update mailinglist from dev to discuss
  • Clean up .gitignore references to personal tools
  • Update the home-page to latest

2.28.1

  • add lib-forward-testing-python3 test job
  • add python 3.6 unit test job
  • import zuul job settings from project-config
  • import zuul job settings from project-config
  • py37: deal with Exception repr changes
  • Update reno for stable/rocky
  • Switch to stestr
  • Add release notes link to README

2.27.0

  • fix tox python3 overrides
  • Stop using encoding='utf-8' with msgpack

2.26.0

  • Unblock requirements changes
  • Remove stale pip-missing-reqs tox test
  • Trivial: Update pypi url to new url
  • set default python to python3
  • add lower-constraints job
  • Updated from global requirements

2.25.0

  • Follow the new PTI for document build
  • Update links in README
  • Update reno for stable/queens
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • msgpack-python has been renamed to msgpack

2.23.0

  • Updated from global requirements
  • Add bandit to pep8 job

2.22.0

  • Remove -U from pip install
  • Avoid tox_install.sh for constraints support
  • Remove setting of version/release from releasenotes
  • Updated from global requirements
  • Updated from global requirements

2.21.2

When serializing an exception return its 'repr'

2.21.1

jsonutils.to_primitive(): add fallback parameter

2.21.0

  • Updated from global requirements
  • warn developers when can't convert value into primitive
  • Updated from global requirements
  • Updated from global requirements
  • Update reno for stable/pike
  • Updated from global requirements

2.20.0

  • Revert "Explicitly raise ValueError in to_primitive"
  • Update URLs in documents according to document migration
  • rearrange existing documentation to fit the new standard layout
  • switch from oslosphinx to openstackdocstheme

2.19.1

2.19.0

  • Updated from global requirements
  • Don't iterate through addresses in netaddr.IPNetwork
  • Updated from global requirements
  • Updated from global requirements
  • Explicitly raise ValueError in to_primitive
  • Updated from global requirements
  • Updated from global requirements

2.18.0

Use Sphinx 1.5 warning-is-error

2.17.0

  • Updated from global requirements
  • [Fix gate]Update test requirement
  • Updated from global requirements
  • Update reno for stable/ocata
  • Remove references to Python 3.4

2.16.0

  • Prefer raising the python2.x type error for b64 decode errors
  • Add Constraints support
  • Don't raise TypeError for invalid b64
  • Replace six.iteritems() with .items()
  • Replaces uuid.uuid4 with uuidutils.generate_uuid()
  • Show team and repo badges on README

2.15.0

  • Updated from global requirements
  • Updated from global requirements
  • Add reno for release notes management
  • Updated from global requirements

2.14.0

  • Add a title to the API Documentation page
  • Updated from global requirements
  • modify the home-page info with the developer documentation

2.13.0

  • Drop H803 in flake8 ignore list
  • Fix serialization of binary strings in Python3

2.12.0

  • Updated from global requirements
  • Fix parameters of assertEqual are misplaced
  • Add Python 3.5 classifier and venv

2.11.0

  • Updated from global requirements
  • Updated from global requirements
  • Use {} instead of dict()

2.10.0

Updated from global requirements

2.9.0

2.8.0

  • Replace TypeError by ValueError in msgpackutils
  • Support serializing ipaddress objs with jsonutils
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements

2.7.0

Trivial: ignore openstack/common in flake8 exclude list

2.6.0

[Trivial] Remove executable privilege of doc/source/conf.py

2.5.0

  • Drop babel as requirement since its not used
  • Updated from global requirements
  • Unified and simplified API for all serializers
  • Make msgpack registries copyable (and add __contains__)
  • msgpack: fix datetime serialization

2.4.0

Updated from global requirements

2.3.0

  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Remove fallback to simplejson in jsonutils

2.2.0

  • Remove iso8601 dependency
  • Updated from global requirements
  • Updated from global requirements
  • Trival: Remove 'MANIFEST.in'

2.1.0

msgpackutils: drop Python 2.6 support

2.0.0

  • Updated from global requirements
  • Updated from global requirements
  • Remove python 2.6 classifier
  • Fix bug track URL
  • Remove python 2.6 and cleanup tox.ini
  • Updated from global requirements

1.11.0

  • Updated from global requirements
  • Use versionadded and versionchanged in doc
  • Fix coverage configuration and execution

1.10.0

  • Add utilities for base64
  • Also check for the existence of `items` in to_primitive()
  • Move 'history' -> release notes section
  • Add jsonutils.dump_as_bytes() function for py3
  • Change ignore-errors to ignore_errors
  • Updated from global requirements

1.9.0

  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements

1.8.0

  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Updated from global requirements
  • Drop stale test - test_jsonutils

1.7.0

  • Stop using deprecated oslo_utils.timeutils.strtime
  • Add tox target to find missing requirements
  • Remove oslo namespace package
  • Do not use oslo namespace for oslo.i18n lib
  • Updated from global requirements
  • Updated from global requirements
  • Switch badges from 'pypip.in' to 'shields.io'

1.6.0

  • Revert "jsonutils: replace strtime() usage with isoformat()"
  • jsonutils: encode dict keys
  • jsonutils: simplify simple value handling
  • jsonutils: replace strtime() usage with isoformat()
  • Advertise support for Python3.4 / Remove support for Python 3.3
  • Do not sync run_cross_tests.sh
  • Updated from global requirements
  • Remove run_cross_tests.sh
  • Fixes wrong folder for listing tests

1.5.0

  • Expose base msgpack exceptions so users don't need to import
  • Uncap library requirements for liberty
  • More docstring cleanups/tweaks
  • Add docstring(s) to handler registry(s)
  • Add pypi download + version badges
  • Cleanup README.rst and setup.cfg
  • Make the msgpackutils handlers more extendable
  • Updated from global requirements

1.4.0

Updated from global requirements

1.3.0

  • add dependency warning to requirements.txt
  • Correctly load and dump items with datetime.date(s)
  • Avoid using strtime for serializing datetimes
  • jsonutils: add set() tests and simplify recursive code
  • jsonutils: support UUID encoding
  • Use default in dumps()
  • Updated from global requirements
  • Update Oslo imports to remove namespace package
  • Add a messagepack utils helper module
  • Bump to hacking 0.10
  • Updated from global requirements
  • fix bug tracker link in README.rst

1.2.0

  • Move files out of the namespace package
  • Activate pep8 check that _ is imported
  • Updated from global requirements
  • Workflow documentation is now in infra-manual

1.1.0

  • Add pbr to installation requirements
  • Updated from global requirements
  • Fix pep8, docs, requirements issues in jsonutils and tests
  • Remove extraneous vim editor configuration comments
  • Support building wheels (PEP-427)
  • Fix coverage testing
  • Updated from global requirements
  • Support 'built-in' datetime module

1.0.0

  • Updated from global requirements
  • Add history/changelog to docs

0.2.0

  • Check for namedtuple_as_object support before using it
  • Work toward Python 3.4 support and testing

0.1.0

  • Updated from global requirements
  • Clean up docs before first release
  • Updated from global requirements
  • Added JSONEncoder and JSONDecoder
  • Add some basic docstring(s)
  • Fix docs build to run properly
  • Use oslo libraries - i18n and utils
  • Specify namedtuple_as_object=False when using simplejson
  • Get unit tests passing
  • exported from oslo-incubator by graduate.sh
  • Added missing jsonutils.dump() function
  • pep8: fixed multiple violations
  • Add kwargs to jsonutils.load(s) functions
  • Enforce unicode json output for jsonutils.load[s]()
  • jsonutils.load() accepts file pointer, not string
  • On Python <= 2.6, use simplejson if available
  • Use oslotest instead of common test module
  • Python3: enable test_jsonutils.py
  • Use six.moves.xmlrpc_client instead of xmlrpclib
  • Translation Message improvements
  • Python3 support for xmlrpclib
  • Use six.iteritems to make dict work on Python2/3
  • Remove vim header
  • Ensure that Message objects will be sent via RPC in unicode format
  • jsonutils: make types py3 compatible
  • jsonutils: do not require xmlrpclib
  • Make dependency on netaddr optional
  • Bump hacking to 0.7.0
  • Replace using tests.utils with openstack.common.test
  • Enable H302 hacking check
  • Add netaddr.IPAddress support to to_primitive()
  • python3: python3 binary/text data compatbility
  • Convert unicode for python3 portability
  • Optimise to_primitive common cases
  • Update to use flake8
  • Fix Copyright Headers - Rename LLC to Foundation
  • Don't LOG.error on max_depth (by default)
  • to_primitive imposes what seems to be an arbitary data structure depth of 3, but there is at least on case in Nova (Security group Rules) which requires a depth beyond this
  • Allow to_primitive to ignore datetimes
  • Replace direct use of testtools BaseTestCase
  • Use testtools as test base class
  • Fixes import order errors
  • Make project pyflakes clean
  • Add missing convert_instances arg
  • Track to_primitive() depth after iteritems()
  • Update iteritems test case to actually test iteritems
  • Use strtime() in to_primitive() for datetime objs
  • Add 'filedecoder' method to the jsonutils wrapper module
  • Make jsonutils.dumps more general
  • Handle xmlrpclib.DateTime types when serializing to JSON
  • Revert "Add a load function wrapper to jsonutils"
  • Add a load function wrapper to jsonutils
  • Create openstack.common.jsonutils

Contributing

If you would like to contribute to the development of oslo's libraries, first you must take a look to this page:

https://specs.openstack.org/openstack/oslo-specs/specs/policy/contributing.html


If you would like to contribute to the development of OpenStack, you must follow the steps in this page: http://docs.openstack.org/infra/manual/developers.html

Once those steps have been completed, changes to OpenStack should be submitted for review via the Gerrit tool, following the workflow documented at: http://docs.openstack.org/infra/manual/developers.html#development-workflow

Pull requests submitted through GitHub will be ignored.

Bugs should be filed on Launchpad, not GitHub: https://bugs.launchpad.net/oslo.serialization

API Reference

base64

Utilities to encode and decode Base64.

New in version 1.10.

oslo_serialization.base64.decode_as_bytes(encoded)
Decode a Base64 encoded string.
Parameters
encoded -- bytes or text Base64 encoded string to be decoded
Returns
decoded bytes string (bytes)

Use decode_as_text() to get the decoded string as text.

A TypeError is raised if the input is invalid (or incorrectly padded).


oslo_serialization.base64.decode_as_text(encoded, encoding='utf-8')
Decode a Base64 encoded string.

Decode the Base64 string and then decode the result from encoding (UTF-8 by default).

Parameters
encoded -- bytes or text Base64 encoded string to be decoded
Returns
decoded text string (bytes)

Use decode_as_bytes() to get the decoded string as bytes.


oslo_serialization.base64.encode_as_bytes(s, encoding='utf-8')
Encode a string using Base64.

If s is a text string, first encode it to encoding (UTF-8 by default).

Parameters
  • s -- bytes or text string to be encoded
  • encoding -- encoding used to encode s if it's a text string

Returns
Base64 encoded byte string (bytes)

Use encode_as_text() to get the Base64 encoded string as text.


oslo_serialization.base64.encode_as_text(s, encoding='utf-8')
Encode a string using Base64.

If s is a text string, first encode it to encoding (UTF-8 by default).

Parameters
  • s -- bytes or text string to be encoded
  • encoding -- encoding used to encode s if it's a text string

Returns
Base64 encoded text string (Unicode)

Use encode_as_bytes() to get the Base64 encoded string as bytes.


jsonutils

JSON related utilities.

This module provides a few things:

1.
A handy function for getting an object down to something that can be JSON serialized. See to_primitive().
2.
Wrappers around loads() and dumps(). The dumps() wrapper will automatically use to_primitive() for you if needed.
3.
This sets up anyjson to use the loads() and dumps() wrappers if anyjson is available.

oslo_serialization.jsonutils.dump(obj, fp, *args, **kwargs)
Serialize obj as a JSON formatted stream to fp
Parameters
  • obj -- object to be serialized
  • fp -- a .write()-supporting file-like object
  • default -- function that returns a serializable version of an object, to_primitive() is used by default.
  • args -- extra arguments, please see documentation of json.dump
  • kwargs --

    extra named parameters, please see documentation of json.dump



Changed in version 1.3: The default parameter now uses to_primitive() by default.


oslo_serialization.jsonutils.dump_as_bytes(obj, default=<function to_primitive>, encoding='utf-8', **kwargs)
Serialize obj to a JSON formatted bytes.
Parameters
  • obj -- object to be serialized
  • default -- function that returns a serializable version of an object, to_primitive() is used by default.
  • encoding -- encoding used to encode the serialized JSON output
  • kwargs -- extra named parameters, please see documentation of json.dumps

Returns
json formatted string

New in version 1.10.


oslo_serialization.jsonutils.dumps(obj, default=<function to_primitive>, **kwargs)
Serialize obj to a JSON formatted str.
Parameters
  • obj -- object to be serialized
  • default -- function that returns a serializable version of an object, to_primitive() is used by default.
  • kwargs --

    extra named parameters, please see documentation of json.dumps


Returns
json formatted string

Use dump_as_bytes() to ensure that the result type is bytes on Python 2 and Python 3.


oslo_serialization.jsonutils.load(fp, encoding='utf-8', **kwargs)
Deserialize fp to a Python object.
Parameters
  • fp -- a .read() -supporting file-like object
  • encoding -- encoding used to interpret the string
  • kwargs -- extra named parameters, please see documentation of json.loads

Returns
python object


oslo_serialization.jsonutils.loads(s, encoding='utf-8', **kwargs)
Deserialize s (a str or unicode instance containing a JSON
Parameters
  • s -- string to deserialize
  • encoding -- encoding used to interpret the string
  • kwargs --

    extra named parameters, please see documentation of json.loads


Returns
python object


oslo_serialization.jsonutils.to_primitive(value, convert_instances=False, convert_datetime=True, level=0, max_depth=3, encoding='utf-8', fallback=None)
Convert a complex object into primitives.

Handy for JSON serialization. We can optionally handle instances, but since this is a recursive function, we could have cyclical data structures.

To handle cyclical data structures we could track the actual objects visited in a set, but not all objects are hashable. Instead we just track the depth of the object inspections and don't go too deep.

Therefore, convert_instances=True is lossy ... be aware.

If the object cannot be converted to primitive, it is returned unchanged if fallback is not set, return fallback(value) otherwise.

Changed in version 2.22: Added fallback parameter.

Changed in version 1.3: Support UUID encoding.

Changed in version 1.6: Dictionary keys are now also encoded.


msgpackutils

MessagePack related utilities.

This module provides a few things:

1.
A handy registry for getting an object down to something that can be msgpack serialized. See HandlerRegistry.
2.
Wrappers around loads() and dumps(). The dumps() wrapper will automatically use the default_registry for you if needed.

New in version 1.3.

class oslo_serialization.msgpackutils.HandlerRegistry
Registry of type specific msgpack handlers extensions.

See: https://github.com/msgpack/msgpack/blob/master/spec.md#formats-ext

Do note that due to the current limitations in the msgpack python library we can not currently dump/load a tuple without converting it to a list.

This may be fixed in: https://github.com/msgpack/msgpack-python/pull/100

New in version 1.5.

copy(unfreeze=False)
Deep copy the given registry (and its handlers).

get(identity)
Get the handler for the given numeric identity (or none).

match(obj)
Match the registries handlers to the given object (or none).

max_value = 127
Applications can assign 0 to 127 to store application (or library) specific type handlers; see above ranges for what is reserved by this library and what is not.

min_value = 0
Applications can assign 0 to 127 to store application (or library) specific type handlers; see above ranges for what is reserved by this library and what is not.

non_reserved_extension_range = Interval(33, 127)
These ranges are always reserved for use by applications building their own type specific handlers (the meaning of extensions in this range will typically vary depending on application).

register(handler, reserved=False, override=False)
Register a extension handler to handle its associated type.

reserved_extension_range = Interval(0, 32)
These ranges are always reserved for use by oslo.serialization and its own add-ons extensions (these extensions are meant to be generally applicable to all of python).


class oslo_serialization.msgpackutils.Interval(min_value, max_value)
Small and/or simple immutable integer/float interval class.

Interval checking is inclusive of the min/max boundaries.


oslo_serialization.msgpackutils.default_registry = <oslo_serialization.msgpackutils.HandlerRegistry object>
Default, read-only/frozen registry that will be used when none is provided.

This registry has msgpack extensions for the following:

  • DateTime objects.
  • Date objects.
  • UUID objects.
  • itertools.count objects/iterators.
  • set and frozenset container(s).
  • netaddr.IPAddress objects (only if netaddr is importable).
  • xmlrpclib.DateTime datetime objects.

New in version 1.5.


oslo_serialization.msgpackutils.dump(obj, fp, registry=None)
Serialize obj as a messagepack formatted stream to fp.

Changed in version 1.5: Added registry parameter.


oslo_serialization.msgpackutils.dumps(obj, registry=None)
Serialize obj to a messagepack formatted str.

Changed in version 1.5: Added registry parameter.


oslo_serialization.msgpackutils.load(fp, registry=None)
Deserialize fp into a Python object.

Changed in version 1.5: Added registry parameter.


oslo_serialization.msgpackutils.loads(s, registry=None)
Deserialize s messagepack str into a Python object.

Changed in version 1.5: Added registry parameter.


RELEASE NOTES

Read also the oslo.serialization Release Notes.

INDICES AND TABLES

  • genindex
  • modindex
  • search

AUTHOR

unknown

COPYRIGHT

2023, OpenStack Foundation

October 26, 2023 5.1.1