numcodecs_safeguards

numcodecs_safeguards

Fearless lossy compression with numcodecs-safeguards

Lossy compression can be scary as valuable information or features of the data may be lost.

By using Safeguards to guarantee your safety requirements, lossy compression can be applied safely and without fear.

Overview

This package provides the SafeguardedCodec adapter / meta-compressor that can be wrapped around any existing (lossy) numcodecs.abc.Codec to guarantee that certain properties of the original data are preserved by compression.

The SafeguardedCodec treats the wrapped inner codec as a blackbox. To guarantee the user's safety requirements, it post-processes the decompressed data, if necessary. If no correction is needed, the SafeguardedCodec only has a three-byte overhead for the compressed data and a computational overhead at compression time (at decompression time, only the checksum is verified).

By using the SafeguardedCodec adapter, badly behaving lossy codecs become safe to use, at the cost of potentially less efficient compression, and lossy compression can be applied without fear.

The SafeguardedCodec must only be used to encode the complete data, i.e. not just a chunk of data, so that non-pointwise safeguards are correctly applied. Please refer to the xarray-safeguards frontend for applying safeguards to chunked data.

Example

You can wrap an existing codec with e.g. a relative error bound of \(eb_{rel} = 1\%\) and preserve data signs as follows:

import numpy as np
from numcodecs.fixedscaleoffset import FixedScaleOffset
from numcodecs_safeguards import SafeguardedCodec

# use any numcodecs-compatible codec
# here we quantize data >= -10 with one decimal digit
lossy_codec = FixedScaleOffset(
    offset=-10, scale=10, dtype="float64", astype="uint8",
)

# wrap the codec in the `SafeguardedCodec` and specify the safeguards to apply
sg_codec = SafeguardedCodec(codec=lossy_codec, safeguards=[
    # guarantee a relative error bound of 1%:
    #   |x - x'| <= |x| * 0.01
    dict(kind="eb", type="rel", eb=0.01),
    # guarantee that the sign is preserved:
    #   sign(x) = sign(x')
    dict(kind="sign"),
])

# some n-dimensional data
data = np.linspace(-10, 10, 21)

# encode and decode the data
encoded = sg_codec.encode(data)
decoded = sg_codec.decode(encoded)

# the safeguard properties are guaranteed to hold
assert np.all(np.abs(data - decoded) <= np.abs(data) * 0.01)
assert np.all(np.sign(data) == np.sign(decoded))

Please refer to the compression_safeguards.SafeguardKind for an enumeration of all supported safeguards.

Referenced by:

Modules:

  • checksum –
  • compute –

    Helper classes for configuring the compute behaviour of the SafeguardedCodec.

  • lossless –

    Helper classes for lossless encoding for the SafeguardedCodec.

Classes:

SafeguardedCodec 

SafeguardedCodec(
    *,
    codec: dict[str, JSON] | Codec,
    safeguards: Collection[dict[str, JSON] | Safeguard],
    fixed_constants: Mapping[str | Parameter, Value]
    | Bindings = EMPTY,
    lossless: None | dict[str, JSON] | Lossless = None,
    compute: None | dict[str, JSON] = None,
    _version: None | str | Version = None,
)

Bases: Codec, CodecCombinatorMixin

flowchart LR numcodecs_safeguards.SafeguardedCodec[SafeguardedCodec] click numcodecs_safeguards.SafeguardedCodec href "" "numcodecs_safeguards.SafeguardedCodec"

An adaptor codec that uses Safeguards to guarantee certain properties / safety requirements are upheld by the wrapped codec.

Parameters:
  • codec (dict[str, JSON] | Codec) –

    The codec that will be wrapped with safeguards. It can either be passed as a codec configuration dict, which is passed to numcodecs.registry.get_codec(config), or an already initialized Codec. If you want to wrap a sequence or stack of codecs, you can use the numcodecs_combinators.stack.CodecStack combinator, e.g. to first replace codec-unsupported values with the numcodecs_replace.ReplaceFilterCodec and later restore them using the SameValueSafeguard.

    The codec must be deterministic during decoding (but can be non-deterministic during encoding) such that decoding the same bytes always produces the same bitwise equivalent decoded result.

    It is desirable to perform lossless compression after applying the safeguards (rather than before), e.g. by customising the Lossless.for_codec field of the lossless parameter.

    The codec combined with its lossless encoding must encode to a 1D buffer of bytes. It is also recommended that the codec can decode without receiving the output data type and shape via the out parameter. If the codec does not fulfil these requirements, it can be wrapped inside the numcodecs_combinators.framed.FramedCodecStack combinator.

    It is also possible to compress the data with just the safeguards (i.e. without a codec that provides proper lossy compression) by passing numcodecs_zero.ZeroCodec() or dict(id="zero") to codec. The zero codec only encodes the data type and shape, not the data values themselves, and decodes to all- zero values, forcing the safeguards to correct (almost) all values. With this configuration, the safeguards thus act as a safe lossy compressor in their own right, as any size reduction comes from the lossless compression of the safeguards corrections (which the safeguards produce to be highly-compressible, if possible).

  • safeguards (Collection[dict[str, JSON] | Safeguard]) –

    The safeguards that will be applied to the codec. They can either be passed as a safeguard configuration dict or an already initialized Safeguard.

    Please refer to the SafeguardKind for an enumeration of all supported safeguards.

    The SafeguardedCodec supports safeguards with late-bound parameters, e.g. the SelectSafeguard, but they must be provided as fixed_constants that are fixed and must be compatible with any data that will be encoded with this codec. Therefore, fixed constants should only be used for late-bound parameters that can be fixed across all uses of the codec.

  • fixed_constants (Mapping[str | Parameter, Value] | Bindings, default: EMPTY ) –

    Mapping of parameter names to fixed constant scalars or arrays that will be provided as late-bound parameters to the safeguards.

    The mapping must resolve all late-bound parameters of the safeguards and include no extraneous parameters.

    The provided values must have a compatible shape and values for any data that will be encoded with this codec, otherwise encode will fail.

    You can use the update_fixed_constants method inside a with statement to temporarily update the late-bound parameters.

    The fixed constants are included in the config of the codec. While int and float scalars are included as-is, numpy scalars and arrays are encoded to the losslessly compressed .npz format and stored in a data:application/x-npz;base64,<data> data URI.

  • lossless (None | dict[str, JSON] | Lossless, default: None ) –

    The lossless encoding that is applied after the codec and the safeguards:

    • Lossless.for_codec specifies the lossless encoding that is applied to the encoded output of the wrapped codec. By default, no additional lossless encoding is applied.
    • Lossless.for_corrections specifies the lossless encoding that is applied to the corrections that the safeguards produce. By default, Zstandard compression is applied after entropy coding.

    The lossless encoding must encode to a 1D buffer of bytes.

  • compute (None | dict[str, JSON] | Compute, default: None ) –

    Compute configuration with options that may affect the compression ratio and time cost of computing the safeguards corrections.

    While these options can change the particular corrections that are produced, the resulting corrections always satisfy the safety requirements.

  • _version (..., default: None ) –

    The version of the codec. Do not provide this parameter explicitly.
Raises:
  • ValueError –

    if codec wraps another SafeguardedCodec, which may create a printer problem.

  • LateBoundParameterResolutionError –

    if fixed_constants does not resolve all late-bound parameters of the safeguards or includes any extraneous parameters.

  • ... –

    if instantiating the codec or a safeguard raises an exception.

Methods:

  • update_fixed_constants –

    Create a new safeguarded codec, where the old fixed constants have been

  • encode –

    Encode the data in buf while safeguarding the compression.

  • decode –

    Decode the data in buf and apply the safeguards corrections.

  • get_config –

    Returns the configuration of this safeguarded codec.

  • from_config –

    Instantiate the safeguarded codec from a configuration dict.

  • map –

    Apply the mapper to this safeguarded codec.

codec_id class-attribute 

codec_id: str = 'safeguards'

codec property 

codec: Codec

The inner (unsafeguarded) codec.

safeguards property 

safeguards: Collection[Safeguard]

The collection of safeguards that will be applied.

late_bound property 

late_bound: Set[Parameter]

The set of late-bound parameters that the safeguards have.

The late-bound parameters must be provided as fixed constants. They must have a compatible shape and values for any data that will be encoded with this codec, otherwise encode will fail.

builtin_late_bound property 

builtin_late_bound: Set[Parameter]

The set of built-in late-bound constants that the numcodecs-safeguards provide automatically, which include the safeguards' built-ins as well as $x_min and $x_max.

update_fixed_constants 

update_fixed_constants(**kwargs: Value) -> SafeguardedCodec

Create a new safeguarded codec, where the old fixed constants have been overridden by new ones from **kwargs.

Only existing late-bound constants may be overridden and no new ones may be added.

The provided values must have a compatible shape and values for any data that will be encoded with this codec, otherwise encode will fail.

This method can be used inside a with statement to temporarily update the late-bound parameters.

Parameters:
  • **kwargs (Value, default: {} ) –

    Mapping from new parameters to values as keyword arguments.
Returns:
  • safeguards( SafeguardedCodec ) –

    The safeguarded codec, with the updated fixed constants.
Referenced by:

encode 

encode(buf: Buffer) -> bytes

Encode the data in buf while safeguarding the compression.

The encoded data is defined by the following stable format:

ULEB128(len(correction_bytes)), checksum, encoded_bytes, correction_bytes

where

  • ULEB128 refers to the unsigned LEB128 (little endian base 128) variable length encoding for unsigned integers
  • checksum refers to the RFC 1071 "Internet Checksum" over the little-endian C-order bytes of the corrected data, stored as two bytes in big-endian order
  • encoded_bytes refers to the encoded bytes produced by the codec and its optional lossless encoding
  • correction_bytes refers to the encoded correction bytes produced by the safeguards and their lossless encoding
  • the above bytestrings are concatenated into a single bytestring

If no correction is required, correction_bytes is empty and there is only a three-byte overhead from using the safeguards.

The buffer must contain the complete data, i.e. not just a chunk of data, so that non-pointwise safeguards are correctly applied and the global minimum $x_min and global maximum $x_max can be correctly provided. Please refer to the xarray-safeguards frontend for applying safeguards to chunked data.

Parameters:
  • buf (Buffer) –

    Data to be encoded. May be any object supporting the new-style buffer protocol.
Returns:
  • enc( bytes ) –

    Encoded data as a bytestring.
Raises:
  • TypeSetError –

    if the buffer uses an unsupported data type.

  • RuntimeError –

    if decoding with the codec with out=None fails.

  • RuntimeError –

    if codec and lossless do not encode to 1D bytes or do not recreate the data's dtype and shape during decoding.

  • RuntimeError –

    if the buffer is chunked but the safeguards use the built-in late-bound parameter $x_min or $x_max for the cross-chunk global minimum / maximum.

  • ... –

    if checking a safeguard or computing the correction for a safeguard raises an exception.
Referenced by:

decode 

decode(buf: Buffer, out: None | Buffer = None) -> Buffer

Decode the data in buf and apply the safeguards corrections.

Parameters:
  • buf (Buffer) –

    Encoded data. Must be an object representing a bytestring, e.g. bytes or a 1D array of np.uint8s etc.

  • out (None | Buffer, default: None ) –

    Writable buffer to store decoded data. N.B. if provided, this buffer must be exactly the right size to store the decoded data.
Returns:
  • dec( Buffer ) –

    Decoded data. May be any object supporting the new-style buffer protocol.
Raises:
  • ValueError –

    if buf is not a 1D bytes buffer.

  • ValueError –

    if buf has a corrupted header.

  • ... –

    if applying the safeguards correction raises an exception.

get_config 

get_config() -> dict[str, JSON]

Returns the configuration of this safeguarded codec.

numcodecs.registry.get_codec(config) can be used to reconstruct this adapter from the returned config.

Returns:
  • config( dict[str, JSON] ) –

    Configuration of this safeguarded codec.
Referenced by:

from_config classmethod 

from_config(config: dict[str, JSON]) -> Self

Instantiate the safeguarded codec from a configuration dict.

Parameters:
  • config (dict[str, JSON]) –

    Configuration of the safeguarded codec.
Returns:
  • safeguards( Self ) –

    Instantiated safeguarded codec.

map 

map(mapper: Callable[[Codec], Codec]) -> SafeguardedCodec

Apply the mapper to this safeguarded codec.

In the returned SafeguardedCodec, the codec is replaced by its mapped codec.

The mapper should recursively apply itself to any inner codecs that also implement the CodecCombinatorMixin mixin.

To automatically handle the recursive application as a caller, you can use 

numcodecs_combinators.map_codec(codec, mapper)
instead.

Parameters:
  • mapper (Callable[[Codec], Codec]) –

    The callable that should be applied to the inner (unsafeguarded) codec to map over this safeguarded codec.
Returns: