Skip to content

Fix #1431: Avoid clipping after inverse transform when RExt__HIGH_BIT_DEPTH_SUPPORT is enabled

The specification defines no clipping after the inverse transform but in practice the output of the inverse transform may be clipped to the range -(2^bitdepth - 1)..(2^bitdepth - 1) for memory reduction purposes without affecting the end result. When the bitdepth is set to 16 and ExtendedPrecision is disabled, clipping to the range -(2^15)..(2^15 - 1) occurs, which is incorrect. The clipping range is modified to the numeric limits of the Pel data type, i.e. -(2^15)..(2^15 - 1) when RExt__HIGH_BIT_DEPTH_SUPPORT is disabled (default, range is unchanged) and -(2^31)..(2^31 - 1) otherwise.

Edited by Frank Bossen

Merge request reports