📝 Submitted to IEEE MMSP 2026
A Projection-Based Surrogate Gradient Interpretation
for Neural Codec Wrappers
Esteban Pesnel*†, Julien Le Tanou*, Michael Ropert*, Aline Roumy†, Thomas Maugey†
*MediaKind, Rennes, France †INRIA, Rennes, France - team COMPACT
Neural Wrappers End-to-end Optimization Codec Modeling Surrogate Gradient
Abstract
Neural wrappers are learned pre- and postprocessing networks designed to enhance the performance of conventional video codecs. Although these approaches can significantly improve compression efficiency, training them remains challenging due to the non-differentiability of video codecs, which arises from the multiple discrete decisions involved in the encoding process. Surrogate gradients have recently emerged as an effective solution for enabling end-to-end learning with conventional codecs. They offer two main advantages: they avoid training an additional network to mimic the codec, and they can improve compression performance. In particular, the recently proposed SCALED method, which leverages the true compression error, has shown strong results for training neural pre-processors such as downscalers. However, this SCALED gradient was originally introduced as a reparameterization trick, which limits its interpretability. In this paper, we show that this surrogate gradient can be interpreted as a first-order local approximation of the video codec, providing insight into its effectiveness. We further demonstrate that it is effective not only for learning downscaling operations, but also for the more challenging task of full neural wrapping with pre- and post-processing networks. Finally, we show that the approach generalizes well across different video codecs, quality factors, and tasks, including multiple downscaling ratios, yielding BD-Rate (PSNR) reductions of up to -23.59% on x264 and -20.07% on VVenC relative to standard resampling baselines.
Key Idea: Modeling the Codec as a Projection
A video codec \(\phi\) can be decomposed as an identity mapping plus a compression error \(\boldsymbol{\epsilon}\):
$$\boldsymbol{\hat{y}} = \phi(\boldsymbol{y}) = \boldsymbol{y} + \boldsymbol{\epsilon}$$
Codec decomposition
Since \(\phi\) is non-differentiable, we need a tractable approximation for backpropagation. We observe empirically that real codecs are approximately idempotent (\(\phi^2 \approx \phi\)), intensity-shift invariant (\(\phi(\boldsymbol{y}+\delta\boldsymbol{1}) \approx \phi(\boldsymbol{y})+\delta\boldsymbol{1}\)), and that their centered compression error is orthogonal to the codec output (\((\boldsymbol{\epsilon}-\bar{\boldsymbol{\epsilon}})^T\boldsymbol{\hat{y}} \approx 0\)). These properties suggest modeling \(\phi\) as a linear projection \(\mathbf{P}\), characterized by a projection direction \(\boldsymbol{u}\) and a subspace vector \(\boldsymbol{v}\):
$$\hat{\phi}(\boldsymbol{y}) = \mathbf{P}\boldsymbol{y} = \left(\mathbf{I} - \frac{\boldsymbol{u}\,\boldsymbol{v}^T}{\boldsymbol{v}^T\boldsymbol{u}}\right)\boldsymbol{y}$$
Linear projection model of the codec
The projection direction is naturally \(\boldsymbol{u} = \boldsymbol{\epsilon}\) (the codec shifts the input by the error vector). Intensity-shift invariance requires \(\boldsymbol{v}\) to be zero-mean, and minimizing the MSE between the true codec and its projection model yields \(\boldsymbol{v}^* = \boldsymbol{\epsilon} - \bar{\boldsymbol{\epsilon}}\) (the centered compression error), which satisfies the orthogonality constraint. The resulting MSE-optimal projector and its Jacobian are:
$$\mathbf{J}_{\text{Proj}} = \mathbf{I} - \frac{\boldsymbol{\epsilon}\,(\boldsymbol{\epsilon} - \bar{\boldsymbol{\epsilon}})^T}{N\,\sigma^2_{\boldsymbol{\epsilon}}} = \mathbf{J}_{\text{SCALED}}$$
Projection-based surrogate Jacobian = SCALED gradient
This Jacobian is exactly equal to the SCALED surrogate gradient from our previous work. This projection-based interpretation explains why SCALED works: its eigenvalues are bounded in \(\{0, 1\}\), preventing gradient explosion; it adapts instantly to any codec or quality setting (zero-shot); and it relies on true compression errors rather than learned approximations.
