Previously, adaptive rounding was proposed to improve quantization, which captures the statistics of the incoming residual signal and adjusts the rounding offsets accordingly. However, the adaptive rounding quantization is still based on the criterion which minimizes the mean-squared quantization error between the original signal and the quantization reconstructed signal. From the sense of rate-distortion optimization, the cost from the rate should also be considered.

The basic idea underlying the rate-distortion optimized quantization is to minimize a cost function D+ λR such that both the rate R and the distortion D are considered in coding decisions. For quantization case, the RD optimal coding is to solve a minimization problem of

(7)

where S is the original signal, and T^-1 denotes the inverse transform operation. Consider that the DCT is a unitary transform, which maintains the Euclidean distance. We use the Euclidean distance for D(.) to avoid the inverse DCT computation. Thus, the above equation becomes

(8)

where W=T(S), and T is the forward transform operation.

A simplified rate-distortion optimized quantization scheme is proposed and implemented in KTA software. In the RDO-Q, assuming the transform coefficients before quantization are W_i, (i=0,…,M-1), then the quantized coefficients/levels Z_i (i=0,…,M-1) are calculated as follows:

1. For a given coefficient position k, k=M-1,…,0, assume that coefficient W_k is the last significant coefficient in the block:
   • For each coefficient W_i, i=k-1,…,0, calculate its Lagrangian cost when the quantized value Z_i is equal to 0, Z_floor and Z_ceil, as shown in Figure 7. The Lagrangian cost J_k,i(λ,Z_i) when coefficient W_i is quantized to Z_i is calculated as:
     (9)
     where err(W_i , Z_i) is the quantization error if the coefficient W_i is quantized to value Z_i, measured by  Euclidean distance; And bits(Z_i) is the number of bits needed to code Z_i. The value of Z_floor and Z_ceil are defined as:
     (10)
     (11)
2. Let the final quantized level Z_i,opt=argmin J_k,i(λ,Z_i) and update Lagrangian cost J_k(λ) using J_k,i(λ,Z_opt).
3. The final quantized vector of quantized coefficients Z_opt=argmin J_k(λ).

Figure 7. Possible quantized values in RDO-Q

To speed up the algorithm the following simplifications are made:

1. If coefficient W_k is closer (measured as absolute difference between the coefficient and its reconstructed value) to Z_floor than to Z_ceil only value Z_floor is considered.
2. If coefficient W_k is closer to Z_floor than to Z_ceil, and Z_floor is equal to zero, coefficient W_k can not be the last nonzero coefficient. Hence the calculation of Lagrangian cost J_k(λ) is skipped for this value of k.
3. The calculation of J_k(λ) is stopped when J_k(λ) starts to increase with decreasing k.

Here, we only discuss the entropy coding by using CABAC, and for the CAVLC a similar estimation method for the number of bits can be applied. To estimate the number of bits required to code a coefficients, we use the tabularized values of entropy of the probabilities corresponding to states in CABAC arithmetic coding engine. But, the CABAC algorithm uses the context modeling. That is, the coding of current coefficient in a block is related to the state of previous coded coefficients.

In general, CABAC consists of three steps, i.e.,

1. Binarization. The so-called UEG0 algorithm is used to convert non-zero transform coefficients into a binary representation so that the binary arithmetic coding engine can be used to code them.
2. Context modeling. CABAC defines a probability model for each binary bit.
3. Binary arithmetic coding. The binary representation is encoded bit by bit using corresponding models.

The RDO-Q is closely related to the context modeling for residual coding. Residual coding by CABAC includes two parts, i.e., coding a so-called significance map and coding non-zero coefficients. Given a zigzag ordered sequence of transform coefficients, its significance map is a binary sequence which indicates the occurrence and location of the non-zero coefficients. The context modeling for coding the significance map is associated with the zig-zag order, and is easy to be included in RDO-Q. The context modeling for coding non-zero coefficients, however, is complicated. For a given sequence, there are in total 10 contexts for coding non-zero coefficients, with 5 of them for coding the first bit of a binary representation and the other 5 dedicated to coding the second to 14th bits. Briefly, contexts are selected as follows,

1. Scan the sequence in the inverse order to initiate two parameters as NumLg1 and NumEq1. NumLg1 is the number of coefficients that are greater than 1 while NumEq1 accords to those equal to one.
2. The context for the first bit is determined by
   (12)
3. The context for the 2-14th bits is selected by
   (13)

There is also a bypass mode with a fixed distribution. Other bits in the binary representation use the bypass mode.

Figure 8. The graph structure for RDO quantization based on CABAC in H.264/AVC

In order to solve the minimization problem, a graph-based algorithm is used to address the computation of the rate function R(.) of CABAC. As shown in Figure 8, the graph is constructed based on coding features of CABAC. Basically, states are defined based on the context model selection. Thus, states are named by values of NumEq1 and NumLg1, as of NumEq1_NumLg1 , e.g., 2_0 accords to NumEq1 = 2 and NumLg1 = 0. When NumLg1 > 0, the context is irrelevant to NumEq1. Thus, there are three states as X_1, X_2, and X_3. The context is fixed for all NumLg1 ≥4. Accordingly, one state X_X is defined. For a 4×4 luma block, there are 16 columns, each of them corresponding to one coefficient. In each column there are up to 8 states. Transitions are established between states according to the increase of NumEq1 and NumLg1 , e.g., the state 1_0 is connected to 1_0, 2_0 and X_1 according to a quantization output of 0, 1, or greater than 1, respectively. In case that the quantization output is greater than 1, parallel transitions are established so that each accords to a unique value. In practice, because the distortion is a quadratic function with respect to the quantization output, it is sufficient to investigate only a few parallel transitions. Thus the complexity is greatly reduced without sacrificing the optimality. Finally, a graph structure as shown in Figure 8 is obtained.

The optimal RDO quantization design now becomes a problem to search for a path in the graph for the minimal RD cost. It is not hard to see that the above graph design allows an element-wise additive computation of the RD cost. The Viterbi algorithm is then used to do the search, which leads to the solution of the minimization problem.

References:

[1].    Gary J. Sullivan, “Adaptive quantization encoding technique using an equal expected-value rule”, Hong Kong JVT meeting contribution JVT-N011, January 2005.

[2].    E.-h. Yang and X. Yang, “Rate distortion optimization of H.264 with main profile compatibility,” pp282 – 286, ISIT, July 2006

[3].    M. Karczewicz, Y. Ye and I. Chong, Rate distortion optimized quantization, VCEG-AH21, Qualcomm,Jan. 2008

As discussed above, if the input p.d.f. is Laplacian distributed and if we can estimate λ, then the optimal f can be found analytically. But, usually the estimate of input p.d.f. is not available, then, how to select the rounding offset f? 

In order to select rounding offset f adaptively, an adaptive quantization encoding technique using an equal expected-value rule is proposed by Gary Sullivan from Microsoft. The adaptive adjustment of the rounding offset f occurs only in the encoding quantization process, which tries to select f without using any priori model knowledge on the input W. The aim is to make that the mean of the absolute value of the input, |W|,  is equal to its expected reconstruction value |W’|, i.e.,

     (5)

Any values in an interval would be reconstructed to some W’, so the distribution of W’ is a probability mass function (p.m.f.), with the probability mass equal to integral of the input probability distribution function (p.d.f.) over the interval: the change of f leads to the shift of quantization intervals, and thus results in the change of p.m.f. of W’.

Figure 6. Adaptive rounding using an equal expected-value rule

The rounding offset parameter f is updated during encoding as follow:

     (6)

where the positive constant k is determined empirically. This equation means that If |W_i| < |W’_i|, then try to decrease f. By decrease f, we increase the dead-zone and the probability mass of zero, and thus the expected value of |W’| would be shifted towards zero. Otherwise, the expected value of |W’| would be shifted away from zero. Here we summarize the algorithm of adaptive rounding:

• Initially set f₀=Δ/3 (for intra) or f₀= Δ/6 (for inter)
• Quantize incoming transform coefficient W₀, obtain W’₀
• Obtain f₁ ; Use it to quantize W₁
• If |W_i|=|W’_i|, no need to update f
• I(.) is the indicator function, which is equal to 1 when its argument is true, and 0 when its argument is false
• Discard samples that fall into the dead-zone, since dead-zone reconstruction value of 0 is in general optimal (input p.d.f. is symmetric)
• Use different f for each of these frequency components, and update separately, since they have different statistics
  –        16 frequency components for luma 4×4 block of Inter 4×4 modes
  –        16 frequency components for luma 4×4 block in Intra 4×4 modes
  –        …

The adaptive quantization encoding technique using an equal expected-value rule is only performed at the encoding quantization process, and is compatible with the standards, and only need modification only at the encoder. Experiential results show that up to 1dB improvement at high PSNR is observed.

2.1. Scalar dead-zone quantization

In this section the principle of H.264/AVC normal quantization scheme is described in a generalized form.

A scalar quantizer for input signal W can be decomposed into a function Z=C[W] called a classification rule that selects an integer-valued class identifier called the quantization index at the encoder, and a reconstruction rule that produces a real-valued output W’=R[Z] at the decoder. Video encoder applies entropy coding to the quantization indices and communicates to the decoder. Although H.264/AVC JM reference software implements some classification functions, only reconstruction function is standardized.

In the quantization step of the encoder, the transform coefficients of the prediction error are quantized. This quantization is used to reduce the precision of the coefficients. Furthermore, the quantizer is designed to map insignificant coefficient values to zero whilst retaining a reduced number of significant, non-zero coefficients. The output of a quantizer is typically a ‘sparse’ array of quantized coefficients, mainly containing zeros.

This uniform quantization can be expressed with equation (1), which describes the quantization step and with equation (2), which describes the inverse quantization step.

(1)

In this equation, the input signal W is mapped to a so called quantization level Z. Therefore, the quantization step-size ∆ and the rounding control parameter f is used. The function floor() rounds to the nearest integer towards minus infinity and sign() returns the sign of the input signal. Figure 1 shows the dead-zone plus uniform threshold quantization, where ∆ is the interval size except dead-zone, 2(∆-f) is the interval size of dead-zone, and f is the rounding offset.

Figure 1. Dead-zone with uniform threshold quantization

The mapping of the quantization level Z to the signal W’ is described in the inverse quantization step that is given by

(2)

Note that in (2), the parameter f is not considered. In the following the influence of the rounding control parameter f is analysed.

2.1 Influence of rounding control parameter f

By modifying f, the rounding behaviour can be controlled.  For example, in Figure 2, a rounding control of f=∆/2, leads to values W’ of the inverse quantized (reconstructed) signal that are located in the centre of a quantization interval, and the range ∆/2<W<=3∆/2 is quantized to W’=∆ .

Figure 2. Dead-zone quantization with f=D/2

If f is changed, the reconstructed values W’ are no longer located in the centre of the corresponding quantization interval. For example, in Figure 3, a quantizer for a rounding control parameter f=∆/4 is chosen.

Figure 3. Dead-zone quantization with f=∆/4

The parameter f=∆/4 has two effects:

1. Compared to f=∆/2, the quantization interval around zero is increased, and the range -3/4∆<W<3/4∆ is quantized to W’=0.
2. The values of the quantized signal for W’!=0 are no longer located in the center of a quantization interval. The range 3/4∆<W<=7/4∆ is quantized to W’=∆, etc.

Observed from the above examples, we can find that rounding offset f controls two aspects:

1. Size of dead-zone, 2(∆-f)
   - Decrease f, dead-zone increases
2. Location of the reconstruction point relative to the quantization interval:
   - Decrease f, all the intervals would be shifted away from zero

2.2 How to calculate the rounding offset f?

To minimize mean-squared quantization error, W’ should be the centroid (conditional mean) in the interval with respect to the probability distribution function (p.d.f.) of W, i.e., W’ = m = E[ W | a ≤ W ≤ a + Δ], or

(3)

where a is one value of the set [∆-f, 2∆-f, 3∆-f,…], and P(W) is the p.d.f of W.

For general distribution, the conditional means m could be at different distance from the lower threshold, for each interval, as conditional p.d.f. is different for different interval. Fortunately, for Laplacian distribution, the conditional means are at the same distance from the lower thresholds, for all intervals. Since DCT coefficients can be modeled by Laplacian distribution, this justifies using the same f for all intervals in H.264/AVC.

The non-uniform probability distribution inside a quantization interval can be approximated by a Laplacian distribution, which is shown in Figure 4.

Figure 4: Probability distribution of transform coefficients W inside a quantization interval.

Suppose that a Laplacian p.d.f. has standard derivation λ, or P(w)= (λ/2)exp(-λ|w|), the optimal f which minimize the distortion can be determined analytically for Laplacian distribution, which decreases with increasing λ (i.e., decreasing standard derivation),

(4)

Due to this distribution, the mean value of W inside a quantization interval is not located in the centre of corresponding interval. In order to reduce the quantization error, this effect is considered by using a rounding control parameter of f<∆/2. Such a rounding control parameter leads to smaller values of the quantized signal W’, which are not located in the centre of a quantization interval.

Figure 5 shows the monotonic property between optimal f and Laplacian standard derivation λ.

Figure 5. The monotonic property between optimal f and Laplacian standard derivation λ

In the reference model of H.264/AVC, the probability distribution varies between Intra and Inter mode, thus different rounding control parameters are used. The rounding control parameter for Intra is f=∆/3 and for Inter it is f=∆/6. Therefore, the values of the quantized signal W’ tend to be smaller than the value of the centre of the corresponding quantization interval. This behaviour is utilized to have a quantization that better fits to a non-uniform probability distribution of the input signal inside a quantization interval.

Currently most image and video coding systems and standards such as MPEG-1/2 and H264/AVC use transform-based techniques followed by quantization and entropy coding for performing compression. The key idea is that transforms de-correlate the signal and compact the energy of a block into few low pass coefficients, which after quantization and de-quantization could still represent the signal rather accurately. Nevertheless, this quantization/de-quantization process needs to be carefully designed in order to have the best possible subjective and objective quality.

In the encoder of H.264/AVC reference software, the scalar dead-zone quantization is adopted. In order to improve further the performance, other two adaptive quantization techniques are also introduced, which are both based on how to adjust the size of dead-zone and control the rounding behavior. In this tutorial, we will first introduce the principle of H.264/AVC normal quantization scheme, then discuss the adaptive rounding method which select adaptively rounding offset using an equal expected-value property. Finally, a more complex rate-distortion optimized quantization will be discussed.