In the past few years, we have seen some great advancement in high dynamic range (HDR) imaging technologies. A significant trend in the content producing industry now is to capture video content with HDR. Meanwhile, leading video content distributors have also been moving at a fast pace in facilitating HDR videos to be delivered to consumers’ home and personal devices. At the other end of the delivery chain, however, how to best visualize the HDR content becomes a problem. Professional HDR displays that could faithfully preserve the creative intent by the content producers are quite expensive right now. Even the consumer HDR TVs of sub-par performance at reproducing the visual effect at post-production studios are at a pretty high price range. The reality is that in the next 5 or more years, even if HDR content can be successfully delivered to consumer devices, a large proportion of them will not be visualized using the displays with proper HDR capabilities.
Fortunately, this issue could be mitigated by the use of the so-called tone mapping operators (TMOs), which take HDR content as input and convert them into standard dynamic range (SDR) content. Apparently, because HDR and SDR videos have different bit depths, certain information loss is inevitable. For the same HDR input, different TMOs will create different SDR outputs, which may look drastically different. So the question is,
“How does one pick the best TMO?”
This turns out to be a difficult question to answer because the TMOs at hand often behave inconsistently across different visual content. One TMO that produces the best-looking SDR output for one video may be inferior to other TMOs for some other videos. To pick the best TMO, the most straightforward solution is to conduct a human subjective experiment. But subjective testing is cumbersome, slow and expensive. Moreover, subjective testing is hard to be used for real-time monitoring or to be incorporated into automated design and optimization systems to improve TMO performance.
What’s really missing in the industry is automatic objective quality metrics that are trustworthy and fast, so that tone-mapped videos can be monitored on the fly, TMOs can be fairly compared, and advanced TMOs can be developed to optimize perceptual quality. Designing objective quality metrics for tone-mapping is a highly challenging problem. Note that standard video quality assessment methods, such as PSNR and SSIM, are not even applicable because they can only compare images of the same dynamic range.
Fortunately, state-of-the-art research has shed some lights on how objective quality metrics of tone mapped visual content may be designed. Essentially, there are two fundamental quality factors when evaluating a tone-mapped image. The first is how well the fine structural details in the HDR content are preserved after tone mapping. For example, the fine structures in very dark or very bright regions in a picture may be removed after tone mapping, and a good quality metric should be able to detect such detail losses. Meanwhile, the quality metric is also expected to identify fake structures that do not exist in the HDR content but are created by tone mapping. The second quality factor is the naturalness and appeal of the final look of the tone mapped images. This is typically assessed by certain naturalness measures in a statistical sense. Both quality factors may play important roles in the overall assessment of the tone mapped visual content and should be incorporated into the objective quality metrics. This approach has demonstrated some remarkable success and the results are reported in the following papers:
- Yeganeh and Z. Wang, “Objective quality assessment of tone mapped images,” IEEE Transactions on Image Processing, vol. 22, no. 2, Feb. 2013.
- Yeganeh, S. Wang, K. Zeng, M. Eisapour and Z. Wang, “Objective quality assessment of tone-mapped videos,” IEEE International Conference on Image Processing, Sept. 2016.
Even further, by incorporating such new quality metrics, novel and perceptually more meaningful TMOs may be designed. A good example is given in the following paper:
- Ma, H. Yeganeh, K. Zeng, and Z. Wang, “High dynamic range image compression by optimizing tone mapped image quality index,” IEEE Transactions on Image Processing, vol. 24, no. 10, Oct. 2015.