Occlusion-Aware Temporal Frame Interpolation in a Highly Scalable Video Setting
D. Ruefenacht, R. Mathew, and D. Taubman
Abstract
We recently proposed a bidirectional hierarchical anchoring (BIHA) of motion fields for highly scalable video coding. The BIHA scheme employs piecewise-smooth motion fields, and uses breakpoints to signal motion discontinuities. In this paper, we show how the fundamental building block of the BIHA scheme can be used to perform bidirectional, occlusion-aware temporal frame interpolation (BOA-TFI).
From a ‘‘parent’’ motion field between two reference frames, we use information about motion discontinuities to compose motion fields from both reference frames to the target frame; these then get inverted so that they can be used to predict the target frame. During the motion inversion process, we compute a reliable occlusion mask, which is used to guide the bidirectional motion-compensated prediction of the target frame.
The scheme can be used in any state-of-the-art codec, but is most beneficial if used in conjunction with a highly scalable video coder which employs piecewise-smooth motion fields with motion discontinuities.
We evaluate the proposed BOA-TFI scheme on a large variety of natural and challenging computer-generated sequences, and our results compare favourably to state-of-the-art TFI methods.
Related Downloads
Preprint
Experimental Results
Sequence alley_1, interpolated frame 2.5
Click on images for full resolution.
 |
 |
| Reference frame f_2.0 | Reference frame f_3.0 |
 |
 |
| Mapped motion mf_2.5_2.0 | Mapped motion mf_2.5_3.0 |
 |
 |
| Disocclusion Mask f_2.5 | Interpolated frame f_2.5 |
Sequence bamboo_2, interpolated frame 25.5
Click on images for full resolution.
 |
 |
| Reference frame f_25.0 | Reference frame f_26.0 |
 |
 |
| Mapped motion mf_25.5_25.0 | Mapped motion mf_25.5_26.0 |
 |
 |
| Disocclusion Mask f_25.5 | Interpolated frame f_25.5 |
Sequence market_2, interpolated frame 16.5
Click on images for full resolution.
 |
 |
| Reference frame f_16.0 | Reference frame f_17.0 |
 |
 |
| Mapped motion mf_16.5_16.0 | Mapped motion mf_16.5_17.0 |
 |
 |
| Disocclusion Mask f_16.5 | Interpolated frame f_16.5 |
Sequence bandage_1, interpolated frame 14.5
Click on images for full resolution.
 |
 |
| Reference frame f_14.0 | Reference frame f_15.0 |
 |
 |
| Mapped motion mf_14.5_14.0 | Mapped motion mf_14.5_15.0 |
 |
 |
| Disocclusion Mask f_14.5 | Interpolated frame f_14.5 |
Sequence cave_4, interpolated frame 4.5
Click on images for full resolution.
 |
 |
| Reference frame f_4.0 | Reference frame f_5.0 |
 |
 |
| Mapped motion mf_4.5_4.0 | Mapped motion mf_4.5_5.0 |
 |
 |
| Disocclusion Mask f_4.5 | Interpolated frame f_4.5 |
Sequence shaman_3, interpolated frame 46.5
Click on images for full resolution.
 |
 |
| Reference frame f_46.0 | Reference frame f_47.0 |
 |
 |
| Mapped motion mf_46.5_46.0 | Mapped motion mf_46.5_47.0 |
 |
 |
| Disocclusion Mask f_46.5 | Interpolated frame f_46.5 |
Sequence cave_2, interpolated frame 4.5
Click on images for full resolution.
 |
 |
| Reference frame f_4.0 | Reference frame f_5.0 |
 |
 |
| Mapped motion mf_4.5_4.0 | Mapped motion mf_4.5_5.0 |
 |
 |
| Disocclusion Mask f_4.5 | Interpolated frame f_4.5 |
Reference
[1] D. Ruefenacht, R. Mathew, and D. Taubman, ‘‘Occlusion-Aware Temporal Frame Interpolation in a Highly Scalable Video Setting,’’ Submitted to APSIPA Transactions on Signal and Information Processing (ATSIP), 2016.
|
