Video Enhancement with Task-Oriented Flow
Abstract
Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.
- Publication:
-
arXiv e-prints
- Pub Date:
- November 2017
- DOI:
- 10.48550/arXiv.1711.09078
- arXiv:
- arXiv:1711.09078
- Bibcode:
- 2017arXiv171109078X
- Keywords:
-
- Computer Science - Computer Vision and Pattern Recognition
- E-Print:
- IJCV 2019. Project page: http://toflow.csail.mit.edu