Magic Layouts: Structural Prior for Component Detection in User Interface Designs
Abstract
We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs, enabling robust detection of UI components; buttons, text boxes and similar. Specifically we learn a prior over mobile UI layouts, encoding common spatial co-occurrence relationships between different UI components. Conditioning region proposals using this prior leads to performance gains on UI layout parsing for both hand-drawn UIs and app screenshots, which we demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2021
- DOI:
- 10.48550/arXiv.2106.07615
- arXiv:
- arXiv:2106.07615
- Bibcode:
- 2021arXiv210607615M
- Keywords:
-
- Computer Science - Computer Vision and Pattern Recognition;
- Computer Science - Human-Computer Interaction;
- Computer Science - Machine Learning
- E-Print:
- CVPR 2021