ViperGPT: Visual Inference via Python Execution for Reasoning
Abstract
Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2023
- DOI:
- 10.48550/arXiv.2303.08128
- arXiv:
- arXiv:2303.08128
- Bibcode:
- 2023arXiv230308128S
- Keywords:
-
- Computer Science - Computer Vision and Pattern Recognition
- E-Print:
- Website: https://viper.cs.columbia.edu/