ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Abstract
Evaluating outputs of large language models (LLMs) is challenging, requiring making -- and making sense of -- many responses. Yet tools that go beyond basic prompting tend to require knowledge of programming APIs, focus on narrow domains, or are closed-source. We present ChainForge, an open-source visual toolkit for prompt engineering and on-demand hypothesis testing of text generation LLMs. ChainForge provides a graphical interface for comparison of responses across models and prompt variations. Our system was designed to support three tasks: model selection, prompt template design, and hypothesis testing (e.g., auditing). We released ChainForge early in its development and iterated on its design with academics and online users. Through in-lab and interview studies, we find that a range of people could use ChainForge to investigate hypotheses that matter to them, including in real-world settings. We identify three modes of prompt engineering and LLM hypothesis testing: opportunistic exploration, limited evaluation, and iterative refinement.
- Publication:
-
arXiv e-prints
- Pub Date:
- September 2023
- DOI:
- 10.48550/arXiv.2309.09128
- arXiv:
- arXiv:2309.09128
- Bibcode:
- 2023arXiv230909128A
- Keywords:
-
- Computer Science - Human-Computer Interaction;
- Computer Science - Artificial Intelligence;
- H.5.2;
- I.2
- E-Print:
- 18 pages, 7 figures, published at CHI 2024