We introduce a new framework for solving an important class of computational problems involving finite permutation groups, which includes calculating set stabilisers, intersections of subgroups, and isomorphisms of combinatorial structures. Our techniques generalise 'partition backtrack', which is the current state-of-the-art algorithm introduced by Jeffrey Leon in 1991, and which has inspired our work. Our backtrack search algorithms are organised around vertex- and arc-labelled directed graphs, which allow us to represent many problems more richly than do ordered partitions. We present the theory underpinning our framework, and we include the results of experiments showing that our techniques often result in smaller search spaces than does partition backtrack. An implementation of our algorithms is available as free software in the GraphBacktracking package for GAP.