Estimating the features of noise is the first step in a chain of protocols that will someday lead to fault tolerant quantum computers. The randomised benchmarking (RB) protocol is designed with this exact mindset, estimating the average strength of noise in a quantum processor with relative ease in practice. However, RB, along with most other benchmarking and characterisation methods, is limited in scope because it assumes that the noise is temporally uncorrelated (Markovian), which is increasingly evident not to be the case. Here, we combine the RB protocol with a recent framework describing non-Markovian quantum phenomena to derive a general analytical expression of the average sequence fidelity (ASF) for non-Markovian RB with the Clifford group. We show that one can identify non-Markovian features of the noise directly from the ASF through its deviations from the Markovian case, proposing a set of methods to collectively estimate these deviations, non-Markovian memory time-scales, and diagnose (in)coherence of non-Markovian noise in an RB experiment. Finally, we demonstrate the efficacy of our proposal by means of several proof-of-principle examples. Our methods are directly implementable and pave the pathway to better understanding correlated noise in quantum processors.