High-dimensional degrees of freedom of photons can encode more quantum information than their two-dimensional counterparts. While the increased information capacity has advantages in quantum applications (such as quantum communication), controlling and manipulating these systems has been challenging. Here we show a method to perform deterministic arbitrary high-dimensional Pauli X gates for single photons carrying orbital angular momentum. The X gate consists of a cyclic permutation of qudit basis vectors and, together with the Z gate, forms the basis for performing arbitrary transformations. The proposed experimental setups only use two basic optical elements such as mode sorters and mode shifters and thus could be implemented in any system where these experimental tools are available. Furthermore the number of involved interferometers scales logarithmically with the dimension, which is important for practical implementation.