Mammals host gut microbiomes of immense physiological consequence, but the determinants of diversity in these communities remain poorly understood. Diet appears to be the dominant factor, but host phylogeny also seems to be an important, if unpredictable, correlate. Here we show that baleen whales, which prey on animals (fish and crustaceans), harbor unique gut microbiomes with surprising parallels in functional capacity and higher level taxonomy to those of terrestrial herbivores. These similarities likely reflect a shared role for fermentative metabolisms despite a shift in primary carbon sources from plant-derived to animal-derived polysaccharides, such as chitin. In contrast, protein catabolism and essential amino acid synthesis pathways in baleen whale microbiomes more closely resemble those of terrestrial carnivores. Our results demonstrate that functional attributes of the microbiome can vary independently even given an animal-derived diet, illustrating how diet and evolutionary history combine to shape microbial diversity in the mammalian gut.