Detecting the third family of compact stars with normalizing flows

We explore the anomaly detection framework based on the normalizing flow (NF) models introduced in Morawski and Bejger [Phys. Rev. C 106, 065802 (2022), 10.1103/PhysRevC.106.065802] to detect the presence of a large (destabilizing) dense matter phase transition in neutron star (NS) observations of masses and radii, and relate the feasibility of detection with parameters of the underlying mass-radius sequence, which is a functional of the dense matter equation of state. Once trained on simulated data featuring continuous M (R ) solutions (i.e., no phase transitions), NF is used to determine the likelihood of a first-order phase transition in a given set of M (R ) observations featuring a discontinuity, i.e., to perform the anomaly detection. Different mock test sets, featuring two branch solutions in the M (R ) diagram, were parametrized by the NS mass at which the phase transition occurs, M_c, and the radius difference between the heaviest hadronic star and lightest hybrid star, Δ R . We analyze the impact of these parameters on the NF performance in detecting the presence of a first-order phase transition. Among the results, we report that given a set of 15 stars with radius uncertainty of 0.2 km, a detection of a two-branch solution is possible with 95% accuracy if Δ R >0.4 km .