Abstract The combined fit of the measured energy spectrum and shower maximum depth distributions of ultra-high-energy cosmic rays is known to constrain the parameters of astrophysical models with homogeneous source distributions. Studies of the distribution of the cosmic-ray arrival directions show a better agreement with models in which a fraction of the flux is non-isotropic and associated with the nearby radio galaxy Centaurus A or with catalogs such as that of starburst galaxies. Here, we present a novel combination of both analyses by a simultaneous fit of arrival directions, energy spectrum, and composition data measured at the Pierre Auger Observatory. The model takes into account a rigidity-dependent magnetic field blurring and an energy-dependent evolution of the catalog contribution shaped by interactions during propagation. We find that a model containing a flux contribution from the starburst galaxy catalog of around 20% at 40 EeV with a magnetic field blurring of around 20° for a rigidity of 10 EV provides a fair simultaneous description of all three observables. The starburst galaxy model is favored with a significance of 4.5σ (considering experimental systematic effects) compared to a reference model with only homogeneously distributed background sources. By investigating a scenario with Centaurus A as a single source in combination with the homogeneous background, we confirm that this region of the sky provides the dominant contribution to the observed anisotropy signal. Models containing a catalog of jetted active galactic nuclei whose flux scales with the γ-ray emission are, however, disfavored as they cannot adequately describe the measured arrival directions.