Search for Higgs bosons decaying to aa in the μμττ final state in pp collisions at √s = 8 TeV with the ATLAS experiment

A search for the decay to a pair of new particles of either the 125 GeV Higgs boson (h) or a second charge parity (CP)-even Higgs boson (H) is presented. The data set corresponds to an integrated luminosity of 20.3  fb−1 of pp collisions at √s = 8  TeV recorded by the ATLAS experiment at the LHC in 2012. The search was done in the context of the next-to-minimal supersymmetric standard model, in which the new particles are the lightest neutral pseudoscalar Higgs bosons (a). One of the two a bosons is required to decay to two muons while the other is required to decay to two τ leptons. No significant excess is observed above the expected backgrounds in the dimuon invariant mass range from 3.7 to 50 GeV. Upper limits are placed on the production of h → aa relative to the standard model gg → h production, assuming no coupling of the a boson to quarks. The most stringent limit is placed at 3.5% for ma = 3.75  GeV. Upper limits are also placed on the production cross section of H → aa from 2.33 to 0.72 pb, for fixed ma = 5  GeV with mH ranging from 100 to 500 GeV.