Kilmarnock took the lead in the eighth-minute lead through Colin Nish, only for Michael Proctor to level seven minutes later.
Kyle restored Kilmarnoch's lead three minutes after the interval but Proctor again levelled on 52 minutes.
Boyd fired Killie in front for a third time on 80 minutes and Danny Invincibile made it 4-2 in the 83rd minute.