Partly, yes. In the glory days, ND enjoyed a lot of advantages over the rest of CFB that have since been lost. Whenever I read someone beating his chest about getting ND "back where it belongs," they tend not to have a good grasp of how difficult it is to win in South Bend these days.
But my post above is mostly about how incredibly risky the coaching carousel is. Kelly's floor at ND has been 8-5, and that is: (1) pretty damned good considering our usual schedule; and (2) a massive improvement over his three predecessors. Murtaugh wrote a good article at OFD a while back explaining how avoiding really bad seasons is way more important than chasing great ones, because the former typically cause serious damage to a program.
If we fire Kelly--a known quantity with a very high floor--because we're unhappy with his apparent ceiling, there's a very real possibility we could end up with a Kiffin, Hoke or Zook. NDFB was at its nadir when Kelly took over, and he brought us to the title game within 3 years. He's never won fewer than 8 games, even throughout an endless stream of injuries to key players, academic suspensions, and scandals. When you fully consider the risks involved with making a change, there's really no case to be made for firing Kelly at all.