There are a lot of people who'd never live in LA specifically because it's so different. If I were choosing between there and say, Michigan, I'd go to Michigan and never turn back. LA's beaches are nasty (esp. compared to great lakes beaches), the traffic is awful (taking away from what I'm sure is very limited time with family anyway), the city life is more or less irrelevant to someone who's always at their job or traveling, and the people are a different breed from most of the country. No one turns down a job over quality of life? That's patently false---people do that all the time. Ask any Harvard hotshot who turned down a Wall street investment job. Also, USC can't fill their stadium unless they're in the top 10 (and even then it's a gamble). It's a very attractive job to be sure, but there are significant downsides just like there are at ND.
Question as an aside: What coaching job would you think would be the easiest/ have the fewest downsides? I'd think FSU---huge amounts of both talent and 'talent' to attract new talent. Decent weather, limited academics, winning tradition, police willing to cover up anything, local travel for recruiting. I'd think the only downside would be the in-state competition.