hmmmm.... this is WAY more complex than an easy-answered inquiry. Here are a few "distant-to-the-situation" opinions by an old college prof:
1). In the entirety of my teaching career the most solid class of students were the so-called "non-traditional" [i.e. older] students. They were the adults. The others were the maybe-getting-to-be adults. The adults got up in the morning with an agenda aimed at the bigger picture of their lives and their responsibilities, while the "kids" were, well, kids. You can guess who tended to actually learn the material in a useable way --- and who were easier to teach. Sure your crude memory won't work as well, but your study methods will far overpower that ephemeral "advantage" of the noisy-brained youth. Plus, you won't care about "their" memory anyway, as you will no longer be distracted by petty competitive urges, and will be driven by healthier personal goals.
2). That said, and as indicated before, there are two rather "practical" elements on the other side of the balance scale that only you can assess: a]. can you really afford this in whatever your circumstances are?; and b]. how important is this new woman to you, and how strongly does she feel about this idea? As to "a", my guess is that you can find a way in this area. If our country cared more about genuinely important things {like guys like you getting their "late" educations in admirable socially-valuable fields}, this would be easier, but even with our John Wayne Pull-Yourself-Up-By-Your-Bootstraps attitude {yes, that pisses me off as an educator}, the sacrifice should be manageable. That leaves "b". Only you can estimate that.
You almost certainly can succeed academically at your age. You probably can make it economically with sacrifice. Your choice, it seems to me turns on your current relationship's dynamics. Both sides of this are matters of the Heart --- always tough calls.