I don't say this to minimize the effort required to get started. I believe it definitely is difficult to get started. It is precisely with this level of difficulty in mind that I make the claim that the next step is even harder.
Many people spend years just getting to the point where their daily exercise routines are something to look forward to. Even once that routine has been established, the body plateaus and people find themselves in a situation where just maintaining their current level of fitness is progressively more difficult. At this point, many will choose new forms of exercise, be it a new workout plan, or a new sport, or something else. Switching to something new, though, is much easier than the alternative.
The alternative is focusing in on what you've managed to accomplish thus far, and designing a new plan to get even better at all of the same stuff. Honing one's craft, if you will.
I'll use running as an example. Going from "couch-to-5K" is difficult enough that people have formed whole support groups to emotionally buttress that transition. It's a major undertaking to teach oneself to run 3.1 miles without stopping, and without feeling miserable. Once accomplished, it is a major success. And yet, at that point, a runner's only ability is to run for about 3.1 miles per day, and finish without stopping.
In order to run faster, that person must identify problems with his/her running form -- and that requires analysis. That person must identify or approximate his/her VO2-max and heart rate zones -- and that requires some testing and analysis. That person must then use this information to develop a feasible training plan aimed at addressing his/her weaknesses: interval training for speed, threshold training for VO2-max improvement, long runs and two-a-days for muscular endurance, form exercises for better running economy, and so on.
Needless to say, all of this requires a time commitment, an expense of mental energy, possible consultation with experts, and then -- yes, only then -- the physically difficult and often painful process of training hard.
This is such a difficult process that many people don't even attempt it. They don't "want" to be that serious about what they're doing. They'd rather just go for a daily run (or engage in some simple daily maintenance activity).
And the fact that so few people do it goes to show just how difficult it is.