I agree with what you said here, Sam, and I particularly love the "old sweater" metaphor.
As someone who has been in an 8-year-long relationship with someone I met in adolescence, I incline to say that the instances where we meet the perfect partner at an early age are pretty rare.
The way I see it, we're billions of people on this planet, and certainly more than one would be a perfect pair for each individual.
I've noticed that couples with a long history will get married and have children at younger ages. While this isn't a good or a bad thing per se, it will limit their life experiences at an age when many others are still looking to explore the world and themselves.
Also, many people look back with nostalgia (or just regret :)) ) at their first love, which is why it seems that not having a second love, or even a third one, is like having fewer experiences in one's life bouquet. Does that make any sense? :D