There is a challenge that I have in life. There are times when you have to be the family guy who just sits around and wait for everyone to come around. This will not take you to Mount Everest, but instead, you will get to climb the local hill with a height of 100 meters. This is the role of a father and a family person.
But, you cannot live your entire life like that, because you might end up sad and bitter someday, feeling that you didn't get to do what you really wanted to do. Sometimes you will have to leave people behind in order for you to get to where you want to be. Maybe you have to tell someone that they cannot come along on a trip, because it will be too long and too rough for them? If you have to do that, then do so.
Maybe they will get angry, but that anger might lead them into practice, so the next time you want to do something similar, they will come along. Or maybe they will just remain bitter and angry. You, however, have reached a goal that you had and you gave the person a chance to catch up.
Just because other people want to be stuck in the mud, doesn't mean that you have to sit in the mud together with them. You are responsible for your life, so step up, get out of the mud, hand a helping hand to the person stuck in the mud, but if they don't want to come along, move on!