I feel any sexual experience should be a mutual one, for mutual gratification, otherwise it seems kind of meaningless to me.
As someone else pointed out -- you making the other feel better should make you feel positive, at least to some degree. Otherwise, yes, you have a one-sided mind. If you only Like making them feel better when you're in the mood (mutual exchange in the moment) -- that doesn't mean you don't have a selfish mind. Remember that. When one's in the mood, it's natural to desire to see them enjoying it as you are -- even if they're a ONS.
By your rationale, you're in the same boat as someone who wouldn't want to give their BF/GF a 5 minute shoulder+back massage to work out a few kinks, because you weren't in the mood to do so. There's a difference between not being in the mood to do so when it's brought up, and it being a really bad time & place to do so.