Would the fact that I've only been with one man make you assume that I might be bad in bed?
Given what you’ve said about your marriage, I would assume the range of your sexual experience would be narrow, not that you were bad in bed.
Would you consider that a turn off?
Not at all, specially since you wished you could have done more, and you’re willing to work towards a more satisfying relationship.
It might be silly, but I do worry about how other men will perceive me if the topic of "how many people have you been with" arises.
That’s a private question you don’t have to answer ever, certainly not before you’ve been intimate. Besides, how-many-people is not a measure of your passion and sensuality, or your willingness to explore a wider palette of sexual delight. What matters is who you are now, not who you were then. One kiss can tell a man whether you’re present, attentive and engaged. That's more important than experience or technique. G’luck