Why am I annoying ?!

I was dropping off our youngest at the childminders today and as I was leaving we were having small talk and my childminder said to my daughter "I find your Daddy annoying".

She is not the first to have said this. I would love to know why some people find me annoying ! I don't find it particularly hurtful or even annoying, it's just that from an academic perspective I'd love to know what it is that I say/do that rubs people up the wrong way.

Of course, I should just ask, but it's very difficult to to see the value of asking. I figure that

1. it's such an off-the-cuff remark that even the "remarker" might not be sure why
2. if I was to ask, the remarker might feel it's too hurtful to say why
3. I am too shy to challenge someone directly
4. by challenging them I might be even MORE annoying

Ho hum ...