"It is an annoyance." If anything, because he finds having to deal with anyone questioning it annoying. He shrugs at her questions, though.

"Cath Palug is my familiar. Not that he ever bothered to act much like it." And isn't here, obviously. "That one started following me, and while it's no use bothering now, having a familiar here could come in handy."

Yep, he's just tolerating it because it'd make for a convenient familiar later, should his powers start working again. That's definitely all there is to it, nothing more!