Well, the worst it could do is bring fleas into the house, but I'm assuming that said neighbor is responsible and keeps their cats de-flea'd, or has collars on them or something. If your cats aren't upset over it, it's probably happened before while you weren't around.

Man, I would be thrilled if cats started flocking to me. Maybe it's your mutant superpower? :D