You're not naive wired, you are kind. That's the difference.
If that guy mentions about staying again or turns up just say you can't because it was noticed by neighbours and you are not allowed people stay over if not immediate family. Or say your son was upset. Or just say you can't.