You’ll likely see a shoe rack at places where they need to be removed, and by watching what other people do you’ll get a good idea of what you should. If in doubt, assume you should take them off; the slight embarrassment of unnecessarily removing your loafers is much better than the offence caused by leaving them on.