Seriously, unless you are extremely specialized and know exactly what you are doing, IMHO the answer is: Always
In what capacity, though? I see potential issues with both server firewalls, and client firewalls. Unless one wants their devices to be offline, there will always be at least one open port (for example, inbound on a server, and outbound on a client) which can be used as an attack vector.