Caddy seems to good to be true? I just set it up and it works perfect, straight out of the box

This seems too straightforward, what's the catch?

Like how secure is it? Should I be turning it off (and disabling the port forwarding) when not using it?

Do I need any additional security? Mainly just want to use it for Jellyfin

Thanks