In general, no. There is nothing in the base system which requires the presence of the source to operate. Some ports, like
<package>sysutils/lsof
</package>, will not build unless the source is installed. In particular, if the port builds a kernel module or directly operates on kernel structures, the source must be installed.