For WMF, basically ALL caching layers are used in their optimal setup. Operation caching in PHP (or actually HHVM these days), in memory caching with memcached and output caching with varnish (for caching at the network request layer) and lastly multiple locations for serving content. Without this, your performance will probably be less than Wikipedia.
Basically, this is what CDNs do as well, but you don't have to configure it yourself with a CDN of course.
Note that RL does a lot more btw. It is also handles dependency management, de-duplication, partitioning (so you only load the stuff you need, kind of important with thousands of lines of CSS and JS that is used throughout MediaWiki), right to left language support etc. etc.
Anwyay, MediaWiki will use RL for all it's own resources regardless, so if you add a bit of JS and your performance tanks, then you should probably do a bit more reading on RL, because you are likely not using it correctly :)