The CPU is a veritible space heater. Removing that heat from the case makes everything else that much cooler. I've seen some people watercool the GPU, north bridge and PSU, but I didn't- usually, they do that more to go fanless than for any sort of speed increase.
The performance bottleneck really depends on the application; in my case, the CPU speed helped, but my FSB was starting to become the bottleneck in the benchmarks I cared about, so 30% CPU overclock only really gave about a 25% speed inrease.
