While many of the speed improvements can be made directly to the way the site is coded, a slow server can cripple your optimization efforts.

URL: http://www.websiteoptimization.com/speed/tweak/time-to-first-byte/