While many of the speed improvements can be made directly to the way the site is coded, a slow server can cripple your optimization efforts.