Dynamic Power Management: A Quantitative Approach
by Johan De Gelas on January 18, 2010 2:00 AM EST- Posted in
- IT Computing
The Hardware
One of the latest low power Xeons is the Intel Xeon L3426. It is the Xeon version of the "Lynnfield" architecture. What really makes this CPU special is the ability to boost its clock speed from 1.86GHz (default) to 3.2GHz without dissipating more power than 45W. The Turbo Boosted clock speed of 3.2GHz can only be reached if only one core is under load. Combine this with the fact that the CPU can cope with eight threads at once (four cores + Hyper-Threading) and you will see why this CPU deserves special attention. It should offer very low power when the load on the server is low as in that case the CPU steps back to a 1.2GHz clock. At the same time the CPU can scale up to 3.2GHz to provide excellent single threaded performance at 3.2GHz. And it also holds the promise that the CPU will never consume more than 45W, even when under heavy load. Performance will be "midrange" in those situations as the CPU cannot clock higher than 1.86GHz, but eight threads will still be running simultaneously.
At $284 this CPU looks like the best Intel offering ever in the entry level server space... but it does have drawbacks of course. In the desktop space, the affordable Core i7 860 "Lynnfield" CPU relegated the expensive Core i7 900 series "Bloomfield" CPUs to the shrinking high-end desktop CPU market. This is not going to happen in the server world: the "Lynnfield" CPU has no QPI links, so it cannot be used in multi-socket servers. The triple channel Xeon X55xx series ("Gulftown") will continue to be Intel's dual socket Xeons until they are followed up by quad- and six-core 32nm Westmere CPUs. So the first drawback is that you will be limited to four physical cores per server (and four logical SMT cores). For those of us running non-virtualized workloads, this is probably more than enough CPU power.
Talking about virtualization, that is the second drawback: each of the memory channels supports three DIMMs. As a result the CPU does not support more than 24GB. This is another reason why the cheap "Lynnfield" Xeon is not going to threaten the Xeon 5500 series anytime soon: a dual Xeon "Gulftown" server supports up to 144GB.
We also have the Xeon X3470 in the lab. When running close to idle, this CPU can also throttle back to 1.2GHz and save a lot of power. For the occasional single threaded task, the X3470 offers the best single threaded performance in the whole server CPU industry: one core can speed up to 3.6GHz. Yes, this is the "Xeonized" Core i7 860. Why does this CPU interest us? Well, the Xeon 3470 (2.93GHz) is only one speed bin faster than the X3460 (2.83GHz). The X3460 costs around the same price ($316) as the L3426 and can Turbo Boost up to 3.46GHz. And the X3460 brings up an interesting question: is a L3426 really the most interesting choice if you want a decent balance between performance and power? The L3426 has the advantage that you are sure you will never break a certain power limit. The X3460 however offers low power at low load too, and has more headroom to handle rare peaks. Below you can find the specs of our Intel server:
Intel SC5650UP
Single Xeon X3470 2.93GHz or Xeon L3426 1.86GHz
S3420GPLC, BIOS version August 24, 2009
Intel 3420 chipset
8GB (4 x 2GB) 1066MHz DDR-3
PSU: 400W HIPro HP-D4001E0
While we did not have a comparable AMD based server yet, this article would not be complete without a look at an AMD based system. AMD promised to send us a low power server, but after some back and forth correspondence it became clear the system would not be able to meet our deadline. Rest assured that we will update you once we get the new low power system from AMD. At the moment AMD looks a bit weak in the low cost server arena as its honor is defended by the 2.5GHz - 2.9GHz Opteron "Suzuka" CPU. That is a single CPU solution based on "Shanghai": four K10 cores and a 6MB L3 cache. This platform is almost EOL: we expect the San Marino and Adelaide platform around CeBIT 2010. Servers based on these new AMD platforms will save quite a bit of power compared to their older siblings. The six-core Lisbon that will find a home in these servers will be a slightly improved version of the current six-core AMD Opteron. Below are the specs of our AMD server:
Supermicro A+ Server 1021M-UR+V
Dual Opteron 2435 "Istanbul" 2.6GHz or Opteron 2389 2.9GHz
Supermicro H8DMU+, BIOS version June 18, 2009
8GB (4 x 2GB) 800MHz
PSU: 650W Cold Watt HE Power Solutions CWA2-0650-10-SM01-1
AMD picked the components in this server for our previous low power comparison of servers. We went with the Opteron 2435 as the six-core Opteron offers a very decent performance/watt ratio on paper. We will update the numbers once the Opteron 2419 EE arrives.
Making the comparison reasonably fair
So we had to work with two different servers. While the AMD versus Intel side of things is not our main focus, how can we make a reasonably fair comparison? The difference in power supplies is hardly a problem: both AMD and Intel feel that these power supplies are among the best available as they were chosen for their low power platforms. Both power supplies are 90% efficient over a very wide range of power load. The problem is the fans.
The fans in the AMD machine are small and fast, with speeds up to 11900 rpm! We disabled fan speed control to keep the power consumption of the fans constant. There are four fans and we measured the fan power consumption by taking out the fans that blow over the memory while keeping the two fans that cool the CPUs. This way we were sure that our CPU would not overheat and leak more power. We carefully measured the temperature of the CPU and jotted down the power measurements in all of our tests. We found out that each fan consumes about 8W. We did the same thing for the Intel machine: the power consumption of each fan was measured at the electrical outlet. The memory DIMMs were also checked: there was no significant difference between DDR2-800 and DDR3-1066, both in idle as well as under load. By taking the fans out of the equation, we can get a very reasonable comparison of both platforms. So how well do the current CPUs manage power?
Facebook
Twitter
RSS
35 Comments
View All Comments
JohanAnandtech - Monday, January 18, 2010 - link
In which utility do you set/manage the frequency of a separate core?n0nsense - Monday, January 18, 2010 - link
Gnome panel applets. CPU frequency monitor I guess it uses cpufreq. Each instance monitors core. So i have 4 of them visible all the time. If you have enabled CPU Frequency scaling (kernel) than you can select the governor (performance, on demand, conservative etc) or a static frequency. I can do it for each core. And it displays what i have set.Of course processor should support frequency scaling.(power now and speed step).
Most mainstream distributions (Ubuntu, Sabayon, Fedora) will use onedemand governor by default when processor with frequency scaling available. No user intervention required.
jordanclock - Monday, January 18, 2010 - link
I really think you're mistaken. Core 2 CPUs don't have any mechanism to allow per-core frequencies. There is one FSB clock and one multiplier. There is no way to set CPU0 to a different frequency than CPU1 (or for quad core, CPU2 and CPU3) because the variables that control the clock speed are chip wide.VJ - Tuesday, January 19, 2010 - link
These people seem to be convinced of per-core Speedstep:https://bugs.launchpad.net/ubuntu/+source/linux-so...">https://bugs.launchpad.net/ubuntu/+source/linux-so...
Maybe someone can ask David Tomaschik for the Intel documentation he refers to?
n0nsense - Monday, January 18, 2010 - link
I heard it in past, but i still tend to believe my eyes :)while writing this reply, i saw any possible combination. My Q9300 has 2 states 2.0GHz and
2.5GHz. It's not a server CPU. Have no reason to mislead you
VJ - Tuesday, January 19, 2010 - link
If there's only two states, then it's possible that one core is in the C2 state while the other is in its C0 state.The core in state C2 may be shown to be operating at 2Ghz (its lowest frequency) while it's really off. The OS may simply be reporting the lowest possible frequency while the core is really not receiving a clock signal.
So in general, if one core is showing its lowest frequency it may be off which still allows the other core to operate (at a different frequency).
It would be very strange if both cores are operating greater than their lowest and less than their highest frequencies at different frequencies.
From a different angle: Has anybody ever seen /proc/cpuinfo report a frequency less than the CPU/Core's lowest active frequency or even zero? Probably not.
n0nsense - Tuesday, January 19, 2010 - link
Nice theory :)But in this case, I see that each core doing something. htop shows that each core somewhere in 15% usage. So the only options left, are
1. Each core frequency can be controlled independently on C2D and C2Q (May be i3 i5 i7 too)
2. The OS is completely unaware of whats going on :) (which is less possible)
mino - Thursday, January 21, 2010 - link
"The OS is completely unaware of whats going on" is the right answer.:)
BTW, only x86 CPU's able to change freq per core are >=K10 for AMD and >=Nehalem for Intel.
VJ - Tuesday, January 19, 2010 - link
Not to defeat your argument/observations, rather for completeness' sake:It's also possible that the differences are due to the reading of the attributes. If the attributes are read in succession, then it's possible that the differences are due to the time of reading the attributes, while at any given instant, notwithstanding the allowable subtle differences in frequency described in this article, all cores are operating at the same frequency.
There's a lot of time at the bottom.
JanR - Tuesday, January 19, 2010 - link
Hi,I completely agree to this:
"It's also possible that the differences are due to the reading of the attributes."
The point is that desktop usage together with ondemand governor leads to a lot of fast frequency changes. Therefore, this is not a good scenario to decide on "per core" vs "per CPU". We did a lot of testing the following way:
Put load on all cores using "taskset" (this avoids C-states). Switch to "userspace" governor and then set frequencies of individual cores differently. You have one control per core but the actual hardware decides what really happens - you can check this in /proc/cpuinfo or using a tool such as "mhz" from lmbench as load generator (this one calculates actual frequency based on CPI and time, it allows also measurement of turbo frequencies).
Trying around, the results are:
AMD K8: One clock domain, maximum of the requested frequencies is taken
Intel Core2 Duo: Same as K8
AMD K10: Individual clock domains, you can clock each core individually
Intel Core 2 Quad: TWO clock domains! These CPUs are two dual core dies glued together so each die has its one multiplicator. Therefore, the cores of each die get the maximum of the requested frequencies but you can clock the two dies independendly.
Intel Nehalem: One clock domain, maximum of requests of all cores that are not in C-state! If you set one core to, e.g., 2.66 GHz and all other to 1.6, all cores clock at 1.6 as long as the core set to 2.66 is not used, they all switch to 2.66 if you put load on that core.
So far to our findings. "cat /proc/cpuinfo" or some funny tools are useless if you do not control the environment (userspace, manual settings). If you then enable ondemand, the system switches fast between different states and looking at it is just a snapshot, maybe taken in the middle of a transition.
Greetings,
Jan