Modern processors have several power saving modes called C-states. They reflect the capability of an idle processor to turn off unused components in order to save power. Whereas C-states have been available for laptops for some time, they are a rather recent trend in the server market (for example, with Intel* processors, C-modes are only available since Nehalem).

When a processor runs in the C0 state, it is executing instructions. A processor running in any other C-state is idle. The higher the C number, the deeper the CPU sleep mode: more components are shut down to save power. Deeper sleep states save more power, but the downside is that they have higher latency (the time the CPU needs to go back to C0).

Some states also have submodes with different power saving latency levels. Which C-states and submodes are supported depends on the respective processor. However, C1 is always available.

Table 11.1, “C-States” gives an overview of the most common C-states.

While a processor operates (in C0 state), it can be in one of several CPU performance states (P-states). Whereas C-states are idle states (all but C0), P-states are operational states that relate to CPU frequency and voltage.

The higher the P-state, the lower the frequency and voltage at which the processor runs. The number of P-states is processor-specific and the implementation differs across the various types. However, P0 is always the highest-performance state. Higher P-state numbers represent slower processor speeds and lower power consumption. For example, a processor in P3 state runs more slowly and uses less power than a processor running at P1 state. To operate at any P-state, the processor must be in the C0 state where the processor is working and not idling. The CPU P-states are also defined in the Advanced Configuration and Power Interface (ACPI) specification, see http://www.acpi.info/spec.htm.

C-states and P-states can vary independently of one another.

T-states refer to throttling the processor clock to lower frequencies in order to reduce thermal effects. This means that the CPU is forced to be idle a fixed percentage of its cycles per second. Throttling states range from T1 (the CPU has no forced idle cycles) to Tn, with the percentage of idle cycles increasing the greater n is.

Note that throttling does not reduce voltage and since the CPU is forced to idle part of the time, processes will take longer to finish and will consume more power instead of saving any power.

T-states are only useful if reducing thermal effects is the primary goal. Since T-states can interfere with C-states (preventing the CPU from reaching higher C-states), they can even increase power consumption in a modern CPU capable of C-states.

Since quite some time, CPU power consumption and performance tuning is not only about frequency scaling anymore. In modern processors, a combination of different means is used to achieve the optimum balance between performance and power savings: deep sleep states, traditional dynamic frequency scaling and hidden boost frequencies. The turbo features (Turbo CORE* or Turbo Boost*) of the latest AMD* or Intel* processors allow to dynamically increase (boost) the clock speed of active CPU cores while other cores are in deep sleep states. This increases the performance of active threads while still complying to Thermal Design Power (TDP) limits.

However, the conditions under which a CPU core may use turbo frequencies are very architecture-specific. Learn how to evaluate the efficiency of those new features in Section 11.3.1, “Using the cpupower Tools”.

You can think of the in-kernel governors as a sort of pre-configured power schemes for the CPU. The CPUfreq governors use P-states to change frequencies and lower power consumption. The dynamic governors can switch between CPU frequencies, based on CPU utilization to allow for power savings while not sacrificing performance. These governors also allow for some tuning so you can customize and change the frequency scaling behavior.

The following governors are available with the CPUfreq subsystem:

If the CPUfreq subsystem in enabled on your system (which it is by default with openSUSE), you can find the relevant files and directories under /sys/devices/system/cpu/. If you list the contents of this directory, you will find a cpu{0..x} subdirectory for each processor, and several other files and directories. A cpufreq subdirectory in each processor directory holds a number of files and directories that define the parameters for CPUfreq. Some of them are writable (for root), some of them are read-only. If your system currently uses the on-demand or conservative governor, you will see a separate subdirectory for those governors in cpufreq, containing the parameters for the governors.

	Different Processor Settings
The settings under the cpufreq directory can be different for each processor. If you want to use the same policies across all processors, you need to adjust the parameters for each processor. Instead of looking up or modifying the current settings manually (in /sys/devices/system/cpu*/cpufreq), we advise to use the tools provided by the cpupower package for that.

After installing the cpupower package, view the available cpupower subcommands with cpupower --help. Access the general man page with man cpupower, and the man pages of the subcommands with man cpupower-subcommand.

The subcommands frequency-info and frequency-set are mostly equivalent to cpufreq-info and cpufreq-set, respectively. However, they provide extended output and there are small differences in syntax and behavior:

analyzing CPU 0:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 0 1 2 3
  CPUs which need to have their frequency coordinated by software: 0
  maximum transition latency: 10.0 us.
  hardware limits: 2.00 GHz - 2.83 GHz
  available frequency steps: 2.83 GHz, 2.34 GHz, 2.00 GHz
  available cpufreq governors: conservative, userspace, powersave, ondemand, performance
  current policy: frequency should be within 2.00 GHz and 2.83 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 2.00 GHz (asserted by call to hardware).
  boost state support:
    Supported: yes
    Active: yes
    

CPUidle driver: acpi_idle
CPUidle governor: menu
     
Analyzing CPU 0:
Number of idle states: 3
Available idle states: C1 C2
C1:
Flags/Description: ACPI FFH INTEL MWAIT 0x0
Latency: 1
Usage: 3156464
Duration: 233680359
C2:
Flags/Description: ACPI FFH INTEL MWAIT 0x10
Latency: 1
Usage: 273007117
Duration: 103148860538

|Mperf               || Idle_Stats  
                              
CPU | C0   | Cx   | Freq || POLL | C1   | C2   | C3   
   0|  3.71| 96.29|  2833||  0.00|  0.00|  0.02| 96.32
   1| 100.0| -0.00|  2833||  0.00|  0.00|  0.00|  0.00
   2|  9.06| 90.94|  1983||  0.00|  7.69|  6.98| 76.45
   3|  7.43| 92.57|  2039||  0.00|  2.60| 12.62| 77.52
     

Another useful tool for monitoring system power consumption is powerTOP. It helps you to identify the reasons for unnecessary high power consumption (for example, processes that are mainly responsible for waking up a processor from its idle state) and to optimize your system settings to avoid these. It supports both Intel and AMD processors.

powerTOP combines various sources of information (analysis of programs, device drivers, kernel options, amounts and sources of interrupts waking up processors from sleep states) and shows them in one screen. Example 11.4, “Example powerTOP Output” shows which information categories are available:

Cn               Avg  residency       P-states   (frequencies) 
                                                      
C0 (cpu running)        (11.6%)       2.00 Ghz       0.1%
polling         0.0ms   ( 0.0%)       2.00 Ghz       0.0%
C1              4.4ms   (57.3%)       1.87 Ghz       0.0%
C2             10.0ms   (31.1%)       1064 Mhz      99.9%
     
     
Wakeups-from-idle per second : 11.2     interval: 5.0s 
no ACPI power usage estimate available 


Top causes for wakeups: 
96.2% (826.0)       <interrupt> : extra timer interrupt
 0.9% (  8.0)     <kernel core> : usb_hcd_poll_rh_status (rh_timer_func)
 0.3% (  2.4)       <interrupt> : megasas
 0.2% (  2.0)     <kernel core> : clocksource_watchdog (clocksource_watchdog)
 0.2% (  1.6)       <interrupt> : eth1-TxRx-0
 0.1% (  1.0)       <interrupt> : eth1-TxRx-4
     
[...]
    
Suggestion:  Enable SATA ALPM link power management via:
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
or press the S key.

For more information, refer to the powerTOP project page at http://www.lesswatts.org/projects/powertop/. It also provides tips and tricks and an informative FAQ section.

The CPUfreq subsystem offers several tuning options for P-states: You can switch between the different governors, influence minimum or maximum CPU frequency to be used or change individual governor parameters.

To switch to another governor at runtime, use cpupower frequency-set with the -g option. For example, running the following command (as root) will activate the on-demand governor:

cpupower frequency-set -g ondemand

If you want the change in governor to persist also after a reboot or shutdown, use the pm-profiler as described in Section 11.5, “Creating and Using Power Management Profiles”.

To set values for the minimum or maximum CPU frequency the governor may select, use the -d or -u option, respectively.

Apart from the governor settings that can be influenced with cpupower or cpufreq*, you can also tune further governor parameters manually, for example, Ignoring Nice Values in Processor Utilization.

	Using the Same Value for All Cores
When setting the ignore_nice_load value for cpu0, the same value is automatically used for all cores. In this case, you do not need to repeat the steps above for each of the processors where you want to modify this governor parameter.

Another parameter that significantly impacts the performance loss caused by dynamic frequency scaling is the sampling rate (rate at which the governor checks the current CPU load and adjusts the processor's frequency accordingly). Its default value depends on a BIOS value and it should be as low as possible. However, in modern systems, an appropriate sampling rate is set by default and does not need manual intervention.

By default, openSUSE uses C-states appropriately. The only parameter you might want to touch for optimization is the sched_mc_power_savings scheduler. Instead of distributing a work load across all cores with the effect that all cores are utilized only at a minimum level, the kernel can try to schedule processes on as few cores as possible so that the others can go idle. This helps to save power as it allows some processors to be idle for a longer time so they can reach a higher C-state. However, the actual savings depend on a number of factors, for example how many processors are available and which C-states are supported by them (especially deeper ones such as C3 to C6).

If sched_mc_power_savings is set to 0 (default value), no special scheduling is done. If it is set to 1, the scheduler tries to consolidate the work onto the fewest number of processors possible in the case that all processors are a little busy. To modify this parameter, proceed as follows: