RHEL6 CFS Scheduler can Cause High Latch Waits for 'latch: cache buffers chains' (文档 ID 1560594.1)

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.3 and later
Linux x86-64\

SYMPTOMS

• After upgrade from RHEL5 / OEL5 (i.e. 2.6.18 kernel versions) to RHEL6 / OEL6 (i.e. 2.6.32 kernel versions) it is observed that jobs/queries having a large number of waits for 'latch: cache buffers chains' perform more slowly than before.
• The total elapsed time is increased with no other AWR statistics evidence except a minor increase in latch waits which indicates that waits for CPU are occurring.

CHANGES

Upgrading from Linux 2.6.18 kernel versions to 2.6.32 kernel versions without any hardware or Oracle database changes.

CAUSE

It has been determined that the previous Linux O(1) scheduler of RHEL5 / OEL5 and 2.6.18 kernel versions allowed for a more intensive CPU usage while Oracle queries are acquiring "cache buffers chains" latches during buffer reads.

With RHE6 / OEL6 kernels the same type queries tested incurred higher latch statistics counters such as: gets, spin gets, misses and sleeps resulting in a drop in percentage of CPU usage (a 9x drop in CPU / elapsed time can be observed for some tested workloads). 

SOLUTION

You can resolve the issue by executing both of the following steps:
\

1. Increase the CPU time slice allocated by scheduler
   
   In other words, increase the sched_latency_ns (default is: 5ms * (1 + log2(min(ncpus,8)))) and sched_min_granularity_ns (default is: 1ms * (1 + log2(min(ncpus,8)))) to: 
   
   kernel.sched_latency_ns= 150000000 (150ms)
   kernel.sched_min_granularity_ns = 75000000 (75ms)
   
   You can change this as follows:\

   echo 150000000 > /proc/sys/kernel/sched_latency_ns 
   echo 75000000 > /proc/sys/kernel/sched_min_granularity_ns

2. Disable pre-emption of wakeup processes through a dynamic debugger:
   \

   mount -t debugfs debug /sys/kernel/debug 
   echo NO_WAKEUP_PREEMPT > /sys/kernel/debug/sched_features