Unit OS3: Concurrency Unit OS3: Concurrency

(1)

Unit OS3: Concurrency Unit OS3: Concurrency

3.5.3.5. Lab Slides & Lab ManualLab Slides & Lab Manual

(2)

Copyright Notice Copyright Notice

These materials are part of the

These materials are part of the Windows Operating Windows Operating System Internals Curriculum Development Kit,

System Internals Curriculum Development Kit, developed by David A. Solomon and Mark E.

developed by David A. Solomon and Mark E.

Russinovich with Andreas Polze Russinovich with Andreas Polze

Microsoft has licensed these materials from David Microsoft has licensed these materials from David Solomon Expert Seminars, Inc. for distribution to Solomon Expert Seminars, Inc. for distribution to academic organizations solely for use in academic academic organizations solely for use in academic environments (and not for commercial use)

environments (and not for commercial use)

(3)

Roadmap for Section 3.5.

Lab experiments investigating:

Viewing the interrupt dispatch table Viewing the interrupt dispatch table

Viewing configuration of programmable interrupt Viewing configuration of programmable interrupt controller (PIC/APIC)

controller (PIC/APIC)

Viewing the interrupt request level (IRQL) on Windows Viewing the interrupt request level (IRQL) on Windows Monitoring Interrupt and DPC activity

Monitoring Interrupt and DPC activity Viewing System Service Activity

Viewing System Service Activity Viewing Global Queued Spinlocks Viewing Global Queued Spinlocks Looking at Wait Queues

Looking at Wait Queues

(4)

x86 Interrupt Controllers - x86 Interrupt Controllers -

Hardware Interrupt Processing Hardware Interrupt Processing

Most x86 systems rely on Most x86 systems rely on

i8259A Programmable Interrupt Controller (PIC) or i8259A Programmable Interrupt Controller (PIC) or

a variant of the i82489 Advanced Programmable a variant of the i82489 Advanced Programmable Interrupt Controller (APIC) - most new computers Interrupt Controller (APIC) - most new computers

PICs work only with uniprocessor systems PICs work only with uniprocessor systems

APICs work with multiprocessor systems APICs work with multiprocessor systems

Lab: Observe PIC / APIC configuration Lab: Observe PIC / APIC configuration

UseUse !pic !pic and and !apic!apic kernel debugger commands kernel debugger commands

(5)

Viewing the IRQL on Windows Viewing the IRQL on Windows

On Windows Server 2003, kernel debugger displays On Windows Server 2003, kernel debugger displays IRQL:

IRQL:

!irql debugger command:

kd> !irql kd> !irql

Debugger saved IRQL for processor 0x0 -- 0 (LOW_LEVEL) Debugger saved IRQL for processor 0x0 -- 0 (LOW_LEVEL)

Processor control region (PCR) and processor control Processor control region (PCR) and processor control block (PRCB) store:

block (PRCB) store:

current IRQL, current IRQL,

pointer to the hardware IDT, pointer to the hardware IDT,

currently running thread, currently running thread,

next thread selected to run.

(6)

Lab: Viewing IRQL/IRQ Assignments Lab: Viewing IRQL/IRQ Assignments

1.1. Display the interrupt vectorDisplay the interrupt vector

 XP/2003: !idtXP/2003: !idt

 Win2000: !kdex2x86.idtWin2000: !kdex2x86.idt

2.2. Dump the KINTERRUPT block for the PS/2 mouse ISR to get the Dump the KINTERRUPT block for the PS/2 mouse ISR to get the IRQLIRQL

(Dt nt!_KINTERRUPT xxxxxx) (Dt nt!_KINTERRUPT xxxxxx)

3.3. With Device Manager, go to the mouse device properties and click With Device Manager, go to the mouse device properties and click on the resources tab to see the IRQ

on the resources tab to see the IRQ

 If you are on a uniprocessor system, the IRQ should be the 27-IRQLIf you are on a uniprocessor system, the IRQ should be the 27-IRQL

Note: IRQL is raised when breaking in with debugger or on a crash Note: IRQL is raised when breaking in with debugger or on a crash

 !pcr displays this changed IRQL!pcr displays this changed IRQL

 !irql displays previous IRQL (Server 2003 & later)!irql displays previous IRQL (Server 2003 & later)

(7)

Lab: Kernel Profiling Lab: Kernel Profiling

Since time spent at DPC level and above is not accounted by driver type, Since time spent at DPC level and above is not accounted by driver type, one way to determine where time has been spent in kernel mode is by one way to determine where time has been spent in kernel mode is by using a

using a profiling/sampling profiling/sampling tooltool Kernrate is a such a tool

Kernrate is a such a tool

Free download from

Free download from http://www.http://www.microsoftmicrosoft.com/.com/whdcwhdc/system//system/sysperfsysperf//krviewkrview.. mspxmspx

Can be used both for kernel time and user mode processes Can be used both for kernel time and user mode processes Can show where time is being spent down to the function level Can show where time is being spent down to the function level

May miss short lived events or events close to the sampling interval May miss short lived events or events close to the sampling interval

Lab:Lab:

Download and install Kernrate Download and install Kernrate

cd c:\program files\krview\kernrates cd c:\program files\krview\kernrates

Kernrate_i386_XP.exe -z ntoskrnl.exe –j srv*c:\symbols Kernrate_i386_XP.exe -z ntoskrnl.exe –j srv*c:\symbols

Perform some system activity (run Windows Media Player, drag windows Perform some system activity (run Windows Media Player, drag windows around, etc)

around, etc)

(8)

Flow of Interrupts Flow of Interrupts

CPU Interrupt Controller

CPU Interrupt

Service Table

ISR Address Spin Lock

Dispatch Code

Peripheral Device Controller

0 2 3

n

Raise IRQL

Lower IRQL

Read from device Acknowledge- Interrupt Request DPC

Interrupt KiInterruptDispatch Driver ISR

Grab Spinlock Drop Spinlock

(9)

Lab: ISR/DPC Tracing Lab: ISR/DPC Tracing

XP SP2 and Server 2003 SP1 and later support tracing ISRs and XP SP2 and Server 2003 SP1 and later support tracing ISRs and DPCsDPCs

1. Start capturing events (tracelog.exe is in Support Tools):

tracelog -start -f kernel.etl -b 64 -UsePerfCounter -eflag 8 0x307 tracelog -start -f kernel.etl -b 64 -UsePerfCounter -eflag 8 0x307 0x4084 0 0 0 0 0 0

0x4084 0 0 0 0 0 0 2. Stop capturing events:

2. Stop capturing events:

tracelog -stop tracelog -stop

3. Generate reports (tracerpt.exe is part of Windows):

tracerpt kernel.etl -df –report -o tracerpt kernel.etl -df –report -o

4. Review workload.txt to determine where ISR/DPC time spent 4. Review workload.txt to determine where ISR/DPC time spent

5. Open "dumpfile.csv" & search for lines with "DPC" or "ISR" in the 5. Open "dumpfile.csv" & search for lines with "DPC" or "ISR" in the

second value. In kernel debugger, do an “ln” on 8

second value. In kernel debugger, do an “ln” on 8^th^th argument (start argument (start address)

address)

(10)

Try to acquire spinlock:

Test, set, was set, loop Test, set, was set, loop Test, set, was set, loop Test, set, was set, loop Test, set, WAS CLEAR (got the spinlock!) Begin updating data Try to acquire spinlock:

Test, set, WAS CLEAR (got the spinlock!) Begin updating data that’s protected by the spinlock

(done with update) Release the spinlock:

Clear the spinlock bit Try to acquire spinlock:

Test, set, WAS CLEAR (got the spinlock!) Begin updating data that’s protected by the spinlock

(done with update) Release the spinlock:

Clear the spinlock bit

Spinlocks in Action Spinlocks in Action

CPU 1 CPU 2

(11)

Looking at Waiting Threads Looking at Waiting Threads

For waiting threads, user-mode utilities only display the wait reason For waiting threads, user-mode utilities only display the wait reason Example: pstat

Example: pstat

To find out

To find out whatwhat a thread is waiting on, must use kernel debugger a thread is waiting on, must use kernel debugger

(12)

Looking at Wait Queues Looking at Wait Queues

!thread command to kernel debugger

Lists addresses of objects being waited on (if a mutex, shows owner) Lists addresses of objects being waited on (if a mutex, shows owner)

!irpfind can search IRPs for an event object address