Unit OS11: Performance Evaluation Unit OS11: Performance Evaluation

(1)

Unit OS11: Performance Evaluation Unit OS11: Performance Evaluation

11.1. System Performance 11.1. System Performance

(2)

Copyright Notice Copyright Notice

These materials are part of the

These materials are part of the Windows Operating Windows Operating System Internals Curriculum Development Kit,

System Internals Curriculum Development Kit, developed by David A. Solomon and Mark E.

developed by David A. Solomon and Mark E.

Russinovich with Andreas Polze Russinovich with Andreas Polze

Microsoft has licensed these materials from David Microsoft has licensed these materials from David Solomon Expert Seminars, Inc. for distribution to Solomon Expert Seminars, Inc. for distribution to academic organizations solely for use in academic academic organizations solely for use in academic environments (and not for commercial use)

environments (and not for commercial use)

(3)

Roadmap for Section 11.1 Roadmap for Section 11.1

Performance Evaluation and Prediction Performance Evaluation and Prediction Tools for Monitoring Windows Internals Tools for Monitoring Windows Internals

Performance Monitor and mmc Performance Monitor and mmc

Scheduling-related Performance Counters Scheduling-related Performance Counters

Memory-related Performance Counters Memory-related Performance Counters

Windows Event Tracing

(4)

Performance Prediction and Performance Prediction and

Evaluation Evaluation

Constructing a model of the system and then Constructing a model of the system and then

using the model to predict the system's behavior using the model to predict the system's behavior

Model reflects system structure or organization as Model reflects system structure or organization as

well as its workload or input well as its workload or input

Analyzed using mathematical techniques Analyzed using mathematical techniques

Alternatively, the model may be simulated Alternatively, the model may be simulated

Benchmarking & Monitoring Benchmarking & Monitoring

Evaluating behavior of a live system Evaluating behavior of a live system

Predefined workloads Predefined workloads

(5)

Modeling Approaches Modeling Approaches

Analytic modeling techniques Analytic modeling techniques

Discrete- and continuous-time Markov chains Discrete- and continuous-time Markov chains Queueing theory, and queueing networks

Queueing theory, and queueing networks

Approximate methods based on these techniques Approximate methods based on these techniques

Operational analysis Operational analysis

Non-stochastic, measurement-based perspective to the Non-stochastic, measurement-based perspective to the analysis of computer systems

analysis of computer systems

Modeled with discrete-event simulation Modeled with discrete-event simulation

Performance metrics from stochastic simulations are Performance metrics from stochastic simulations are subject to statistical analysis (as are data obtained from subject to statistical analysis (as are data obtained from real systems)

real systems)

(6)

Validity of Models Validity of Models

Models, whether analytic or simulation, can be Models, whether analytic or simulation, can be

inaccurate or implemented incorrectly inaccurate or implemented incorrectly

An important aspect of any kind of performance An important aspect of any kind of performance

modeling study is to validate the model and its modeling study is to validate the model and its

implementation to whatever extent is possible implementation to whatever extent is possible

One way to do this is to study a system using more One way to do this is to study a system using more

than one model, e.g., a simulation model and an than one model, e.g., a simulation model and an

analytic model analytic model

Analytic modeling of many systems is Analytic modeling of many systems is

computationally demanding

(7)

Monitoring Windows - Monitoring Windows -

How to obtain Performance Data How to obtain Performance Data

Windows is thoroughly instrumented Windows is thoroughly instrumented

Performance counters allow for monitoring of most kernel objects Performance counters allow for monitoring of most kernel objects

Many tools available to dig into Windows internals Many tools available to dig into Windows internals Helps to see internals behavior “in action”

Helps to see internals behavior “in action”

Several sources of tools Several sources of tools

Support Tools Support Tools

Resource Kit Tools Resource Kit Tools Debugging Tools Debugging Tools Sysinternals.com Sysinternals.com

Additional tool packages with internals information Additional tool packages with internals information

Platform Software Development Kit (SDK) Platform Software Development Kit (SDK) Device Driver Development Kit (DDK) Device Driver Development Kit (DDK)

(8)

Tool Image Name Origin

File Monitor FILEMON www.sysinternals.com

Global Flags GFLAGS Support Tools

Handle Viewer HANDLE www.sysinternals.com

Kernel debuggers WINDBG, KD Debugging tools, Platform SDK, Windows DDK

Live Kernel Debugging LIVEKD www.sysinternals.com

Open Handles OH Resource kits

Page Fault Monitor PFMON Support Tools, Resource kits, Platform SDK

Pending File Moves PENDMOVES www.sysinternals.com Performance tool PERFMON.MSC Windows built-in tool

Pool Monitor POOLMON Support Tools, Windows DDK

Process Explorer PROCEXP www.sysinternals.com

Process Statistics PSTAT Support Tools, Windows 2000 Resource kits, Platform SDK, www.reskit.com

Quick Slice QSLICE Windows 2000 resource kits

Task (Process) List TLIST Debugging tools

Task Manager TASKMGR Windows built-in tool

TDImon TDIMON www.sysinternals.com

Tools for Windows Performance Tools for Windows Performance

Monitoring

(9)

Process Explorer (Sysinternals) Process Explorer (Sysinternals)

Shows performance-related data Shows performance-related data

……plus full image path, command line, plus full image path, command line, environment variables, parent process, environment variables, parent process, security access token, open handles, security access token, open handles, loaded DLLs & mapped files

loaded DLLs & mapped files

(10)

Obtain System Information Obtain System Information

with Process Explorer with Process Explorer

Click View->System Information Click View->System Information

(11)

Overview of Performance Data Overview of Performance Data

Collection Collection

Windows defines performance data in terms of objects, Windows defines performance data in terms of objects,

counters, and instances counters, and instances

A performance object is any resource, application, or service that can A performance object is any resource, application, or service that can be measured

be measured

System Monitor and Performance Logs and Alerts allow to select System Monitor and Performance Logs and Alerts allow to select performance objects, counters, and instances to collect and present performance objects, counters, and instances to collect and present performance data

performance data

Objects have performance counters Objects have performance counters

Objects may also have instances, which are unique copies of a Objects may also have instances, which are unique copies of a particular object type

particular object type

Not all object types support multiple instances Not all object types support multiple instances

_Total instance represents the sum of the values for all instances of _Total instance represents the sum of the values for all instances of the object for a specific counter

the object for a specific counter

(12)

Vast Array of Performance Data

(13)

Performance Counter Aggregation Performance Counter Aggregation

into Performance Logs (via mmc)

(14)

Real-time Data Collection with Real-time Data Collection with

Performance Monitor

(15)

Windows Performance Counters - Windows Performance Counters -

Categories Categories

Monitoring Memory Management Monitoring Memory Management

Memory\ Page Reads/sec Memory\ Page Reads/sec Memory\ Page Writes/sec Memory\ Page Writes/sec

Memory\ Available Bytes Memory\ Available Bytes

Process\ Working Set Process\ Working Set

Process\ Private Bytes Process\ Private Bytes

(16)

Windows Performance Counters - Windows Performance Counters -

Categories (contd.) Categories (contd.)

Monitoring Physical and Logical Disk I/O Monitoring Physical and Logical Disk I/O

PhysicalDisk\ % Disk Time PhysicalDisk\ % Disk Time

PhysicalDisk\ Avg. Disk Queue Length PhysicalDisk\ Avg. Disk Queue Length PhysicalDisk\ Current Disk Queue Length PhysicalDisk\ Current Disk Queue Length PhysicalDisk\ Avg. Disk Sec/Read

PhysicalDisk\ Avg. Disk Sec/Read PhysicalDisk\ Avg. Disk Sec/Write PhysicalDisk\ Avg. Disk Sec/Write PhysicalDisk\ Disk Read Bytes/sec PhysicalDisk\ Disk Read Bytes/sec PhysicalDisk\ Disk Write Bytes/sec PhysicalDisk\ Disk Write Bytes/sec PhysicalDisk\ Avg. Disk Bytes/Read PhysicalDisk\ Avg. Disk Bytes/Read PhysicalDisk\ Avg. Disk Bytes/Write PhysicalDisk\ Avg. Disk Bytes/Write PhysicalDisk\ Disk Reads/sec

PhysicalDisk\ Disk Reads/sec PhysicalDisk\ Disk Writes/sec PhysicalDisk\ Disk Writes/sec

(17)

Windows Performance Counters - Windows Performance Counters -

Categories (contd.) Categories (contd.)

Monitoring Network Activities Monitoring Network Activities

Network Interface\ Bytes Total/sec Network Interface\ Bytes Total/sec Network Interface\ Bytes Sent/sec Network Interface\ Bytes Sent/sec

Network Interface\ Bytes Received/sec Network Interface\ Bytes Received/sec

Protocol_layer_object\ Segments Received/sec Protocol_layer_object\ Segments Received/sec Protocol_layer_object\ Segments Sent/sec

Protocol_layer_object\ Segments Sent/sec Protocol_layer_object\ Frames Sent/sec Protocol_layer_object\ Frames Sent/sec

Protocol_layer_object\ Frames Received/sec Protocol_layer_object\ Frames Received/sec Server\ Bytes Total/sec

Server\ Bytes Total/sec

Server\ Bytes Received/sec Server\ Bytes Received/sec Server\ Bytes Sent/sec

Server\ Bytes Sent/sec

Network Segment\ % Network Utilization Network Segment\ % Network Utilization

(18)

Analyzing Processor Activity Analyzing Processor Activity

Determine the baseline on normal workload (from several Determine the baseline on normal workload (from several

weeks to a month) weeks to a month)

Processor\ % Processor Time counter Processor\ % Processor Time counter System\Processor Queue Length counter System\Processor Queue Length counter

Be aware of the Idle process … Be aware of the Idle process …

The Idle process runs a thread on each processor The Idle process runs a thread on each processor

To measure the Idle process, use the Process(Idle)\ % Processor To measure the Idle process, use the Process(Idle)\ % Processor Time counter, or Processes tab in Task Manager

Time counter, or Processes tab in Task Manager

Zero idle time could mean that the processor is handling a lot of Zero idle time could mean that the processor is handling a lot of work, but it could also mean that the processor or central

work, but it could also mean that the processor or central processing unit (CPU) is overloaded

processing unit (CPU) is overloaded

(19)

Detecting Processor Bottlenecks Detecting Processor Bottlenecks

CPU bottlenecks are indicated by:

Processor\ % Processor Time often exceeds 80 percent (and Processor\ % Processor Time often exceeds 80 percent (and

there is no compute-bound workload) there is no compute-bound workload)

System\ Processor Queue Length is often greater than 2 on a System\ Processor Queue Length is often greater than 2 on a

single-processor system single-processor system

Queue Length is the single most important parameter Queue Length is the single most important parameter

Other indications:

Unusually high values appear for the Processor(_Total)\

Interrupts/sec or System\ Context Switches/sec counters Interrupts/sec or System\ Context Switches/sec counters

(20)

Evaluating Memory Usage Evaluating Memory Usage

Establish a reference point (or baseline) for physical Establish a reference point (or baseline) for physical memory usage under normal workload

memory usage under normal workload

Create logs of memory usage over an extended period (from Create logs of memory usage over an extended period (from

several weeks to a month) several weeks to a month)

Relevant Performance Counters Relevant Performance Counters

\Memory\Available Bytes

\Paging File(_Total)\% Usage

Exclude spikes; the range of values that seem to appear Exclude spikes; the range of values that seem to appear consistently constitutes your baseline

consistently constitutes your baseline

(21)

Detecting Memory Bottlenecks Detecting Memory Bottlenecks

Indication for insufficient memory:

Value for

Value for Memory\Available BytesMemory\Available Bytes is consistently is consistently low (e.g. less than 5% of RAM)

low (e.g. less than 5% of RAM)

If available memory is consistently low, the If available memory is consistently low, the

computer becomes unresponsive:

It is occupied exclusively with disk I/O operations It is occupied exclusively with disk I/O operations

During paging due to low memory, the processor is During paging due to low memory, the processor is

idle while waiting for the disk to finish idle while waiting for the disk to finish

(22)

Examining Disk Performance Examining Disk Performance

Monitor disk counters along with counters from other Monitor disk counters along with counters from other

objects. The following is a list of recommended counters.

LogicalDisk\% Free Space LogicalDisk\% Free Space PhysicalDisk\Disk Reads/sec PhysicalDisk\Disk Reads/sec PhysicalDisk\Disk Writes/sec PhysicalDisk\Disk Writes/sec

PhysicalDisk\Avg. Disk Queue Length PhysicalDisk\Avg. Disk Queue Length Memory\Available Bytes

Memory\Available Bytes Memory\Cache Bytes Memory\Cache Bytes Memory\Pages/sec Memory\Pages/sec

Processor(All_Instances)\% Processor Time Processor(All_Instances)\% Processor Time System\Processor Queue Length

System\Processor Queue Length

(23)

Detecting a Disk Bottleneck Detecting a Disk Bottleneck

Avg. Disk Queue Length for LogicalDisk or PhysicalDisk Avg. Disk Queue Length for LogicalDisk or PhysicalDisk

If the value of Avg. Disk Queue Length exceeds twice the If the value of Avg. Disk Queue Length exceeds twice the

number of spindles, then you are likely developing a number of spindles, then you are likely developing a

bottleneck bottleneck

With a volume set, a queue that is never shorter than the With a volume set, a queue that is never shorter than the

number of active physical disks indicates that you are number of active physical disks indicates that you are

developing a bottleneck developing a bottleneck

Notice that this might overstate the true length of the queue, Notice that this might overstate the true length of the queue,

because the counter includes both queued and in-service because the counter includes both queued and in-service

requests requests

(24)

Counters by Feature Counters by Feature

Internet Information Service Internet Information Service

Active Server Pages Active Server Pages FTP Service

FTP Service Web Service Web Service

Internet Information Services Internet Information Services Global

Global

Indexing Service Indexing Service

Indexing Service Filter Indexing Service Filter HTTPHTTP Indexing Service Indexing Service Message Queuing

Message Queuing

MSMQ Session MSMQ Session MSMQ IS

MSMQ IS MSMQ Queue MSMQ Queue MSMQ Service MSMQ Service

Quality of Service (QoS)

Quality of Service (QoS) Admission Admission Control

Control

ACS/RSVP Service ACS/RSVP Service ACS/RSVP Interfaces ACS/RSVP Interfaces ACS/RSVP Policy ACS/RSVP Policy

Routing and Remote Access Routing and Remote Access

(RRAS) (RRAS)

RAS Port RAS Port RAS Total RAS Total

File Replication Service File Replication Service

FileReplicaConn FileReplicaConn FileReplicaSet FileReplicaSet

Terminal Service Terminal Service

Terminal Services Session Terminal Services Session

Active Directory™

NTDSNTDS

Windows services and apps may bring their Windows services and apps may bring their own performance objects

own performance objects

(25)

Kernel Event Tracing Kernel Event Tracing

Windows kernel and core device drivers are instrumented Windows kernel and core device drivers are instrumented

to record trace data to record trace data

Event Tracing for Windows (ETW) Event Tracing for Windows (ETW)

Common infrastructure in the kernel that provides trace data to Common infrastructure in the kernel that provides trace data to the user-mode facility

the user-mode facility

ETW is accessed by:

Controllers

Controllers that start and stop logging sessions and manages that start and stop logging sessions and manages buffer pools

buffer pools Providers

Providers that define GUIDs for the event classes they can that define GUIDs for the event classes they can produce traces for; act on Controllers’ commands

produce traces for; act on Controllers’ commands Consumer

Consumer select one or more trace sessions for which the select one or more trace sessions for which the want to read trace data (in real-time or in log files)

want to read trace data (in real-time or in log files)

(26)

ETW Providers and Kernel Logger ETW Providers and Kernel Logger

Windows Server systems include several built-in Windows Server systems include several built-in

providers in user mode providers in user mode

Active Directory, Kerberos, and Netlogon Active Directory, Kerberos, and Netlogon

ETW defines a logging session with the name NT ETW defines a logging session with the name NT Kernel Logger (kernel logger) for use by the kernel Kernel Logger (kernel logger) for use by the kernel

and core drivers.

NT Kernel Logger provider is implemented in the NT Kernel Logger provider is implemented in the

kernel kernel

(27)

User mode controllers may enable User mode controllers may enable

kernel logger - ETW Operation kernel logger - ETW Operation

ETW library sends I/O control request to the WMI driver to enable ETW library sends I/O control request to the WMI driver to enable tracing on a particular event class

tracing on a particular event class

ETW library is implemented in \Windows\System32\Ntdll.dll ETW library is implemented in \Windows\System32\Ntdll.dll

If file logging is configured the WMI driver creates a system thread in If file logging is configured the WMI driver creates a system thread in system process that creates a log file

system process that creates a log file

Alternatively, logging may use an in-memory buffer Alternatively, logging may use an in-memory buffer

The WMI driver records trace events to a buffer The WMI driver records trace events to a buffer

File logging thread wakes up once per second to dump the contents File logging thread wakes up once per second to dump the contents of the buffers to the log file

of the buffers to the log file

Trace records generated for the kernel logger have a standard ETW Trace records generated for the kernel logger have a standard ETW trace event header

trace event header

Header records timestamp, process, and thread IDs, info on event class Header records timestamp, process, and thread IDs, info on event class Event classes can provide additional data specific to their events

Event classes can provide additional data specific to their events

(28)

Trace and Logging Data may be Trace and Logging Data may be

accessed via mmc

(29)

Kernel Logger Trace Classes Kernel Logger Trace Classes

Traces classes and their generating components Traces classes and their generating components

Disk I/O - disk class driver Disk I/O - disk class driver File I/O - file system drivers File I/O - file system drivers

Hardware Configuration - plug&play manager Hardware Configuration - plug&play manager

Image Load/Unload - system image loader in the kernele Image Load/Unload - system image loader in the kernele Page Faults - memory manager

Page Faults - memory manager

Process Create/Delete - process manager Process Create/Delete - process manager Thread Create/Delete - process manager Thread Create/Delete - process manager Registry Activity - Configuration Manager Registry Activity - Configuration Manager TCP/UDP Activity - TCP/IP driver

TCP/UDP Activity - TCP/IP driver

ETW controllers/providers described in Platform SDK ETW controllers/providers described in Platform SDK

(30)

Using Event Logs and Performance Using Event Logs and Performance

Counters for Optimization Counters for Optimization

Performance must be tuned to a workload Performance must be tuned to a workload

A sequence of service requests, commands, I/Os that A sequence of service requests, commands, I/Os that

exercise the software exercise the software

Often produced by workload generators rather than Often produced by workload generators rather than

real-world service provision real-world service provision

Allow for replay or generation of service requests at a Allow for replay or generation of service requests at a maximum rate so that bottlenecks can be identified in maximum rate so that bottlenecks can be identified in

systems systems

Most workloads are domain-specific Most workloads are domain-specific

I.e.; TPC benchmarks and workloads I.e.; TPC benchmarks and workloads

No “optimal” system configuration per se No “optimal” system configuration per se

(31)

Source Code References Source Code References

Windows Research Kernel (WRK):

\base\ntos\perf – core performance logging support

\base\ntos\ex\sysinfo.c – system/process

\base\ntos\ex\sysinfo.c – system/process performance query functions

performance query functions

Unit OS11: Performance Evaluation Unit OS11: Performance Evaluation