Unit OS2: Unit OS2: Operating System Principles Operating System Principles

(1)

Unit OS2:

Operating System Principles Operating System Principles

2.5.2.5. DemosDemos

(2)

Copyright Notice Copyright Notice

These materials are part of the

These materials are part of the Windows Operating Windows Operating System Internals Curriculum Development Kit,

System Internals Curriculum Development Kit, developed by David A. Solomon and Mark E.

developed by David A. Solomon and Mark E.

Russinovich with Andreas Polze Russinovich with Andreas Polze

Microsoft has licensed these materials from David Microsoft has licensed these materials from David Solomon Expert Seminars, Inc. for distribution to Solomon Expert Seminars, Inc. for distribution to academic organizations solely for use in academic academic organizations solely for use in academic environments (and not for commercial use)

environments (and not for commercial use)

(3)

Roadmap for Section 2.5.

Demos investigating:

Process Execution Process Execution

Object Manager & Handles Object Manager & Handles

Interrupt Handling Interrupt Handling

Memory Pools Labs Memory Pools Labs

System Threads System Threads

System Processes

(4)

Lab: Examining Privileged vs. User Lab: Examining Privileged vs. User Time Time

1. Run MLTITHRD 1. Run MLTITHRD

Click on “Mltithrd->Bounce” menu item Click on “Mltithrd->Bounce” menu item

Observe system activity with Task Manager and Qslice Observe system activity with Task Manager and Qslice

2. Run CPUSTRES (in ResKit) 2. Run CPUSTRES (in ResKit)

Change thread 1 activity to “Maximum”

Observe system activity with Task Manager and Qslice Observe system activity with Task Manager and Qslice

(5)

Tools for Obtaining Process & Thread Tools for Obtaining Process & Thread

Information Information

Many overlapping tools (most show one item the others do not) Many overlapping tools (most show one item the others do not) Built-in tools in Windows 2000/XP:

Built-in tools in Windows 2000/XP:

Task Manager, Performance Tool Task Manager, Performance Tool Tasklist (new in XP)

Tasklist (new in XP)

Support Tools Support Tools

pviewer - process and thread details (GUI) pviewer - process and thread details (GUI) pmon - process list (character cell)

pmon - process list (character cell)

tlist - shows process tree and thread details (character cell) tlist - shows process tree and thread details (character cell)

Resource Kit tools:

apimon - system call and page fault monitoring (GUI) apimon - system call and page fault monitoring (GUI) oh – display open handles (character cell)

oh – display open handles (character cell)

pviewer - processes and threads and security details (GUI) pviewer - processes and threads and security details (GUI) ptree – display process tree and kill remote processes (GUI) ptree – display process tree and kill remote processes (GUI) pulist - lists processes and usernames (character cell)

pulist - lists processes and usernames (character cell)

pstat - process/threads and driver addresses (character cell) pstat - process/threads and driver addresses (character cell) qslice - can show process-relative thread activity (GUI)

qslice - can show process-relative thread activity (GUI)

Tools from www.sysinternals.com Tools from www.sysinternals.com

Process Explorer – super Task Manager – shows open files, loaded DLLs, security info, etc.

Pslist – list processes on local or remote systems Pslist – list processes on local or remote systems

(6)

How Process Explorer Works How Process Explorer Works

Uses undocumented functions for:

Enumerating loaded modules with full path names Enumerating loaded modules with full path names

Enumerating processes and handles Enumerating processes and handles

Obtains handle names using the aid of a driver Obtains handle names using the aid of a driver

Related Tools:

Handle – command-line handle viewer Handle – command-line handle viewer

Listdlls – command-line DLL viewer Listdlls – command-line DLL viewer

(7)

Process Explorer Lab: Refresh Process Explorer Lab: Refresh

Highlighting Highlighting

1. 1. Press space bar to pause refresh Press space bar to pause refresh

2. 2. Run Notepad Run Notepad

3. 3. In ProcExp, hit F5 and notice new process In ProcExp, hit F5 and notice new process

4. 4. Exit Notepad Exit Notepad

5. 5. In ProcExp, hit F5 and notice Notepad in red In ProcExp, hit F5 and notice Notepad in red

6. 6. Press space bar to resume normal refresh Press space bar to resume normal refresh Uses:

Uses:

Understanding process startup sequences Understanding process startup sequences

Detecting appearance of processes coming and

(8)

Process Explorer Lab: Column Process Explorer Lab: Column

Selection And Username Selection And Username

1. 1. Notice additional details show for each process Notice additional details show for each process (icon, description)

(icon, description)

2. 2. Click on View->Select Columns Click on View->Select Columns

 Add username columnAdd username column

3. 3. Compare username column in Task Manager Compare username column in Task Manager with Process Explorer – what is the difference?

with Process Explorer – what is the difference?

4. 4. Deselect View->Show Processes From All Deselect View->Show Processes From All

(9)

Process Explorer Lab: Command Process Explorer Lab: Command Line Line

1. 1. Double click on date/time in task bar (lower Double click on date/time in task bar (lower right of screen)

right of screen)

2. 2. In Process Explorer, hit F5 to refresh In Process Explorer, hit F5 to refresh

3. 3. Find new process created (RUNDLL32.EXE) Find new process created (RUNDLL32.EXE)

4. 4. Examine command line arguments Examine command line arguments

Example: cmd.exe process was consuming Example: cmd.exe process was consuming

lots of CPU time lots of CPU time

Command line argument showed which .BAT file Command line argument showed which .BAT file

was running was running

(10)

Process Explorer Lab: Process Process Explorer Lab: Process

Performance Statistics Performance Statistics

Click on Performance Tab of process properties Click on Performance Tab of process properties

Note: all these numbers can be configured as columns Note: all these numbers can be configured as columns

(11)

Examining CPU Time Examining CPU Time

Open process Open process properties and properties and

look at CPU usage look at CPU usage

history on the history on the

performance performance

graph page graph page

Hover the mouse Hover the mouse

over a point to see over a point to see

the time of that the time of that

value

(12)

TCP/IP Endpoints TCP/IP Endpoints

Process properties Process properties TCP/IP tab shows TCP/IP tab shows process’ TCP and process’ TCP and UDP endpoints

UDP endpoints

Resolves addresses Resolves addresses

in the background in the background

TCPView from TCPView from

Sysinternals shows all Sysinternals shows all

endpoints endpoints

(13)

Process Explorer Lab: Environment Process Explorer Lab: Environment

Variables Variables

 Click on Environment Tab of process Click on Environment Tab of process properties

properties

(14)

Process Explorer Lab: Environment Process Explorer Lab: Environment

Variables Variables

1. 1. Open a command prompt Open a command prompt

2. 2. Run Notepad.exe from command prompt Run Notepad.exe from command prompt

3. 3. Type “set abc=xyz” Type “set abc=xyz”

4. 4. In ProcExp, hit F5 and examine In ProcExp, hit F5 and examine

environment variables for Cmd.exe and environment variables for Cmd.exe and

Notepad.exe Notepad.exe

• Notice Notepad.exe does not have abc

(15)

Process Explorer: Thread Details Process Explorer: Thread Details

Process Explorer “Threads”

tab shows which thread(s) tab shows which thread(s) are running

are running

Start address represents Start address represents where the thread began where the thread began running (not where it is running (not where it is now)now)

Click Module to get Click Module to get details on module details on module

containing thread start containing thread start address

address

Can also kill threads Can also kill threads

May be useful in an ISAPI May be useful in an ISAPI process with a runaway process with a runaway

(16)

Thread Start Functions Thread Start Functions

Process Explorer can map the addresses within a Process Explorer can map the addresses within a

module to the names of functions module to the names of functions

This can help identify which component within a process is This can help identify which component within a process is responsible for CPU usage

responsible for CPU usage

Requires access to:

Symbol file for that module Symbol file for that module

Proper version of Dbghelp.dll (part of Windows Debugging Proper version of Dbghelp.dll (part of Windows Debugging Tools)

Tools)

Process Explorer looks for Process Explorer looks for

: :

(17)

Viewing Call Stacks Viewing Call Stacks

with Process with Process

Explorer Explorer

Click Stack to view call stack Click Stack to view call stack

Lists functions in reverse Lists functions in reverse chronological order

chronological order

Note that start address on Note that start address on

Threads tab is different than Threads tab is different than

first function shown in stack first function shown in stack

This is because all user This is because all user threads start in a Windows threads start in a Windows library function which calls library function which calls

(18)

Example: Solving Hung Processes Example: Solving Hung Processes

Problem: Powerpoint was hanging for 1 minute Problem: Powerpoint was hanging for 1 minute

on startup on startup

Thread stack shows waiting on a printer driver

(19)

Kernel mode code always uses the current thread’s kernel mode Kernel mode code always uses the current thread’s kernel mode stack

stack

Kernel stack attributes:

One for each thread One for each thread

Mapped in system address space Mapped in system address space Normally nonpageable

Normally nonpageable

Because kernel mode code might be running at dispatch level IRQL or above, in Because kernel mode code might be running at dispatch level IRQL or above, in which context page faults will cause a crash

which context page faults will cause a crash

GDI requests a larger kernel stack size:

MmGrowKernelStack( stackPointer );

Might not succeed Might not succeed

Not documented for drivers; used internally by GDI Not documented for drivers; used internally by GDI

Kernel Mode Stack

(20)

Suspending Processes Suspending Processes

Process Explorer can suspend a process Process Explorer can suspend a process Why would you want to do this?

Why would you want to do this?

You’ve started a long running job but want to pause You’ve started a long running job but want to pause

it to do something else it to do something else

Lowering the priority still leaves it running…

You’ve started a long download but want to have You’ve started a long download but want to have

your network bandwidth temporarily your network bandwidth temporarily

Some multi-service system process activity is due Some multi-service system process activity is due

to other processes calling upon their services to other processes calling upon their services

(21)

Process Explorer Lab: Suspend Process Explorer Lab: Suspend

Start Notepad Start Notepad

From a command prompt:

1. 1. Suspend Notepad process with Process Suspend Notepad process with Process Explorer

Explorer

2. 2. Try to switch back to Notepad (should not Try to switch back to Notepad (should not respond)

respond)

3. 3. Resume Notepad Resume Notepad

(22)

PS Tools PS Tools

PsFile – lists & closes remote file opens PsFile – lists & closes remote file opens

PsShutdown – remote shutdown, lock workstation, log off user PsShutdown – remote shutdown, lock workstation, log off user

PsExec – run an app on a remote system PsExec – run an app on a remote system

PsList – list processes & threads PsList – list processes & threads

PsUptime – system up time PsUptime – system up time

PsInfo – display general system info PsInfo – display general system info

PsGetsid – displays computer or user SIDs PsGetsid – displays computer or user SIDs

PsService – service process control (like SC in XP) PsService – service process control (like SC in XP)

PsLoglist – dumps event log in text PsLoglist – dumps event log in text

PsSuspend – suspend a process PsSuspend – suspend a process

(23)

HAL Choices HAL Choices

To see the HAL list, do an “update driver” on the drivers To see the HAL list, do an “update driver” on the drivers for the “Computer” and specify manual selection from the for the “Computer” and specify manual selection from the list;

list;

(24)

Variations of Routine Names…

Private versions of public routines Private versions of public routines

Both public (exported) and private entry points may exist Both public (exported) and private entry points may exist

Private version is not callable outside of the module that defines Private version is not callable outside of the module that defines themthem

Basic routine name has “p” added to the end of its prefix Basic routine name has “p” added to the end of its prefix

IopCallDriver => private version of IoCallDriver IopCallDriver => private version of IoCallDriver

Public routine may simply be private routine with name redefined Public routine may simply be private routine with name redefined

Internal routines Internal routines

Not callable outside of the defining module – and no public version Not callable outside of the defining module – and no public version But may be invoked by other means (traps, interrupts)

But may be invoked by other means (traps, interrupts)

(25)

Native Images Native Images

.EXEs not linked against any subsystem .EXEs not linked against any subsystem

Interface to NT executive routines directly via Interface to NT executive routines directly via NTDLL.DLL

NTDLL.DLL

Two examples:

smss.exe

smss.exe (Session Manager -- starts before (Session Manager -- starts before subsystems start)

subsystems start) csrss.exe

csrss.exe (Windows subsystem)(Windows subsystem)

(26)

Examining Open Handles: MS Tools Examining Open Handles: MS Tools

Resource Kit “oh” (Open Handles) tool Resource Kit “oh” (Open Handles) tool

Can show named & unnamed handles (“-a” switch) Can show named & unnamed handles (“-a” switch)

Can select by object type (e.g. “oh -t file” shows all open files) Can select by object type (e.g. “oh -t file” shows all open files) Does not show full path name

Does not show full path name

XP & 2003: openfiles /query command XP & 2003: openfiles /query command

Both of these require the “maintain a list of objects for each type” NT “global flag” registry bit to Both of these require the “maintain a list of objects for each type” NT “global flag” registry bit to be set

be set

Oh turns this on for you (or you can run Gflags.exe) Oh turns this on for you (or you can run Gflags.exe) Requires

Requires reboot to take effectreboot to take effect

See HKEY_LOCAL_MACHINE\System\CurrentControlSet See HKEY_LOCAL_MACHINE\System\CurrentControlSet

(27)

Increased System Memory Limits Increased System Memory Limits

Key system memory limits raised in XP & Server 2003 Key system memory limits raised in XP & Server 2003 Windows 2000 limit of 200 GB of mapped file data

Windows 2000 limit of 200 GB of mapped file data eliminated

eliminated

Previously limited size of files that could be backed up Previously limited size of files that could be backed up

Maximum System Page Table Entries (PTEs) increased Maximum System Page Table Entries (PTEs) increased

Can now describe 1.3 GB of system space (960 MB Can now describe 1.3 GB of system space (960 MB

contiguous) contiguous)

Windows 2000 limit was 660 MB (220 MB contiguous) Windows 2000 limit was 660 MB (220 MB contiguous)

Increases number of users on Terminal Servers Increases number of users on Terminal Servers

Also means maximum device driver size is now 960 MB (was Also means maximum device driver size is now 960 MB (was

220 MB) 220 MB)

(28)

Monitoring Pool Usage Monitoring Pool Usage

Poolmon.exe (Support Tools) Poolmon.exe (Support Tools)

Shows paged and nonpaged pool consumption by data structure “tag”

Must first turn on “pool tagging” with Resource Kit gflags tool & reboot Must first turn on “pool tagging” with Resource Kit gflags tool & reboot

On by default in Windows Server 2003 (not in XP or Win2000) On by default in Windows Server 2003 (not in XP or Win2000)

(29)

Finding All the Drivers Finding All the Drivers

Note that while most drivers are in Note that while most drivers are in

\Windows\System32\Drivers, they can be loaded from

\Windows\System32\Drivers, they can be loaded from anywhere

anywhere

To check the location of all drivers:

Run Msinfo32.exe, click on Software Environment->System Drivers, Run Msinfo32.exe, click on Software Environment->System Drivers, sort by Path

sort by Path

Or, type “Driverquery /v” (XP & 2003) Or, type “Driverquery /v” (XP & 2003)

Or view loaded DLL list of System process with Process Explorer Or view loaded DLL list of System process with Process Explorer

However, some drivers are deleted after they are loaded However, some drivers are deleted after they are loaded

Binary file and registry key can be deleted after load Binary file and registry key can be deleted after load Examples: Process Explorer, Filemon, Regmon Examples: Process Explorer, Filemon, Regmon

To list all loaded modules, run Drivers.exe or type “lm k” in To list all loaded modules, run Drivers.exe or type “lm k” in

Kernel Debugger (note: only works with LiveKd) Kernel Debugger (note: only works with LiveKd)

(30)

Pool Usage with Kernel Debugger Pool Usage with Kernel Debugger

!poolused

!poolused 1 [pooltag]

!poolused 1 [pooltag] Show full detailsShow full details

!poolused 2 [pooltag] Show nonpaged pool Show nonpaged pool sorted by usage

sorted by usage

!poolused 4 [pooltag] Show paged pool sorted Show paged pool sorted by usage

by usage

!poolused 8 [pooltag] Show session pool usageShow session pool usage

Pooltag specified can have wildcards (* or ?)

(31)

Troubleshooting Pool Leaks With Troubleshooting Pool Leaks With

Verifier Verifier

Use Driver Verifier (described in the I/O section) Use Driver Verifier (described in the I/O section)

Enable pool tracking for driver(s) of interest Enable pool tracking for driver(s) of interest

Causes system to track pool usage by driver Causes system to track pool usage by driver

vs Poolmon, which looks at pool usage by structure tag vs Poolmon, which looks at pool usage by structure tag

Reboot and monitor pool usage of driver Reboot and monitor pool usage of driver

Use “verifier /log file.txt /interval nnn” to store the output in a text Use “verifier /log file.txt /interval nnn” to store the output in a text filefile

Check to see if it’s going up…

(32)

Troubleshooting Pool Leaks With Troubleshooting Pool Leaks With

Verifier Verifier

A leaker exhibits the following A leaker exhibits the following

Current allocations is always close to or equal to the peak Current allocations is always close to or equal to the peak The peak grows over time

The peak grows over time

If the leak is significant the peak allocations or bytes will be If the leak is significant the peak allocations or bytes will be large

large

GUI interface to view usage:

(33)

Services Services

How do services interact with the system?

Must register with service control manager when started Must register with service control manager when started (otherwise process is killed)

(otherwise process is killed)

Get startup configuration parameters from Registry Get startup configuration parameters from Registry Log errors to Windows 2000 Event Log

Log errors to Windows 2000 Event Log

Use some form of IPC mechanism for client communication and Use some form of IPC mechanism for client communication and control

control

Likely make use of Win2K security impersonation Likely make use of Win2K security impersonation

Service implementation Service implementation

One .EXE may have >1 service (type code in Registry indicates) One .EXE may have >1 service (type code in Registry indicates)

Examples of services installed by default Examples of services installed by default

Event Log, Task Scheduler Event Log, Task Scheduler

Examples of add-on services Examples of add-on services

(34)

Service Control Tools Service Control Tools

Net start/stop – local system only Net start/stop – local system only

Sc.exe (built in to XP/2003; also in Win2000 Resource Kit) Sc.exe (built in to XP/2003; also in Win2000 Resource Kit)

Command line interface to

Command line interface to all service control/configuration all service control/configuration functions

functions

Works on local or remote systems Works on local or remote systems

Psservice (Sysinternals) – similar to SC Psservice (Sysinternals) – similar to SC

Other tools in Resource Kit Other tools in Resource Kit

Instsrv.exe – install/remove services (command line) Instsrv.exe – install/remove services (command line) Srvinstw.exe – install/remove services (GUI)

Srvinstw.exe – install/remove services (GUI)

Why are service creation tools included in Reskit?

(35)

Understanding Svchost.exe Understanding Svchost.exe

CPU Time Consumption CPU Time Consumption

If a multi-service process or other multi- If a multi-service process or other multi-

component process such Inetinfo.exe component process such Inetinfo.exe

(IIS) or Dllhost.exe (COM) is consuming (IIS) or Dllhost.exe (COM) is consuming CPU time, how do you determine which CPU time, how do you determine which

service is responsible?

Need to drill down to thread granularity Need to drill down to thread granularity

Go to Threads tab in Process Explorer and Go to Threads tab in Process Explorer and

sort by CPU usage

(36)

Properties of a Service Properties of a Service

General tab General tab

Logon tab Logon tab

Recovery tab Recovery tab

Dependencies tab

(37)

Lab: Minimal Process Set Lab: Minimal Process Set

Run Process Explorer Run Process Explorer

Kill Smss.exe (must do this first!) Kill Smss.exe (must do this first!)

Then right click on Winlogon and end process Then right click on Winlogon and end process tree tree

Kill all other processes except Process Explorer Kill all other processes except Process Explorer

From Process Explorer, you can still run From Process Explorer, you can still run

programs programs

E.g. cmd.exe, iexplore.exe, etc.