Compaq AA-RH99A-TE Manual de usuario

Tru64 UNIXKernel DebuggingPart Number: AA-RH99A-TEJuly 1999Product Version: Tru64 UNIX Version 5.0 or higherThis manual explains how to use tools to d

%$A percent sign represents the C shell system prompt.A dollar sign represents the system prompt for theBourne, Korn, and POSIX shells.#A number sign

4Crash Analysis ExamplesFinding problems in crash dump files is a task that takes practice andexperience to do well. Exactly how you determine what ca

related data structures and functions that appear earlier in the stack.An earlier function might have passed corrupt data to the functionthat caused a

c":753, 0xfffffc00003c4ae4]1 panic(s = 0xfffffc000044b618 = "mode = 0%o, inum = %d, pref = %d fs = %s\n")\["../../../../src/kernel

c3200](kdbx) qdbx (pid 29939) died. Exiting...1 Use the sum command to get a summary of the system.2 Display the panic string (panicstr).3 Perform a s

4, 0xfffffc00003e0c78]3 _XentMM() ["/usr/sde/alpha/build/alpha.nightly/src/kernel/arch/alpha/locore.\s":702, 0xfffffc00003d4ff4](dbx) kps5PI

(dbx) px savedefp[28]80xfffffc000032972c(dbx) savedefp[28]/i9[nfs_putpage:2344, 0xfffffc000032972c] ldl r5, 40(r1)(dbx) savedefp[23]/i10[ubc_invalidat

11 Point the dbx debugger to the nfs_putpage function.12 Display the name of the source file that contains the nfs_putpagefunction.13 Point the dbx de

4.4 Finding a Panic String in a Thread Other Than theCurrent ThreadThe dbx and kdbx debuggers have the concept of the current thread. Inmany cases, wh

Source not availablethread 0x8d42dd70 stopped at [thread_block:1289 +0x18,0xfffffc00003394b8] \Source not available(dbx)tset 0x8d42f5d04thread 0x8d42f

1Introduction to Kernel DebuggingKernel debugging is a task normally performed by systems engineers writingkernel programs. A kernel program is one th

% dbx -k ./vmunix.1 ./vmzcore.1dbx version 5.0Type ’help’ for help.stopped at [boot:1494 ,0xfffffc0000442918] Source not available(dbx) p ustsname1str

11 pgrp_ref(0xffffffffa52c6000, 0x0, 0xfffffc000023ee20, 0x6b7, 0xfffffc000\05e1080) ["../../../../src/kernel/bsd/kern_proc.c":561, 0xfffffc

panic function were performed after the system was corrupt andduring an attempt to save data. Normally, any events that occur afterthe initial call to

AOutput from the crashdc CommandThis appendix contains a sample crash-data.n file created by thecrashdc command (using a compressed crash-dump file, v

asc1 at tcds0 slot 1rz8 at asc1 bus 1 target 0 lun 0 (DEC RZ57 (C) DEC 5000)rz9 at asc1 bus 1 target 1 lun 0 (DEC RZ56 (C) DEC 0300)fb0 at tc0 slot 81

_proc_thread_list_end:_dump_begin:7> 0 boot(reason = 0, howto = 0) ["../../../../src/kernel/arch/alpha/machdep.c":1118, 0xfffffc0000374a0

_kernel_memory_fault_data_begin: 10struct {fault_va = 0x0fault_pc = 0x0fault_ra = 0xfffffc000028951cfault_sp = 0xffffffff96199a48access = 0xffffffffff

_kdbx_sum_start:Hostname : system.dec.comcpu: AlphaServer 4100 5/400 avail: 1Boot-time: Tues Jul 7 10:33:25 1998Time: Mon Jul 13 13:58:52 1998Kernel :

trap: invalid memory ifetch access from kernel modeThis message describes the kernel memory fault and indicates that thekernel was unable to fetch a n

11 The _stack_trace_begin line begins a trace of the current threadblock’s stack at the time of the crash. In this case the _XentMM functioncalled the

text file that describes the hardware and software that will be present onthe running system. Using this information, the bootstrap linker links themo

IndexAabscallout kdbx extension, 2–19access variable, A–6addr_to_proc function, 3–3alias command, 2–13Alpha hardware architecturedocumentation, 1–1arp

crash-data.n fileexplanation of contents, A–1crashdc commandexplanation of output from, A–1crashdc utility, 2–44customizing kdbx debuggerenvironment,

using to determine type of kernel,1–2file kdbx extension, 2–21firmware versiondisplaying, 2–10format_addr function, 3–10free_sym function, 3–11Ggatewa

krash function, 3–11Lld commandusing to build a kernel image file,1–2libkdbx.a library, 3–1library functionsfor extensions to kdbx debugger,3–2library

Qquit command, 2–15quit function, 3–17Rread_field_vals function, 3–17read_line function, 3–18read_memory function, 3–18read_response function, 3–19rea

displaying using the inpcb kdbxextension, 2–22test system, 2–38testing kernel programs, 2–37threaddisplaying the process control blockfor, 2–28thread

1.2 Debugging Kernel ProgramsKernel programs can be difficult to debug because you normally cannotcontrol kernel execution. To make debugging kernel p

# dbx -k /vmunix /dev/memThis command invokes dbx with the kernel debugging flag, −k, whichmaps kernel addresses to make kernel debugging easier. The

1.4 Analyzing a Crash Dump FileIf your system crashes, you can often find the cause of the crash by usingdbx or kdbx to debug or analyze a crash dump

At system reboot time, the copy of core memory saved in the swap partitionsis copied into a file, called a crash dump file. You can analyze the crashd

2Kernel Debugging UtilitiesThe Tru64 UNIX system provides several tools you can use to debug thekernel and kernel programs. The Ladebug debugger (avai

______________________ Note _______________________Starting with Tru64 UNIX Version 5.0, all the previouslymentioned tools can be used with compressed

• The system core memory imageThese files may be files from a running system, such as /vmunix and/dev/mem, or dump files, such as vmunix.n and vmzcore

© 1999 Compaq Computer CorporationCOMPAQ and the Compaq logo Registered in U.S. Patent and Trademark Office. Alpha and Tru64 aretrademarks of Compaq I

want to add a symbol table to your current debugging session rather thanend the session and start a new one. To add a symbol table to your currentdebu

• During the dbx session, if you want to load a module dynamically, firstset the $module_path dbx variable and then use the addobj commandto load the

The following examples show how to use dbx to examine kernel images:(dbx) _realstart/Xfffffc00002a4008: c020000243c4153e(dbx) _realstart/i[_realstart:

struct timeval {int tv_sec;int tv_usec;} it_value;};2.1.7 Debugging Multiple ThreadsYou can use the dbx debugger to examine the state of the kernel’s

/usr/include/machine/reg.h header file to determine where registersare stored in the exception frame.The savedefp variable contains the location of th

To specify a full crash dump permanently so that this setting remains ineffect after a reboot, use the patch command in dbx, as shown in thefollowing

2 The up 8 command moves the debugging context 8 activation levels upthe stack to one of the recursive calls within the user program code.3 The print

For debugging purposes, set the lockmode attribute to 4. Follow these stepsto set the lockmode attribute to 4:1. Create a stanza-formatted file named,

[0] 0xfffffc000065c580[1] 0xfffffc000065c780}• Lock statistics are recorded to allow you to determine what kind ofcontention you have on a particular

/dev/mem, respectively. By default, crash dump files are created in the/var/adm/crash directory (see the System Administration manual).Use the followi

ContentsAbout This Manual1 Introduction to Kernel Debugging1.1Linking a Kernel Image for Debugging ... 1–11.2Debugging Kernel

context proc | userSets context to the user’s aliases or the extension’s aliases. Thiscommand is used only by the extensions.coredata start_address en

print stringDisplays string on the terminal. If this command is used by anextension, the terminal receives no output.quitExits the kdbx debugger.sourc

Notation Address Type Replaces Examplekksegfffffc00 k0x00487c48u user space00000000 u0x86406200?Unrecognized orrandom type?0x3782cc33The sections that

to the start_address is usually of the form&arrayname[0].flagsIf the you specify the −head flag, the next argumentappears as the table header.If t

0xfffffc0000473838 = "syscalltrace"0xfffffc0000473848 = "boothowto"0xfffffc0000473858 = "do_virtual_tables"0xfffffc00004

wakeup k0x0187a220 374923thread_timeout k0x010ee950 376286thread_timeout k0x0132f220 40724481realitexpire k0x01069950 80436086thread_timeout k0x01bba9

configFor example:(kdbx) configBus #0 (0xfffffc000048c6a0): Name - "tc" Connected to - "nexus"Config 1 - tcconfl1 Config 2 - tccon

2.2.3.9 Disassembling InstructionsThe dis extension disassembles some number of instructions. Thisextension has the following format:dis start-address

=========== ==== === === ======= =========== =========== ====== =====v0x90406000 file 4 0 vnops v0x90259550 v0x863d5540 68 r wv0x90406058 file 1 0 vno

list_action " type" next-field end-addr start-addr [ flags] commandThe arguments to the list_action extension are as follows:"type &quo

2.2.3.9Disassembling Instructions ...2–212.2.3.10Displaying Remote Exported Entries ... ... 2–212.2.3.11Dis

For example:(kdbx) list_action "struct proc *" p_nxt 0 allproc p \%c.task.u_address.uu_comm %c.p_pid"list_action" 1382"dbx&qu

−class nameDisplays the lockstats structures for the specifiedlock class. (Use the lockinfo command to displayinformation about the names of lock clas

This extension has the following format:lockinfo [ -class name ]The −class flag allows you to display the lockinfo structure for aparticular class of

2.2.3.17 Displaying the Namecache StructuresThe namecache extension displays the namecache structures on the system,and has the following format:namec

2.2.3.19 Converting the Contents of Memory to SymbolsThe paddr extension converts a range of memory to symbolic referencesand has the following format

substitution, which the dbx debugger’s printf command does not. Thisextension has the following format:printf format-string [ args]The arguments to th

v0x81a28210 5301 5276 5301 1138 0 00080002 00000000 NULLin pagv ctty execv0x819aad80 195 1 195 0 0 00080628 00000000 NULL in pagvv0x8197c210 6346 1 63

(kdbx) sumHostname : system.dec.comcpu: DEC3000 - M500 avail: 1Boot-time: Tue Nov 3 15:01:37 1992Time: Fri Nov 6 09:59:00 1998Kernel : OSF1 release 1.

v0x81a0ab70 3 1 v0x819f1ce0 INSWAPPED v0x81a0af20 v0x81a0ad80 5276v0x81a26b70 3 1 v0x819f1760 INSWAPPED v0x81a26f20 v0x81a26d80 5278v0x819f2b70 3 1 v0

−uDisplays the stack trace of all user threadsFor example:(kdbx) trace*** stack trace of thread 0xffffffff819af590 pid=0 ***> 0 thread_run(new_thre

3.2.7Checking Arguments Passed to an Extension ... 3–73.2.8Checking the Fields in a Structure ...3–73.2.9Setting

u [ proc-addr]If you omit arguments, the extension displays the u structure of thecurrently running process.For example:(kdbx) u ffffffff9027ff38procp

-uthreadDisplays all ucreds referenced by the uthreadstructures-fileDisplays all ucreds referenced by the file structures-bufDisplays all ucreds refer

=================== ====== =======0xffffffff863d5a40 4 42.2.3.32 Removing AliasesThe unaliasall extension removes all aliases, including the predefine

For example:(kdbx) vnodeADDR_VNODE V_TYPE V_TAG USECNT V_MOUNT=========== ====== ====== ====== ===========v0x9021e000 VBLK VT_NON 1 k0x00467ee8v0x9021

and perform other debugging tasks, just as you would when debugging userspace programs.The ability to debug a running kernel is provided through remot

To use the kdebug debugger, you must set up your build, gateway, andtest systems as described in Section 2.3.1. Once you complete the setup,you invoke

The $kdebug_host variable is the name of the gateway system. Bydefault, $kdebug_host is set to localhost, assuming no gatewaysystem is being used.The

Setting this system attribute makes debugging on an SMP system easier.For information about the advantages provided see Section 2.1.11.8. Set the OPTI

dbx debugging commands. See Section 2.1, the dbx(1)reference page, orthe Programmer’s Guide for information on dbx debugging commands.If you are unabl

– Check the /etc/inittab file to see if any processes are using thatline. If so, disable these lines until you finish with the kdebugsession. See the

A Output from the crashdc CommandIndexExamples3–1Template Extension Using Lists ... ... 3–233–2Extension That Uses Linke

– Remove any settings of the $kdebug_line variable as follows:set $kdebug_line=• Start dbx on the build system. You should see informational messageso

uses existing system tools and utilities to extract information from crashdumps. The information garnered from crash dump files or from the runningker

3Writing Extensions to the kdbx DebuggerTo assist in debugging kernel code, you can write an extension to the kdbxdebugger. Extensions interact with k

discussed in Section 3.3 can help you understand what is involved in writingan extension and provide good examples of using the kdbx library functions

The values in comm and local provide the error code interpreted byprint_status.• The FieldRec data type, which is used to declare a field of interesti

3.2.3 Getting a Representation of an Array ElementThe array_element function returns a representation of one element of anarray. The function returns

data type. It returns the pointer value if the data type of the array elementis a pointer data type.This function returns TRUE if it is successful, FA

3.2.5 Returning the Size of an ArrayThe array_size function returns the size of the specified array. Thisfunction has the following syntax:unsigned in

For example:if(!cast(addr, "struct file", &fil, &error)){fprintf(stderr, "Couldn’t cast address to a file:\n");fprintf(std

About This ManualThis manual provides information on the tools used to debug a kernel andanalyze a crash dump file of the Tru64™ UNIX (formerly DIGITA

char** hints);ArgumentInput/OutputDescriptionsymbolInput Names the structure to be checkedfieldsInput Describes the fields to be checkednfieldsInput S

3.2.10 Passing Commands to the dbx DebuggerThe dbx function passes a command to the dbx debugger. The function hasan argument, expect_output, that con

structure = deref_pointer(struct_pointer);3.2.12 Displaying the Error Messages Stored in FieldsThe field_errors function displays the error messages s

format_addr((long) struct_addr(ele), address);format_addr((long) fields[2].data, cred);format_addr((long) fields[3].data, data);sprintf(buf, "%s

ArgumentInput/OutputDescriptioncommandInput Names the command to be executedquoteInput If set to TRUE causes the quotecharacter, apostrophe, and backs

• The next argument is the address of the next node in the list; forexample, the next node might be at address 0xffffffff8196d050.3.2.16 Getting the A

3.2.17 Passing an Extension to kdbxThe new_proc function directs kdbx to execute a proc command witharguments specified in args. The args arguments ca

For example:resp = read_response_status();next_number(resp, NULL, &size);ret->size = size;3.2.19 Getting the Next Token as a StringThe next_tok

}3.2.20 Displaying a MessageThe print function displays a message on the terminal screen. Because ofthe input and output redirection done by kdbx, all

if(status.type != OK){print_status("read_line failed", &status);quit(1);}3.2.22 Exiting from an ExtensionThe quit function sends a quit

OrganizationThis manual consists of four chapters and one appendix:Chapter 1 Introduces the concepts of kernel debugging andcrash dump analysis.Chapte

return(False);}3.2.24 Returning a Line of kdbx OutputThe read_line function returns the next line of the output from the lastkdbx command executed. If

You can use this function to look up any type of value; however it is mostuseful for retrieving the value of pointers that point to other pointers.For

3.2.27 Reading Symbol RepresentationsThe read_sym function returns a representation of the named symbol. Thisfunction has the following format:DataStr

3.2.29 Reading the Value of a SymbolThe read_sym_val function returns the value of the specified symbol. Thisfunction has the following format:Boolean

print(buf);sprintf(buf, "\tConfig 1 - %s\tConfig 2 - %s",addr_to_proc((long) bus_fields[3].data),addr_to_proc((long) bus_fields[4].data));pr

• Extensions that use arrays. Example 3–3 provides a C languagetemplate and Example 3–4 is the source code for the /var/kdbx/fileextension, which show

Example 3–1: Template Extension Using Lists (cont.)quit(0);}1 The help string is output by the check_args function if the user entersthe help extensio

Example 3–2: Extension That Uses Linked Lists: callout.c (cont.)#define NUM_PROCESSOR_FIELDS(sizeof(processor_fields)/sizeof(processor_fields[0]))#def

Example 3–2: Extension That Uses Linked Lists: callout.c (cont.)sprintf(buf, "set $hexints=0");dbx(buf, False);sprintf(buf, "p \*(long

Example 3–2: Extension That Uses Linked Lists: callout.c (cont.)}} /* end of for */quit(0);} /* end of main() */Example 3–3: Template Extensions Using

G Manuals for general usersS Manuals for system and network administratorsP Manuals for programmersR Manuals for reference page usersSome manuals in t

Example 3–3: Template Extensions Using Arrays (cont.)}}1 The help string is output by the check_args function if the user entersthe help extension_nam

Example 3–4: Extension That Uses Arrays: file.c (cont.)char buffer[256];/* *** Implement addresses *** */FieldRec fields[] = {{ ".f_type", N

Example 3–4: Extension That Uses Arrays: file.c (cont.)else if((long) (fields[5].data) == vn_addr) ops = " vnops";else if((long) (fields[5].

Example 3–4: Extension That Uses Arrays: file.c (cont.)field_errors(fields, NUM_UTASK_FIELDS);quit(1);}if(!read_sym_addr("vnops", &vn_ad

Example 3–4: Extension That Uses Arrays: file.c (cont.)* Figure out the location of the utask structure in the supertask* #define proc_to_utask(p) (lo

Example 3–4: Extension That Uses Arrays: file.c (cont.){if((pid_entry_ele = array_element(pid_entry_struct, index, &error))==NULL){fprintf(stderr,

Example 3–4: Extension That Uses Arrays: file.c (cont.)quit(1);}if (first_file) {sprintf(buf, "[Process ID: %d]", fields_pid[0].data);print(

Example 3–5: Extension That Uses Global Symbols: sum.c (cont.)* cpup no longer exists, emmulate platform_string(),* a.k.a. get_system_type_string().re

The following example shows how to invoke the test extension from withinthe kdbx debugger:# kdbx -k /vmunixdbx version 5.0Type ’help’ for help.(kdbx)

2. Set up kdbx and dbx to communicate with each other. In the kdbxsession, enter the procpd alias to create the files /tmp/pipein and/tmp/pipeout as f