=================================================================== RCS file: /cvsrepo/anoncvs/cvs/www/ddb.html,v retrieving revision 1.13 retrieving revision 1.14 diff -u -r1.13 -r1.14 --- www/ddb.html 2016/08/15 02:22:06 1.13 +++ www/ddb.html 2016/09/24 03:22:12 1.14 @@ -8,6 +8,9 @@ + @@ -20,7 +23,7 @@

Minimum information for kernel problems

Familiarize yourself with the general bug reporting procedures @@ -29,50 +32,185 @@ When reporting a kernel panic or crash, please remember:

We need the console output on the screen. - Capture it and save it. - Serial consoles are best, but if you are on a VGA console you can - scroll the console back - and take readable pictures with a phone or camera.
+
We need the console output on the screen. + Capture it and save it. + Serial consoles are best, but if you are on a VGA console you can + scroll the console back + and take readable pictures with a phone or camera.
-
If the kernel panicked we need the traceback. - It may be displayed on the screen. - If you are at a - ddb> - prompt, type trace. - If you are running SMP, use the mach ddbcpu N command for each - of the N processors you have and repeat the trace - command for each processor.
+
If the kernel panicked we need the traceback. + It may be displayed on the screen. + If you are at a + ddb> + prompt, type trace. + If you are running SMP, use the mach ddbcpu N command for each + of the N processors you have and repeat the trace + command for each processor.
-
We need the process list. - Use the command ps to get us that. +
We need the process list. + Use the command ps to get that.

- - Reports without the above information are useless. - This is the minimum we need to be able to track down the issue. - + +Reports without the above information are useless. +This is the minimum we need to be able to track down the issue. + -

Additional information you can send

+ In some situations more information is desirable. Below are outlined some additional steps you can take in certain situations: +

... XXX boot crash? XXX -
... XXX show regs? XXX -
If your crash appears to involve filesystems. - The following additional things would be helpful -
- The output of the - ddb> - command show uvm -
- The output of the - ddb> - command show bcstats -
- The output of the mount command from your running machine, - so we know what filesystems are mounted and how. +
- If your crash appears to involve filesystems. + The following additional things would be helpful +
  - The output of the + ddb> command + show uvm +
  - The output of the + ddb> + command show bcstats +
  - The output of the mount command from your running machine, so + we know what filesystems are mounted and how. +
  +
- ... XXX boot crash? XXX +
- ... XXX show regs? XXX
-

+ +

Lost the panic message?

+ +Under some circumstances, you may lose the very first message of a panic, +stating the reason for the panic. + +

+ddb> show panic
+0:      kernel: page fault trap, code=0
+ddb>
+

+ +

Note for SMP systems

+ +You should get a trace from each processor as part of your report: + +

+ddb{0}> trace
+pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226
+fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5
+fxp_intr(d0a62000) at fxp_intr+0x1e7
+Xintr_ioapic0() at Xintr_ioapic0+0x6d
+--- interrupt ---
+idle_loop+0x21:
+ddb{0}> machine ddbcpu 1
+Stopped at      Debugger+0x4:   leave
+ddb{1}> trace
+Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4
+i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb
+i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x
+4a
+Xintripi() at Xintripi+0x47
+--- interrupt ---
+i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37
+Xintrltimer() at Xintrltimer+0x47
+--- interrupt ---
+idle_loop+0x21:
+ddb{1}>
+

+ +Repeat the machine ddbcpu x followed by trace for each +processor in your machine. + +

How do I gather further information from a kernel crash?

+ +A typical kernel crash on OpenBSD might look like this: +(things to watch for are marked with bold font) + +

+kernel: page fault trap, code=0
+Stopped at    _pf_route+0x263:        mov     0x40(%edi),%edx
+ddb>
+

+ +The first command to run from the ddb> prompt is trace +(see ddb(4) for details): + +

+ddb> trace
+_pf_route(e28cb7e4,e28bc978,2,1fad,d0b8b120) at _pf_route+0x263
+_pf_test(2,1f4ad,e28cb7e4,b4c1) at _pf_test+0x706
+_pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at _pf_route+0x207
+_pf_test(2,d0a65440,e28cbb00,d023c282) at _pf_test+0x706
+_ip_output(d0b6a200,0,0,0,0) at _ip_output+0xb67
+_icmp_send(d0b6a200,0,1,a012) at _icmp_send+0x57
+_icmp_reflect(d0b6a200,0,1,0,3) at _icmp_reflect+0x26b
+_icmp_input(d0b6a200,14,0,0,d0b6a200) at _icmp_input+0x42c
+_ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at _ipv4_input+0x6eb
+_ipintr(10,10,e289f140,e289f140,e28cbd38) at _ipintr+0x8d
+Bad frame pointer: 0xe28cbcac
+ddb>
+

+ +This tells us what function calls lead to the crash. + +

+To find out the particular line of C code that caused the crash, you can +do the following: + +

+Find the source file where the crashing function is defined in. +In this example, that would be pf_route() in sys/net/pf.c. +Recompile that source file with debug information: + +

+# cd /usr/src/sys/arch/$(uname -m)/compile/GENERIC
+# rm pf.o
+# DEBUG=-g make pf.o
+

+ +Then use objdump(1) to get the +disassembly: + +

+# objdump --line --disassemble --reloc pf.o >pf.dis
+

+ +In the output, grep for the function name: + +

+# grep "<_pf_route>:" pf.dis
+00007d88 <_pf_route>:
+

+ +Take this first hex number and add the offset from the Stopped at line: +0x7d88 + 0x263 == 0x7feb. + +

+Scroll down to that line (the assembler instruction should match the one +quoted in the Stopped at line), then up to the nearest C line number: + +

+# more pf.dis
+/usr/src/sys/arch/i386/compile/GENERIC/../../../../net/pf.c:3872
+    7fe7:       0f b7 43 02             movzwl 0x2(%ebx),%eax
+    7feb:       8b 57 40                mov    0x40(%edi),%edx
+    7fee:       39 d0                   cmp    %edx,%eax
+    7ff0:       0f 87 92 00 00 00       ja     8088 <_pf_route+0x300>
+

+ +So, it's precisely line 3872 of pf.c that crashes: + +

+# cat -n pf.c | head -n 3872 | tail -n 1
+3872          if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {
+

+ +Note that the kernel that produced the crash output and the object file +for objdump must be compiled from the exact same source file, otherwise +the offsets won't match. + +

+If you provide both the ddb trace output and the relevant objdump section, +that's very helpful.