Annotation of www/ddb.html, Revision 1.21
1.21 ! bentley 1: <!doctype html>
! 2: <html lang=en>
! 3: <meta charset=utf-8>
! 4:
1.10 tj 5: <title>OpenBSD: Crash Reports</title>
1.1 beck 6: <meta name="description" content="How to report an OpenBSD kernel crash">
1.9 tb 7: <meta name="viewport" content="width=device-width, initial-scale=1">
8: <link rel="stylesheet" type="text/css" href="openbsd.css">
1.13 tb 9: <link rel="canonical" href="https://www.openbsd.org/report.html">
1.21 ! bentley 10: <style>
! 11: h3, h4 {
! 12: color: var(--blue);
! 13: }
1.14 tb 14: </style>
1.9 tb 15:
1.21 ! bentley 16: <h2 id=OpenBSD>
1.9 tb 17: <a href="index.html">
1.21 ! bentley 18: <i>Open</i><b>BSD</b></a>
! 19: Crash Reports
1.9 tb 20: </h2>
1.21 ! bentley 21:
1.9 tb 22: <hr>
1.1 beck 23:
1.14 tb 24: <h3>Minimum information for kernel problems</h3>
1.1 beck 25:
1.21 ! bentley 26: <p>
1.6 tb 27: Familiarize yourself with
28: <a href="report.html">the general bug reporting procedures</a>
29: first.
30: All of that will apply.
1.1 beck 31: When reporting a kernel panic or crash, please remember:
32:
33: <ul>
1.21 ! bentley 34: <li><em>We need the console output on the screen</em>.
1.14 tb 35: Capture it and save it.
36: Serial consoles are best, but if you are on a VGA console you can
37: <a href="faq/faq7.html">scroll the console back</a>
1.21 ! bentley 38: and take readable pictures with a phone or camera.
1.14 tb 39:
1.21 ! bentley 40: <li><em>If the kernel panicked we need the traceback.</em>
1.14 tb 41: It may be displayed on the screen.
42: If you are at a
1.21 ! bentley 43: <samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp>
! 44: prompt, type <kbd>trace</kbd>.
! 45: If you are running SMP, use the <kbd>mach ddbcpu N</kbd> command for each
! 46: of the <var>N</var> processors you have and repeat the <kbd>trace</kbd>
! 47: command for each processor.
1.1 beck 48:
1.21 ! bentley 49: <li><em>We need the process list.</em>
! 50: Use the command <kbd>ps</kbd> to get that.
1.1 beck 51: </ul>
52:
1.21 ! bentley 53: <p>
! 54: <em>
1.14 tb 55: Reports without the above information are useless.
56: This is the minimum we need to be able to track down the issue.
1.21 ! bentley 57: </em>
1.14 tb 58:
59: <h3>Additional information you can send</h3>
1.1 beck 60:
1.21 ! bentley 61: <p>
1.6 tb 62: In some situations more information is desirable.
63: Below are outlined some additional steps you can take in certain situations:
1.14 tb 64:
1.1 beck 65: <ul>
1.14 tb 66: <li><i>If your crash appears to involve filesystems.</i>
67: The following additional things would be helpful
68: <ul>
69: <li>The output of the
1.21 ! bentley 70: <samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp> command
! 71: <kbd>show uvm</kbd>
1.14 tb 72: <li>The output of the
1.21 ! bentley 73: <samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp>
! 74: command <kbd>show bcstats</kbd>
! 75: <li>The output of the <kbd>mount</kbd> command from your running machine, so
1.14 tb 76: we know what filesystems are mounted and how.
77: </ul>
78: <li> ... XXX boot crash? XXX
79: <li> ... XXX show regs? XXX
1.1 beck 80: </ul>
1.14 tb 81:
82: <h3>Lost the panic message?</h3>
83:
1.21 ! bentley 84: <p>
1.14 tb 85: Under some circumstances, you may lose the very first message of a panic,
86: stating the reason for the panic.
87:
1.20 tj 88: <pre class="cmdbox">
1.14 tb 89: ddb> <b>show panic</b>
90: 0: kernel: page fault trap, code=0
91: ddb>
1.20 tj 92: </pre>
1.14 tb 93:
94: <h3>Note for SMP systems</h3>
95:
1.21 ! bentley 96: <p>
1.14 tb 97: You should get a trace from each processor as part of your report:
98:
1.20 tj 99: <pre class="cmdbox">
1.14 tb 100: ddb{0}> <b>trace</b>
101: pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226
102: fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5
103: fxp_intr(d0a62000) at fxp_intr+0x1e7
104: Xintr_ioapic0() at Xintr_ioapic0+0x6d
105: --- interrupt ---
106: idle_loop+0x21:
107: ddb{0}> <b>machine ddbcpu 1</b>
108: Stopped at Debugger+0x4: leave
109: ddb{1}> <b>trace</b>
110: Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4
111: i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb
112: i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x
113: 4a
114: Xintripi() at Xintripi+0x47
115: --- interrupt ---
116: i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37
117: Xintrltimer() at Xintrltimer+0x47
118: --- interrupt ---
119: idle_loop+0x21:
120: ddb{1}>
1.20 tj 121: </pre>
1.14 tb 122:
1.21 ! bentley 123: <p>
! 124: Repeat the <code>machine ddbcpu x</code> followed by <code>trace</code> for each
1.14 tb 125: processor in your machine.
126:
127: <h3>How do I gather further information from a kernel crash?</h3><p>
128:
1.21 ! bentley 129: <p>
1.14 tb 130: A typical kernel crash on OpenBSD might look like this:
131:
1.20 tj 132: <pre class="cmdbox">
1.14 tb 133: kernel: page fault trap, code=0
1.18 tb 134: Stopped at <b>pf_route+0x263</b>: mov 0x40(%edi),%edx
1.14 tb 135: ddb>
1.20 tj 136: </pre>
1.14 tb 137:
1.21 ! bentley 138: <p>
! 139: This crash happened at offset <code>0x263</code> in the function <code>pf_route</code>.
1.17 tb 140:
141: <p>
142: The first command to run from the
1.21 ! bentley 143: <a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <code>trace</code>:
1.14 tb 144:
1.20 tj 145: <pre class="cmdbox">
1.14 tb 146: ddb> <b>trace</b>
1.18 tb 147: <b>pf_route</b>(e28cb7e4,e28bc978,2,1fad,d0b8b120) at <b>pf_route+0x263</b>
148: pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706
149: pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at pf_route+0x207
150: pf_test(2,d0a65440,e28cbb00,d023c282) at pf_test+0x706
151: ip_output(d0b6a200,0,0,0,0) at ip_output+0xb67
152: icmp_send(d0b6a200,0,1,a012) at icmp_send+0x57
153: icmp_reflect(d0b6a200,0,1,0,3) at icmp_reflect+0x26b
154: icmp_input(d0b6a200,14,0,0,d0b6a200) at icmp_input+0x42c
155: ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at ipv4_input+0x6eb
156: ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d
1.14 tb 157: Bad frame pointer: 0xe28cbcac
158: ddb>
1.20 tj 159: </pre>
1.14 tb 160:
1.21 ! bentley 161: <p>
1.14 tb 162: This tells us what function calls lead to the crash.
163:
164: <p>
165: To find out the particular line of C code that caused the crash, you can
166: do the following:
167:
168: <p>
1.17 tb 169: Find the source file where the crashing function is defined.
1.21 ! bentley 170: In this example, that would be <code>pf_route()</code> in <code>/sys/net/pf.c</code>.
1.19 tb 171: Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the
1.17 tb 172: disassembly:
1.14 tb 173:
1.20 tj 174: <pre class="cmdbox">
1.17 tb 175: $ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b>
1.21 ! bentley 176: $ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b>
1.20 tj 177: </pre>
1.14 tb 178:
1.21 ! bentley 179: <p>
1.17 tb 180: In the output, grep for the function name:
1.14 tb 181:
1.20 tj 182: <pre class="cmdbox">
1.21 ! bentley 183: $ <b>grep "<pf_route>:" /tmp/pf.dis</b>
! 184: 0000<b>7d88</b> <pf_route>:
1.20 tj 185: </pre>
1.14 tb 186:
1.21 ! bentley 187: <p>
! 188: Take this first hex number <code>7d88</code> and add the offset <code>0x263</code> from
! 189: the <code>Stopped at</code> line:
1.14 tb 190:
1.20 tj 191: <pre class="cmdbox">
1.17 tb 192: $ <b>printf '%x\n' $((0x7d88 + 0x263))</b>
193: 7feb
1.20 tj 194: </pre>
1.14 tb 195:
1.21 ! bentley 196: <p>
! 197: Scroll down to the line <code>7feb</code>.
! 198: The assembler instruction should match the one quoted in the <code>Stopped at</code>
1.18 tb 199: line.
200: Then scroll up to the nearest C line number:
1.14 tb 201:
1.20 tj 202: <pre class="cmdbox">
1.17 tb 203: $ <b>more /tmp/pf.dis</b>
204: /sys/net/pf.c:<b>3872</b>
1.14 tb 205: 7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax
1.18 tb 206: <b>7feb</b>: 8b 57 40 <b>mov 0x40(%edi),%edx</b>
1.14 tb 207: 7fee: 39 d0 cmp %edx,%eax
1.21 ! bentley 208: 7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300>
1.20 tj 209: </pre>
1.14 tb 210:
1.21 ! bentley 211: <p>
! 212: So, it's precisely line <code>3872</code> of <code>pf.c</code> that crashes:
1.14 tb 213:
1.20 tj 214: <pre class="cmdbox">
1.17 tb 215: $ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b>
1.21 ! bentley 216: 3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {
1.20 tj 217: </pre>
1.14 tb 218:
1.21 ! bentley 219: <p>
1.17 tb 220: The kernel that produced the crash output and the object file for objdump must
221: be compiled from the exact same source file, otherwise the offsets won't match.
1.14 tb 222:
223: <p>
224: If you provide both the ddb trace output and the relevant objdump section,
225: that's very helpful.
![[BACK]](/icons/back.gif){kind=icon}
Return to ddb.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / www