Annotation of www/ddb.html, Revision 1.14
1.12 tb 1: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
1.1 beck 2: <html>
3: <head>
1.10 tj 4: <title>OpenBSD: Crash Reports</title>
1.1 beck 5: <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
6: <meta name="description" content="How to report an OpenBSD kernel crash">
1.9 tb 7: <meta name="copyright" content="This document copyright 1998-2016 by OpenBSD.">
8: <meta name="viewport" content="width=device-width, initial-scale=1">
9: <link rel="stylesheet" type="text/css" href="openbsd.css">
1.13 tb 10: <link rel="canonical" href="https://www.openbsd.org/report.html">
1.14 ! tb 11: <style type="text/css">
! 12: h3, h4 { color: #0000e0; }
! 13: </style>
1.1 beck 14: </head>
15:
16: <body bgcolor="#ffffff" text="#000000" link="#23238e">
1.9 tb 17:
18: <h2>
19: <a href="index.html">
20: <font color="#0000ff"><i>Open</i></font><font color="#000084">BSD</font></a>
1.10 tj 21: <font color="#e00000">Crash Reports</font>
1.9 tb 22: </h2>
23: <hr>
1.1 beck 24: <p>
25:
1.14 ! tb 26: <h3>Minimum information for kernel problems</h3>
1.1 beck 27:
1.6 tb 28: Familiarize yourself with
29: <a href="report.html">the general bug reporting procedures</a>
30: first.
31: All of that will apply.
1.1 beck 32: When reporting a kernel panic or crash, please remember:
33:
34: <ul>
1.14 ! tb 35: <li><i>We need the console output on the screen</i>.
! 36: Capture it and save it.
! 37: Serial consoles are best, but if you are on a VGA console you can
! 38: <a href="faq/faq7.html">scroll the console back</a>
! 39: and take readable pictures with a phone or camera.<br>
! 40:
! 41: <li><i>If the kernel panicked we need the traceback.</i>
! 42: It may be displayed on the screen.
! 43: If you are at a
! 44: <tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt>
! 45: prompt, type <tt>trace</tt>.
! 46: If you are running SMP, use the <tt>mach ddbcpu N</tt> command for each
! 47: of the <tt>N</tt> processors you have and repeat the <tt>trace</tt>
! 48: command for each processor.<br>
1.1 beck 49:
1.14 ! tb 50: <li><i>We need the process list.</i>
! 51: Use the command <tt>ps</tt> to get that.
1.1 beck 52: </ul>
53:
1.14 ! tb 54: <i>
! 55: Reports without the above information are useless.
! 56: This is the minimum we need to be able to track down the issue.
! 57: </i>
! 58:
! 59: <h3>Additional information you can send</h3>
1.1 beck 60:
1.6 tb 61: In some situations more information is desirable.
62: Below are outlined some additional steps you can take in certain situations:
1.14 ! tb 63:
1.1 beck 64: <ul>
1.14 ! tb 65: <li><i>If your crash appears to involve filesystems.</i>
! 66: The following additional things would be helpful
! 67: <ul>
! 68: <li>The output of the
! 69: <tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> command
! 70: <tt>show uvm</tt>
! 71: <li>The output of the
! 72: <tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt>
! 73: command <tt>show bcstats</tt>
! 74: <li>The output of the <tt>mount</tt> command from your running machine, so
! 75: we know what filesystems are mounted and how.
! 76: </ul>
! 77: <li> ... XXX boot crash? XXX
! 78: <li> ... XXX show regs? XXX
1.1 beck 79: </ul>
1.14 ! tb 80:
! 81: <h3>Lost the panic message?</h3>
! 82:
! 83: Under some circumstances, you may lose the very first message of a panic,
! 84: stating the reason for the panic.
! 85:
! 86: <blockquote><pre>
! 87: ddb> <b>show panic</b>
! 88: 0: kernel: page fault trap, code=0
! 89: ddb>
! 90: </pre></blockquote>
! 91:
! 92: <h3>Note for SMP systems</h3>
! 93:
! 94: You should get a trace from each processor as part of your report:
! 95:
! 96: <blockquote><pre>
! 97: ddb{0}> <b>trace</b>
! 98: pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226
! 99: fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5
! 100: fxp_intr(d0a62000) at fxp_intr+0x1e7
! 101: Xintr_ioapic0() at Xintr_ioapic0+0x6d
! 102: --- interrupt ---
! 103: idle_loop+0x21:
! 104: ddb{0}> <b>machine ddbcpu 1</b>
! 105: Stopped at Debugger+0x4: leave
! 106: ddb{1}> <b>trace</b>
! 107: Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4
! 108: i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb
! 109: i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x
! 110: 4a
! 111: Xintripi() at Xintripi+0x47
! 112: --- interrupt ---
! 113: i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37
! 114: Xintrltimer() at Xintrltimer+0x47
! 115: --- interrupt ---
! 116: idle_loop+0x21:
! 117: ddb{1}>
! 118: </pre></blockquote>
! 119:
! 120: Repeat the <tt>machine ddbcpu x</tt> followed by <tt>trace</tt> for each
! 121: processor in your machine.
! 122:
! 123: <h3>How do I gather further information from a kernel crash?</h3><p>
! 124:
! 125: A typical kernel crash on OpenBSD might look like this:
! 126: (things to watch for are marked with bold font)
! 127:
! 128: <blockquote><pre>
! 129: kernel: page fault trap, code=0
! 130: Stopped at <b>_pf_route+0x263</b>: mov 0x40(%edi),%edx
! 131: ddb>
! 132: </pre></blockquote>
! 133:
! 134: The first command to run from the <tt>ddb></tt> prompt is <tt>trace</tt>
! 135: (see <a href="http://man.openbsd.org/ddb">ddb(4)</a> for details):
! 136:
! 137: <blockquote><pre>
! 138: ddb> <b>trace</b>
! 139: <b>_pf_route</b>(e28cb7e4,e28bc978,2,1fad,d0b8b120) at <b>_pf_route+0x263</b>
! 140: _pf_test(2,1f4ad,e28cb7e4,b4c1) at _pf_test+0x706
! 141: _pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at _pf_route+0x207
! 142: _pf_test(2,d0a65440,e28cbb00,d023c282) at _pf_test+0x706
! 143: _ip_output(d0b6a200,0,0,0,0) at _ip_output+0xb67
! 144: _icmp_send(d0b6a200,0,1,a012) at _icmp_send+0x57
! 145: _icmp_reflect(d0b6a200,0,1,0,3) at _icmp_reflect+0x26b
! 146: _icmp_input(d0b6a200,14,0,0,d0b6a200) at _icmp_input+0x42c
! 147: _ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at _ipv4_input+0x6eb
! 148: _ipintr(10,10,e289f140,e289f140,e28cbd38) at _ipintr+0x8d
! 149: Bad frame pointer: 0xe28cbcac
! 150: ddb>
! 151: </pre></blockquote>
! 152:
! 153: This tells us what function calls lead to the crash.
! 154:
! 155: <p>
! 156: To find out the particular line of C code that caused the crash, you can
! 157: do the following:
! 158:
! 159: <p>
! 160: Find the source file where the crashing function is defined in.
! 161: In this example, that would be <tt>pf_route()</tt> in <tt>sys/net/pf.c</tt>.
! 162: Recompile that source file with debug information:
! 163:
! 164: <blockquote><pre>
! 165: # <b>cd /usr/src/sys/arch/$(uname -m)/compile/GENERIC</b>
! 166: # <b>rm pf.o</b>
! 167: # <b>DEBUG=-g make pf.o</b>
! 168: </pre></blockquote>
! 169:
! 170: Then use <a href="http://man.openbsd.org/objdump">objdump(1)</a> to get the
! 171: disassembly:
! 172:
! 173: <blockquote><pre>
! 174: # <b>objdump --line --disassemble --reloc pf.o >pf.dis</b>
! 175: </pre></blockquote>
! 176:
! 177: In the output, grep for the function name:
! 178:
! 179: <blockquote><pre>
! 180: # <b>grep "<_pf_route>:" pf.dis</b>
! 181: 0000<b>7d88</b> <_pf_route>:
! 182: </pre></blockquote>
! 183:
! 184: Take this first hex number and add the offset from the <tt>Stopped at</tt> line:
! 185: <tt>0x7d88 + 0x263 == 0x7feb</tt>.
! 186:
! 187: <p>
! 188: Scroll down to that line (the assembler instruction should match the one
! 189: quoted in the <tt>Stopped at</tt> line), then up to the nearest C line number:
! 190:
! 191: <blockquote><pre>
! 192: # <b>more pf.dis</b>
! 193: /usr/src/sys/arch/i386/compile/GENERIC/../../../../net/pf.c:<b>3872</b>
! 194: 7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax
! 195: <b>7feb</b>: 8b 57 40 mov 0x40(%edi),%edx
! 196: 7fee: 39 d0 cmp %edx,%eax
! 197: 7ff0: 0f 87 92 00 00 00 ja 8088 <_pf_route+0x300>
! 198: </pre></blockquote>
! 199:
! 200: So, it's precisely line <tt>3872</tt> of <tt>pf.c</tt> that crashes:
! 201:
! 202: <blockquote><pre>
! 203: # <b>cat -n pf.c | head -n 3872 | tail -n 1</b>
! 204: 3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {
! 205: </pre></blockquote>
! 206:
! 207: Note that the kernel that produced the crash output and the object file
! 208: for objdump must be compiled from the exact same source file, otherwise
! 209: the offsets won't match.
! 210:
! 211: <p>
! 212: If you provide both the ddb trace output and the relevant objdump section,
! 213: that's very helpful.
1.1 beck 214:
215: <p>
216: </body>
217: </html>
![[BACK]](/icons/back.gif){kind=icon}
Return to ddb.html CVS log ![[TXT]](/icons/text.gif){kind=icon}
![[DIR]](/icons/dir.gif){kind=icon}
Up to [local] / www