version 1.13, 2016/08/15 02:22:06 |
version 1.14, 2016/09/24 03:22:12 |
|
|
<meta name="viewport" content="width=device-width, initial-scale=1"> |
<meta name="viewport" content="width=device-width, initial-scale=1"> |
<link rel="stylesheet" type="text/css" href="openbsd.css"> |
<link rel="stylesheet" type="text/css" href="openbsd.css"> |
<link rel="canonical" href="https://www.openbsd.org/report.html"> |
<link rel="canonical" href="https://www.openbsd.org/report.html"> |
|
<style type="text/css"> |
|
h3, h4 { color: #0000e0; } |
|
</style> |
</head> |
</head> |
|
|
<body bgcolor="#ffffff" text="#000000" link="#23238e"> |
<body bgcolor="#ffffff" text="#000000" link="#23238e"> |
|
|
<hr> |
<hr> |
<p> |
<p> |
|
|
<h3><font color="#0000e0">Minimum information for kernel problems</font></h3> |
<h3>Minimum information for kernel problems</h3> |
|
|
Familiarize yourself with |
Familiarize yourself with |
<a href="report.html">the general bug reporting procedures</a> |
<a href="report.html">the general bug reporting procedures</a> |
|
|
When reporting a kernel panic or crash, please remember: |
When reporting a kernel panic or crash, please remember: |
|
|
<ul> |
<ul> |
<li><em>We need the console output on the screen</em>. |
<li><i>We need the console output on the screen</i>. |
Capture it and save it. |
Capture it and save it. |
Serial consoles are best, but if you are on a VGA console you can |
Serial consoles are best, but if you are on a VGA console you can |
<a HREF="faq/faq7.html">scroll the console back</a> |
<a href="faq/faq7.html">scroll the console back</a> |
and take readable pictures with a phone or camera.<BR> |
and take readable pictures with a phone or camera.<br> |
|
|
<li><em>If the kernel panicked we need the traceback.</em> |
<li><i>If the kernel panicked we need the traceback.</i> |
It may be displayed on the screen. |
It may be displayed on the screen. |
If you are at a |
If you are at a |
<tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> |
<tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> |
prompt, type <tt>trace</tt>. |
prompt, type <tt>trace</tt>. |
If you are running SMP, use the <tt>mach ddbcpu N</tt> command for each |
If you are running SMP, use the <tt>mach ddbcpu N</tt> command for each |
of the <tt>N</tt> processors you have and repeat the <tt>trace</tt> |
of the <tt>N</tt> processors you have and repeat the <tt>trace</tt> |
command for each processor.<BR> |
command for each processor.<br> |
|
|
<li><em>We need the process list.</em> |
<li><i>We need the process list.</i> |
Use the command <tt>ps</tt> to get us that. |
Use the command <tt>ps</tt> to get that. |
</ul> |
</ul> |
|
|
<em> |
<i> |
Reports without the above information are useless. |
Reports without the above information are useless. |
This is the minimum we need to be able to track down the issue. |
This is the minimum we need to be able to track down the issue. |
</em> |
</i> |
|
|
<h3><font color="#0000e0">Additional information you can send</font></h3> |
<h3>Additional information you can send</h3> |
<P> |
|
In some situations more information is desirable. |
In some situations more information is desirable. |
Below are outlined some additional steps you can take in certain situations: |
Below are outlined some additional steps you can take in certain situations: |
|
|
<ul> |
<ul> |
<li> ... XXX boot crash? XXX |
<li><i>If your crash appears to involve filesystems.</i> |
<li> ... XXX show regs? XXX |
The following additional things would be helpful |
<li> <em>If your crash appears to involve filesystems.</em> |
<ul> |
The following additional things would be helpful |
<li>The output of the |
<ul> |
<tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> command |
<li> The output of the |
<tt>show uvm</tt> |
<tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> |
<li>The output of the |
command <tt>show uvm</tt> |
<tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> |
<li> The output of the |
command <tt>show bcstats</tt> |
<tt><a href="http://man.openbsd.org/ddb.4">ddb</a>></tt> |
<li>The output of the <tt>mount</tt> command from your running machine, so |
command <tt>show bcstats</tt> |
we know what filesystems are mounted and how. |
<li> The output of the <tt>mount</tt> command from your running machine, |
</ul> |
so we know what filesystems are mounted and how. |
<li> ... XXX boot crash? XXX |
|
<li> ... XXX show regs? XXX |
</ul> |
</ul> |
</ul> |
|
|
<h3>Lost the panic message?</h3> |
|
|
|
Under some circumstances, you may lose the very first message of a panic, |
|
stating the reason for the panic. |
|
|
|
<blockquote><pre> |
|
ddb> <b>show panic</b> |
|
0: kernel: page fault trap, code=0 |
|
ddb> |
|
</pre></blockquote> |
|
|
|
<h3>Note for SMP systems</h3> |
|
|
|
You should get a trace from each processor as part of your report: |
|
|
|
<blockquote><pre> |
|
ddb{0}> <b>trace</b> |
|
pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226 |
|
fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5 |
|
fxp_intr(d0a62000) at fxp_intr+0x1e7 |
|
Xintr_ioapic0() at Xintr_ioapic0+0x6d |
|
--- interrupt --- |
|
idle_loop+0x21: |
|
ddb{0}> <b>machine ddbcpu 1</b> |
|
Stopped at Debugger+0x4: leave |
|
ddb{1}> <b>trace</b> |
|
Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4 |
|
i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb |
|
i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x |
|
4a |
|
Xintripi() at Xintripi+0x47 |
|
--- interrupt --- |
|
i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37 |
|
Xintrltimer() at Xintrltimer+0x47 |
|
--- interrupt --- |
|
idle_loop+0x21: |
|
ddb{1}> |
|
</pre></blockquote> |
|
|
|
Repeat the <tt>machine ddbcpu x</tt> followed by <tt>trace</tt> for each |
|
processor in your machine. |
|
|
|
<h3>How do I gather further information from a kernel crash?</h3><p> |
|
|
|
A typical kernel crash on OpenBSD might look like this: |
|
(things to watch for are marked with bold font) |
|
|
|
<blockquote><pre> |
|
kernel: page fault trap, code=0 |
|
Stopped at <b>_pf_route+0x263</b>: mov 0x40(%edi),%edx |
|
ddb> |
|
</pre></blockquote> |
|
|
|
The first command to run from the <tt>ddb></tt> prompt is <tt>trace</tt> |
|
(see <a href="http://man.openbsd.org/ddb">ddb(4)</a> for details): |
|
|
|
<blockquote><pre> |
|
ddb> <b>trace</b> |
|
<b>_pf_route</b>(e28cb7e4,e28bc978,2,1fad,d0b8b120) at <b>_pf_route+0x263</b> |
|
_pf_test(2,1f4ad,e28cb7e4,b4c1) at _pf_test+0x706 |
|
_pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at _pf_route+0x207 |
|
_pf_test(2,d0a65440,e28cbb00,d023c282) at _pf_test+0x706 |
|
_ip_output(d0b6a200,0,0,0,0) at _ip_output+0xb67 |
|
_icmp_send(d0b6a200,0,1,a012) at _icmp_send+0x57 |
|
_icmp_reflect(d0b6a200,0,1,0,3) at _icmp_reflect+0x26b |
|
_icmp_input(d0b6a200,14,0,0,d0b6a200) at _icmp_input+0x42c |
|
_ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at _ipv4_input+0x6eb |
|
_ipintr(10,10,e289f140,e289f140,e28cbd38) at _ipintr+0x8d |
|
Bad frame pointer: 0xe28cbcac |
|
ddb> |
|
</pre></blockquote> |
|
|
|
This tells us what function calls lead to the crash. |
|
|
|
<p> |
|
To find out the particular line of C code that caused the crash, you can |
|
do the following: |
|
|
|
<p> |
|
Find the source file where the crashing function is defined in. |
|
In this example, that would be <tt>pf_route()</tt> in <tt>sys/net/pf.c</tt>. |
|
Recompile that source file with debug information: |
|
|
|
<blockquote><pre> |
|
# <b>cd /usr/src/sys/arch/$(uname -m)/compile/GENERIC</b> |
|
# <b>rm pf.o</b> |
|
# <b>DEBUG=-g make pf.o</b> |
|
</pre></blockquote> |
|
|
|
Then use <a href="http://man.openbsd.org/objdump">objdump(1)</a> to get the |
|
disassembly: |
|
|
|
<blockquote><pre> |
|
# <b>objdump --line --disassemble --reloc pf.o >pf.dis</b> |
|
</pre></blockquote> |
|
|
|
In the output, grep for the function name: |
|
|
|
<blockquote><pre> |
|
# <b>grep "<_pf_route>:" pf.dis</b> |
|
0000<b>7d88</b> <_pf_route>: |
|
</pre></blockquote> |
|
|
|
Take this first hex number and add the offset from the <tt>Stopped at</tt> line: |
|
<tt>0x7d88 + 0x263 == 0x7feb</tt>. |
|
|
|
<p> |
|
Scroll down to that line (the assembler instruction should match the one |
|
quoted in the <tt>Stopped at</tt> line), then up to the nearest C line number: |
|
|
|
<blockquote><pre> |
|
# <b>more pf.dis</b> |
|
/usr/src/sys/arch/i386/compile/GENERIC/../../../../net/pf.c:<b>3872</b> |
|
7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax |
|
<b>7feb</b>: 8b 57 40 mov 0x40(%edi),%edx |
|
7fee: 39 d0 cmp %edx,%eax |
|
7ff0: 0f 87 92 00 00 00 ja 8088 <_pf_route+0x300> |
|
</pre></blockquote> |
|
|
|
So, it's precisely line <tt>3872</tt> of <tt>pf.c</tt> that crashes: |
|
|
|
<blockquote><pre> |
|
# <b>cat -n pf.c | head -n 3872 | tail -n 1</b> |
|
3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) { |
|
</pre></blockquote> |
|
|
|
Note that the kernel that produced the crash output and the object file |
|
for objdump must be compiled from the exact same source file, otherwise |
|
the offsets won't match. |
|
|
|
<p> |
|
If you provide both the ddb trace output and the relevant objdump section, |
|
that's very helpful. |
|
|
<p> |
<p> |
</body> |
</body> |