version 1.20, 2018/09/05 14:27:12 |
version 1.21, 2019/05/27 22:55:19 |
|
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> |
<!doctype html> |
<html> |
<html lang=en> |
<head> |
<meta charset=utf-8> |
|
|
<title>OpenBSD: Crash Reports</title> |
<title>OpenBSD: Crash Reports</title> |
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> |
|
<meta name="description" content="How to report an OpenBSD kernel crash"> |
<meta name="description" content="How to report an OpenBSD kernel crash"> |
<meta name="copyright" content="This document copyright 1998-2016 by OpenBSD."> |
|
<meta name="viewport" content="width=device-width, initial-scale=1"> |
<meta name="viewport" content="width=device-width, initial-scale=1"> |
<link rel="stylesheet" type="text/css" href="openbsd.css"> |
<link rel="stylesheet" type="text/css" href="openbsd.css"> |
<link rel="canonical" href="https://www.openbsd.org/report.html"> |
<link rel="canonical" href="https://www.openbsd.org/report.html"> |
<style type="text/css"> |
<style> |
h3, h4 { color: #0000e0; } |
h3, h4 { |
|
color: var(--blue); |
|
} |
</style> |
</style> |
</head> |
|
|
|
<body bgcolor="#ffffff" text="#000000" link="#23238e"> |
<h2 id=OpenBSD> |
|
|
<h2> |
|
<a href="index.html"> |
<a href="index.html"> |
<font color="#0000ff"><i>Open</i></font><font color="#000084">BSD</font></a> |
<i>Open</i><b>BSD</b></a> |
<font color="#e00000">Crash Reports</font> |
Crash Reports |
</h2> |
</h2> |
|
|
<hr> |
<hr> |
<p> |
|
|
|
<h3>Minimum information for kernel problems</h3> |
<h3>Minimum information for kernel problems</h3> |
|
|
|
<p> |
Familiarize yourself with |
Familiarize yourself with |
<a href="report.html">the general bug reporting procedures</a> |
<a href="report.html">the general bug reporting procedures</a> |
first. |
first. |
|
|
When reporting a kernel panic or crash, please remember: |
When reporting a kernel panic or crash, please remember: |
|
|
<ul> |
<ul> |
<li><i>We need the console output on the screen</i>. |
<li><em>We need the console output on the screen</em>. |
Capture it and save it. |
Capture it and save it. |
Serial consoles are best, but if you are on a VGA console you can |
Serial consoles are best, but if you are on a VGA console you can |
<a href="faq/faq7.html">scroll the console back</a> |
<a href="faq/faq7.html">scroll the console back</a> |
and take readable pictures with a phone or camera.<br> |
and take readable pictures with a phone or camera. |
|
|
<li><i>If the kernel panicked we need the traceback.</i> |
<li><em>If the kernel panicked we need the traceback.</em> |
It may be displayed on the screen. |
It may be displayed on the screen. |
If you are at a |
If you are at a |
<tt><a href="https://man.openbsd.org/ddb.4">ddb</a>></tt> |
<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp> |
prompt, type <tt>trace</tt>. |
prompt, type <kbd>trace</kbd>. |
If you are running SMP, use the <tt>mach ddbcpu N</tt> command for each |
If you are running SMP, use the <kbd>mach ddbcpu N</kbd> command for each |
of the <tt>N</tt> processors you have and repeat the <tt>trace</tt> |
of the <var>N</var> processors you have and repeat the <kbd>trace</kbd> |
command for each processor.<br> |
command for each processor. |
|
|
<li><i>We need the process list.</i> |
<li><em>We need the process list.</em> |
Use the command <tt>ps</tt> to get that. |
Use the command <kbd>ps</kbd> to get that. |
</ul> |
</ul> |
|
|
<i> |
<p> |
|
<em> |
Reports without the above information are useless. |
Reports without the above information are useless. |
This is the minimum we need to be able to track down the issue. |
This is the minimum we need to be able to track down the issue. |
</i> |
</em> |
|
|
<h3>Additional information you can send</h3> |
<h3>Additional information you can send</h3> |
|
|
|
<p> |
In some situations more information is desirable. |
In some situations more information is desirable. |
Below are outlined some additional steps you can take in certain situations: |
Below are outlined some additional steps you can take in certain situations: |
|
|
|
|
The following additional things would be helpful |
The following additional things would be helpful |
<ul> |
<ul> |
<li>The output of the |
<li>The output of the |
<tt><a href="https://man.openbsd.org/ddb.4">ddb</a>></tt> command |
<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp> command |
<tt>show uvm</tt> |
<kbd>show uvm</kbd> |
<li>The output of the |
<li>The output of the |
<tt><a href="https://man.openbsd.org/ddb.4">ddb</a>></tt> |
<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp> |
command <tt>show bcstats</tt> |
command <kbd>show bcstats</kbd> |
<li>The output of the <tt>mount</tt> command from your running machine, so |
<li>The output of the <kbd>mount</kbd> command from your running machine, so |
we know what filesystems are mounted and how. |
we know what filesystems are mounted and how. |
</ul> |
</ul> |
<li> ... XXX boot crash? XXX |
<li> ... XXX boot crash? XXX |
|
|
|
|
<h3>Lost the panic message?</h3> |
<h3>Lost the panic message?</h3> |
|
|
|
<p> |
Under some circumstances, you may lose the very first message of a panic, |
Under some circumstances, you may lose the very first message of a panic, |
stating the reason for the panic. |
stating the reason for the panic. |
|
|
|
|
|
|
<h3>Note for SMP systems</h3> |
<h3>Note for SMP systems</h3> |
|
|
|
<p> |
You should get a trace from each processor as part of your report: |
You should get a trace from each processor as part of your report: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
|
|
ddb{1}> |
ddb{1}> |
</pre> |
</pre> |
|
|
Repeat the <tt>machine ddbcpu x</tt> followed by <tt>trace</tt> for each |
<p> |
|
Repeat the <code>machine ddbcpu x</code> followed by <code>trace</code> for each |
processor in your machine. |
processor in your machine. |
|
|
<h3>How do I gather further information from a kernel crash?</h3><p> |
<h3>How do I gather further information from a kernel crash?</h3><p> |
|
|
|
<p> |
A typical kernel crash on OpenBSD might look like this: |
A typical kernel crash on OpenBSD might look like this: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
|
|
ddb> |
ddb> |
</pre> |
</pre> |
|
|
This crash happened at offset <tt>0x263</tt> in the function <tt>pf_route</tt>. |
<p> |
|
This crash happened at offset <code>0x263</code> in the function <code>pf_route</code>. |
|
|
<p> |
<p> |
The first command to run from the |
The first command to run from the |
<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <tt>trace</tt>: |
<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <code>trace</code>: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
ddb> <b>trace</b> |
ddb> <b>trace</b> |
|
|
ddb> |
ddb> |
</pre> |
</pre> |
|
|
|
<p> |
This tells us what function calls lead to the crash. |
This tells us what function calls lead to the crash. |
|
|
<p> |
<p> |
|
|
|
|
<p> |
<p> |
Find the source file where the crashing function is defined. |
Find the source file where the crashing function is defined. |
In this example, that would be <tt>pf_route()</tt> in <tt>/sys/net/pf.c</tt>. |
In this example, that would be <code>pf_route()</code> in <code>/sys/net/pf.c</code>. |
Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the |
Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the |
disassembly: |
disassembly: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
$ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b> |
$ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b> |
$ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b> |
$ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b> |
</pre> |
</pre> |
|
|
|
<p> |
In the output, grep for the function name: |
In the output, grep for the function name: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
$ <b>grep "<pf_route>:" /tmp/pf.dis</b> |
$ <b>grep "<pf_route>:" /tmp/pf.dis</b> |
0000<b>7d88</b> <pf_route>: |
0000<b>7d88</b> <pf_route>: |
</pre> |
</pre> |
|
|
Take this first hex number <tt>7d88</tt> and add the offset <tt>0x263</tt> from |
<p> |
the <tt>Stopped at</tt> line: |
Take this first hex number <code>7d88</code> and add the offset <code>0x263</code> from |
|
the <code>Stopped at</code> line: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
$ <b>printf '%x\n' $((0x7d88 + 0x263))</b> |
$ <b>printf '%x\n' $((0x7d88 + 0x263))</b> |
7feb |
7feb |
</pre> |
</pre> |
|
|
Scroll down to the line <tt>7feb</tt>. |
<p> |
The assembler instruction should match the one quoted in the <tt>Stopped at</tt> |
Scroll down to the line <code>7feb</code>. |
|
The assembler instruction should match the one quoted in the <code>Stopped at</code> |
line. |
line. |
Then scroll up to the nearest C line number: |
Then scroll up to the nearest C line number: |
|
|
|
|
7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax |
7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax |
<b>7feb</b>: 8b 57 40 <b>mov 0x40(%edi),%edx</b> |
<b>7feb</b>: 8b 57 40 <b>mov 0x40(%edi),%edx</b> |
7fee: 39 d0 cmp %edx,%eax |
7fee: 39 d0 cmp %edx,%eax |
7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300> |
7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300> |
</pre> |
</pre> |
|
|
So, it's precisely line <tt>3872</tt> of <tt>pf.c</tt> that crashes: |
<p> |
|
So, it's precisely line <code>3872</code> of <code>pf.c</code> that crashes: |
|
|
<pre class="cmdbox"> |
<pre class="cmdbox"> |
$ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b> |
$ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b> |
3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) { |
3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) { |
</pre> |
</pre> |
|
|
|
<p> |
The kernel that produced the crash output and the object file for objdump must |
The kernel that produced the crash output and the object file for objdump must |
be compiled from the exact same source file, otherwise the offsets won't match. |
be compiled from the exact same source file, otherwise the offsets won't match. |
|
|
<p> |
<p> |
If you provide both the ddb trace output and the relevant objdump section, |
If you provide both the ddb trace output and the relevant objdump section, |
that's very helpful. |
that's very helpful. |
|
|
<p> |
|
</body> |
|
</html> |
|