www/ddb.html - diff

Return to ddb.html CVS log

Up to [local] / www

Diff for /www/ddb.html between version 1.20 and 1.21

version 1.20, 2018/09/05 14:27:12

version 1.21, 2019/05/27 22:55:19

Line 1

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">

<!doctype html>

<html>

<head>

<title>OpenBSD: Crash Reports</title>

<style>

h3, h4 { color: #0000e0; }

h3, h4 {

color: var(--blue);

}

</style>

</head>

<h2>

<font color="#e00000">Crash Reports</font>

Crash Reports

</h2>

<hr>

<p>

<h3>Minimum information for kernel problems</h3>

<p>

Familiarize yourself with

<a href="report.html">the general bug reporting procedures</a>

first.

Line 32

Line 31

When reporting a kernel panic or crash, please remember:

<ul>

<li><i>We need the console output on the screen</i>.

<li><em>We need the console output on the screen</em>.

Capture it and save it.

Serial consoles are best, but if you are on a VGA console you can

<a href="faq/faq7.html">scroll the console back</a>

and take readable pictures with a phone or camera.<br>

and take readable pictures with a phone or camera.

<li><i>If the kernel panicked we need the traceback.</i>

<li><em>If the kernel panicked we need the traceback.</em>

It may be displayed on the screen.

If you are at a

prompt, type <tt>trace</tt>.

prompt, type <kbd>trace</kbd>.

If you are running SMP, use the <tt>mach ddbcpu N</tt> command for each

If you are running SMP, use the <kbd>mach ddbcpu N</kbd> command for each

of the <tt>N</tt> processors you have and repeat the <tt>trace</tt>

of the <var>N</var> processors you have and repeat the <kbd>trace</kbd>

command for each processor.<br>

command for each processor.

<li><i>We need the process list.</i>

<li><em>We need the process list.</em>

Use the command <tt>ps</tt> to get that.

Use the command <kbd>ps</kbd> to get that.

</ul>

<i>

<p>

<em>

Reports without the above information are useless.

This is the minimum we need to be able to track down the issue.

</i>

</em>

<h3>Additional information you can send</h3>

<p>

In some situations more information is desirable.

Below are outlined some additional steps you can take in certain situations:

Line 66

Line 67

The following additional things would be helpful

<ul>

<li>The output of the

<tt><a href="https://man.openbsd.org/ddb.4">ddb</a>></tt> command

<samp><a href="https://man.openbsd.org/ddb.4">ddb</a>></samp> command

<li>The output of the

command <tt>show bcstats</tt>

command <kbd>show bcstats</kbd>

<li>The output of the <tt>mount</tt> command from your running machine, so

<li>The output of the <kbd>mount</kbd> command from your running machine, so

we know what filesystems are mounted and how.

</ul>

<li> ... XXX boot crash? XXX

Line 80

Line 81

<h3>Lost the panic message?</h3>

<p>

Under some circumstances, you may lose the very first message of a panic,

stating the reason for the panic.

Line 91

Line 93

<h3>Note for SMP systems</h3>

<p>

You should get a trace from each processor as part of your report:

Line 117

Line 120

ddb{1}>

</pre>

Repeat the <tt>machine ddbcpu x</tt> followed by <tt>trace</tt> for each

<p>

Repeat the <code>machine ddbcpu x</code> followed by <code>trace</code> for each

processor in your machine.

<h3>How do I gather further information from a kernel crash?</h3><p>

<p>

A typical kernel crash on OpenBSD might look like this:

Line 130

Line 135

ddb>

</pre>

This crash happened at offset <tt>0x263</tt> in the function <tt>pf_route</tt>.

<p>

This crash happened at offset <code>0x263</code> in the function <code>pf_route</code>.

<p>

The first command to run from the

<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <tt>trace</tt>:

<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <code>trace</code>:

ddb> <b>trace</b>

Line 152

Line 158

ddb>

</pre>

<p>

This tells us what function calls lead to the crash.

<p>

Line 160

Line 167

<p>

Find the source file where the crashing function is defined.

In this example, that would be <tt>pf_route()</tt> in <tt>/sys/net/pf.c</tt>.

In this example, that would be <code>pf_route()</code> in <code>/sys/net/pf.c</code>.

Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the

disassembly:

$ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b>

$ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b>

</pre>

<p>

In the output, grep for the function name:

$ <b>grep "<pf_route>:" /tmp/pf.dis</b>

0000<b>7d88</b> <pf_route>:

</pre>

Take this first hex number <tt>7d88</tt> and add the offset <tt>0x263</tt> from

<p>

the <tt>Stopped at</tt> line:

Take this first hex number <code>7d88</code> and add the offset <code>0x263</code> from

the <code>Stopped at</code> line:

$ <b>printf '%x\n' $((0x7d88 + 0x263))</b>

7feb

</pre>

Scroll down to the line <tt>7feb</tt>.

<p>

The assembler instruction should match the one quoted in the <tt>Stopped at</tt>

Scroll down to the line <code>7feb</code>.

The assembler instruction should match the one quoted in the <code>Stopped at</code>

line.

Then scroll up to the nearest C line number:

Line 195

Line 205

7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax

7fee: 39 d0 cmp %edx,%eax

7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300>

</pre>

So, it's precisely line <tt>3872</tt> of <tt>pf.c</tt> that crashes:

<p>

So, it's precisely line <code>3872</code> of <code>pf.c</code> that crashes:

$ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b>

3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {

</pre>

<p>

The kernel that produced the crash output and the object file for objdump must

be compiled from the exact same source file, otherwise the offsets won't match.

<p>

If you provide both the ddb trace output and the relevant objdump section,

that's very helpful.

<p>

</body>

</html>