version 1.19, 2017/06/26 17:18:57 |
version 1.20, 2018/09/05 14:27:12 |
|
|
Under some circumstances, you may lose the very first message of a panic, |
Under some circumstances, you may lose the very first message of a panic, |
stating the reason for the panic. |
stating the reason for the panic. |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
ddb> <b>show panic</b> |
ddb> <b>show panic</b> |
0: kernel: page fault trap, code=0 |
0: kernel: page fault trap, code=0 |
ddb> |
ddb> |
</pre></blockquote> |
</pre> |
|
|
<h3>Note for SMP systems</h3> |
<h3>Note for SMP systems</h3> |
|
|
You should get a trace from each processor as part of your report: |
You should get a trace from each processor as part of your report: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
ddb{0}> <b>trace</b> |
ddb{0}> <b>trace</b> |
pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226 |
pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226 |
fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5 |
fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5 |
|
|
--- interrupt --- |
--- interrupt --- |
idle_loop+0x21: |
idle_loop+0x21: |
ddb{1}> |
ddb{1}> |
</pre></blockquote> |
</pre> |
|
|
Repeat the <tt>machine ddbcpu x</tt> followed by <tt>trace</tt> for each |
Repeat the <tt>machine ddbcpu x</tt> followed by <tt>trace</tt> for each |
processor in your machine. |
processor in your machine. |
|
|
|
|
A typical kernel crash on OpenBSD might look like this: |
A typical kernel crash on OpenBSD might look like this: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
kernel: page fault trap, code=0 |
kernel: page fault trap, code=0 |
Stopped at <b>pf_route+0x263</b>: mov 0x40(%edi),%edx |
Stopped at <b>pf_route+0x263</b>: mov 0x40(%edi),%edx |
ddb> |
ddb> |
</pre></blockquote> |
</pre> |
|
|
This crash happened at offset <tt>0x263</tt> in the function <tt>pf_route</tt>. |
This crash happened at offset <tt>0x263</tt> in the function <tt>pf_route</tt>. |
|
|
|
|
The first command to run from the |
The first command to run from the |
<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <tt>trace</tt>: |
<a href="https://man.openbsd.org/ddb">ddb(4)</a> prompt is <tt>trace</tt>: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
ddb> <b>trace</b> |
ddb> <b>trace</b> |
<b>pf_route</b>(e28cb7e4,e28bc978,2,1fad,d0b8b120) at <b>pf_route+0x263</b> |
<b>pf_route</b>(e28cb7e4,e28bc978,2,1fad,d0b8b120) at <b>pf_route+0x263</b> |
pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706 |
pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706 |
|
|
ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d |
ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d |
Bad frame pointer: 0xe28cbcac |
Bad frame pointer: 0xe28cbcac |
ddb> |
ddb> |
</pre></blockquote> |
</pre> |
|
|
This tells us what function calls lead to the crash. |
This tells us what function calls lead to the crash. |
|
|
|
|
Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the |
Use <a href="https://man.openbsd.org/objdump">objdump(1)</a> to get the |
disassembly: |
disassembly: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
$ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b> |
$ <b>cd /sys/arch/$(uname -m)/compile/GENERIC</b> |
$ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b> |
$ <b>objdump -dlr obj/pf.o >/tmp/pf.dis</b> |
</pre></blockquote> |
</pre> |
|
|
In the output, grep for the function name: |
In the output, grep for the function name: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
$ <b>grep "<pf_route>:" /tmp/pf.dis</b> |
$ <b>grep "<pf_route>:" /tmp/pf.dis</b> |
0000<b>7d88</b> <pf_route>: |
0000<b>7d88</b> <pf_route>: |
</pre></blockquote> |
</pre> |
|
|
Take this first hex number <tt>7d88</tt> and add the offset <tt>0x263</tt> from |
Take this first hex number <tt>7d88</tt> and add the offset <tt>0x263</tt> from |
the <tt>Stopped at</tt> line: |
the <tt>Stopped at</tt> line: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
$ <b>printf '%x\n' $((0x7d88 + 0x263))</b> |
$ <b>printf '%x\n' $((0x7d88 + 0x263))</b> |
7feb |
7feb |
</pre></blockquote> |
</pre> |
|
|
Scroll down to the line <tt>7feb</tt>. |
Scroll down to the line <tt>7feb</tt>. |
The assembler instruction should match the one quoted in the <tt>Stopped at</tt> |
The assembler instruction should match the one quoted in the <tt>Stopped at</tt> |
line. |
line. |
Then scroll up to the nearest C line number: |
Then scroll up to the nearest C line number: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
$ <b>more /tmp/pf.dis</b> |
$ <b>more /tmp/pf.dis</b> |
/sys/net/pf.c:<b>3872</b> |
/sys/net/pf.c:<b>3872</b> |
7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax |
7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax |
<b>7feb</b>: 8b 57 40 <b>mov 0x40(%edi),%edx</b> |
<b>7feb</b>: 8b 57 40 <b>mov 0x40(%edi),%edx</b> |
7fee: 39 d0 cmp %edx,%eax |
7fee: 39 d0 cmp %edx,%eax |
7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300> |
7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300> |
</pre></blockquote> |
</pre> |
|
|
So, it's precisely line <tt>3872</tt> of <tt>pf.c</tt> that crashes: |
So, it's precisely line <tt>3872</tt> of <tt>pf.c</tt> that crashes: |
|
|
<blockquote><pre> |
<pre class="cmdbox"> |
$ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b> |
$ <b>nl -ba /sys/net/pf.c | sed -n 3872p</b> |
3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) { |
3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) { |
</pre></blockquote> |
</pre> |
|
|
The kernel that produced the crash output and the object file for objdump must |
The kernel that produced the crash output and the object file for objdump must |
be compiled from the exact same source file, otherwise the offsets won't match. |
be compiled from the exact same source file, otherwise the offsets won't match. |