ddb> show panic 0: kernel: page fault trap, code=0 ddb>
Repeat the machine ddbcpu x followed by trace for each processor in your machine.ddb{0}> trace pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226 fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5 fxp_intr(d0a62000) at fxp_intr+0x1e7 Xintr_ioapic0() at Xintr_ioapic0+0x6d --- interrupt --- idle_loop+0x21: ddb{0}> machine ddbcpu 1 Stopped at Debugger+0x4: leave ddb{1}> trace Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4 i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x 4a Xintripi() at Xintripi+0x47 --- interrupt --- i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37 Xintrltimer() at Xintrltimer+0x47 --- interrupt --- idle_loop+0x21: ddb{1}>
A typical kernel crash on OpenBSD might look like this: (things to watch for are marked with bold font)
The first command to run from the ddb> prompt is trace (see ddb(4) for details):kernel: page fault trap, code=0 Stopped at _pf_route+0x263: mov 0x40(%edi),%edx ddb>
This tells us what function calls lead to the crash.ddb> trace _pf_route(e28cb7e4,e28bc978,2,1fad,d0b8b120) at _pf_route+0x263 _pf_test(2,1f4ad,e28cb7e4,b4c1) at _pf_test+0x706 _pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at _pf_route+0x207 _pf_test(2,d0a65440,e28cbb00,d023c282) at _pf_test+0x706 _ip_output(d0b6a200,0,0,0,0) at _ip_output+0xb67 _icmp_send(d0b6a200,0,1,a012) at _icmp_send+0x57 _icmp_reflect(d0b6a200,0,1,0,3) at _icmp_reflect+0x26b _icmp_input(d0b6a200,14,0,0,d0b6a200) at _icmp_input+0x42c _ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at _ipv4_input+0x6eb _ipintr(10,10,e289f140,e289f140,e28cbd38) at _ipintr+0x8d Bad frame pointer: 0xe28cbcac ddb>
To find out the particular line of C code that caused the crash, you can do the following:
Find the source file where the crashing function is defined in. In this example, that would be pf_route() in sys/net/pf.c. Recompile that source file with debug information:
Then use objdump(1) to get the disassembly:# cd /usr/src/sys/arch/$(uname -m)/compile/GENERIC # rm pf.o # DEBUG=-g make pf.o
In the output, grep for the function name:# objdump --line --disassemble --reloc pf.o >pf.dis
Take this first hex number and add the offset from the Stopped at line: 0x7d88 + 0x263 == 0x7feb.# grep "<_pf_route>:" pf.dis 00007d88 <_pf_route>:
Scroll down to that line (the assembler instruction should match the one quoted in the Stopped at line), then up to the nearest C line number:
So, it's precisely line 3872 of pf.c that crashes:# more pf.dis /usr/src/sys/arch/i386/compile/GENERIC/../../../../net/pf.c:3872 7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax 7feb: 8b 57 40 mov 0x40(%edi),%edx 7fee: 39 d0 cmp %edx,%eax 7ff0: 0f 87 92 00 00 00 ja 8088 <_pf_route+0x300>
Note that the kernel that produced the crash output and the object file for objdump must be compiled from the exact same source file, otherwise the offsets won't match.# cat -n pf.c | head -n 3872 | tail -n 1 3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {
If you provide both the ddb trace output and the relevant objdump section, that's very helpful.