ddb> show panic 0: kernel: page fault trap, code=0 ddb>
Repeat the machine ddbcpu x followed by trace for each processor in your machine.ddb{0}> trace pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226 fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5 fxp_intr(d0a62000) at fxp_intr+0x1e7 Xintr_ioapic0() at Xintr_ioapic0+0x6d --- interrupt --- idle_loop+0x21: ddb{0}> machine ddbcpu 1 Stopped at Debugger+0x4: leave ddb{1}> trace Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4 i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x 4a Xintripi() at Xintripi+0x47 --- interrupt --- i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37 Xintrltimer() at Xintrltimer+0x47 --- interrupt --- idle_loop+0x21: ddb{1}>
A typical kernel crash on OpenBSD might look like this:
This crash happened at offset 0x263 in the function pf_route.kernel: page fault trap, code=0 Stopped at pf_route+0x263: mov 0x40(%edi),%edx ddb>
The first command to run from the ddb(4) prompt is trace:
This tells us what function calls lead to the crash.ddb> trace pf_route(e28cb7e4,e28bc978,2,1fad,d0b8b120) at pf_route+0x263 pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706 pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at pf_route+0x207 pf_test(2,d0a65440,e28cbb00,d023c282) at pf_test+0x706 ip_output(d0b6a200,0,0,0,0) at ip_output+0xb67 icmp_send(d0b6a200,0,1,a012) at icmp_send+0x57 icmp_reflect(d0b6a200,0,1,0,3) at icmp_reflect+0x26b icmp_input(d0b6a200,14,0,0,d0b6a200) at icmp_input+0x42c ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at ipv4_input+0x6eb ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d Bad frame pointer: 0xe28cbcac ddb>
To find out the particular line of C code that caused the crash, you can do the following:
Find the source file where the crashing function is defined. In this example, that would be pf_route() in /sys/net/pf.c. Use objdump(1) to get the disassembly:
In the output, grep for the function name:$ cd /sys/arch/$(uname -m)/compile/GENERIC $ objdump -dlr obj/pf.o >/tmp/pf.dis
Take this first hex number 7d88 and add the offset 0x263 from the Stopped at line:$ grep "<pf_route>:" /tmp/pf.dis 00007d88 <pf_route>:
Scroll down to the line 7feb. The assembler instruction should match the one quoted in the Stopped at line. Then scroll up to the nearest C line number:$ printf '%x\n' $((0x7d88 + 0x263)) 7feb
So, it's precisely line 3872 of pf.c that crashes:$ more /tmp/pf.dis /sys/net/pf.c:3872 7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax 7feb: 8b 57 40 mov 0x40(%edi),%edx 7fee: 39 d0 cmp %edx,%eax 7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300>
The kernel that produced the crash output and the object file for objdump must be compiled from the exact same source file, otherwise the offsets won't match.$ nl -ba /sys/net/pf.c | sed -n 3872p 3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {
If you provide both the ddb trace output and the relevant objdump section, that's very helpful.