ddb> show panic 0: kernel: page fault trap, code=0 ddb>
ddb{0}> trace pool_get(d05e7c20,0,dab19ef8,d0169414,80) at pool_get+0x226 fxp_add_rfabuf(d0a62000,d3c12b00,dab19f10,dab19f10) at fxp_add_rfabuf+0xa5 fxp_intr(d0a62000) at fxp_intr+0x1e7 Xintr_ioapic0() at Xintr_ioapic0+0x6d --- interrupt --- idle_loop+0x21: ddb{0}> machine ddbcpu 1 Stopped at Debugger+0x4: leave ddb{1}> trace Debugger(d0319e28,d05ff5a0,dab1bee8,d031cc6e,d0a61800) at Debugger+0x4 i386_ipi_db(d0a61800,d05ff5a0,dab1bef8,d01eb997) at i386_ipi_db+0xb i386_ipi_handler(b0,d05f0058,dab10010,d01d0010,dab10010) at i386_ipi_handler+0x 4a Xintripi() at Xintripi+0x47 --- interrupt --- i386_softintlock(0,58,dab10010,dab10010,d01e0010) at i386_softintlock+0x37 Xintrltimer() at Xintrltimer+0x47 --- interrupt --- idle_loop+0x21: ddb{1}>Repeat the machine ddbcpu x followed by trace for each processor in your machine.
A typical kernel crash on OpenBSD might look like this:
kernel: page fault trap, code=0 Stopped at pf_route+0x263: mov 0x40(%edi),%edx ddb>This crash happened at offset 0x263 in the function pf_route.
The first command to run from the ddb(4) prompt is trace:
ddb> trace pf_route(e28cb7e4,e28bc978,2,1fad,d0b8b120) at pf_route+0x263 pf_test(2,1f4ad,e28cb7e4,b4c1) at pf_test+0x706 pf_route(e28cbb00,e28bc978,2,d0a65440,d0b8b120) at pf_route+0x207 pf_test(2,d0a65440,e28cbb00,d023c282) at pf_test+0x706 ip_output(d0b6a200,0,0,0,0) at ip_output+0xb67 icmp_send(d0b6a200,0,1,a012) at icmp_send+0x57 icmp_reflect(d0b6a200,0,1,0,3) at icmp_reflect+0x26b icmp_input(d0b6a200,14,0,0,d0b6a200) at icmp_input+0x42c ipv4_input(d0b6a200,e289f140,d0a489e0,e289f140) at ipv4_input+0x6eb ipintr(10,10,e289f140,e289f140,e28cbd38) at ipintr+0x8d Bad frame pointer: 0xe28cbcac ddb>This tells us what function calls lead to the crash.
To find out the particular line of C code that caused the crash, you can do the following:
Find the source file where the crashing function is defined. In this example, that would be pf_route() in /sys/net/pf.c. Use objdump(1) to get the disassembly:
$ cd /sys/arch/$(uname -m)/compile/GENERIC $ objdump -dlr obj/pf.o >/tmp/pf.disIn the output, grep for the function name:
$ grep "<pf_route>:" /tmp/pf.dis 00007d88 <pf_route>:Take this first hex number 7d88 and add the offset 0x263 from the Stopped at line:
$ printf '%x\n' $((0x7d88 + 0x263)) 7febScroll down to the line 7feb. The assembler instruction should match the one quoted in the Stopped at line. Then scroll up to the nearest C line number:
$ more /tmp/pf.dis /sys/net/pf.c:3872 7fe7: 0f b7 43 02 movzwl 0x2(%ebx),%eax 7feb: 8b 57 40 mov 0x40(%edi),%edx 7fee: 39 d0 cmp %edx,%eax 7ff0: 0f 87 92 00 00 00 ja 8088 <pf_route+0x300>So, it's precisely line 3872 of pf.c that crashes:
$ nl -ba /sys/net/pf.c | sed -n 3872p 3872 if ((u_int16_t)ip->ip_len <= ifp->if_mtu) {The kernel that produced the crash output and the object file for objdump must be compiled from the exact same source file, otherwise the offsets won't match.
If you provide both the ddb trace output and the relevant objdump section, that's very helpful.