Discussion:
How to get anything useful out of kgdb?
(too old to reply)
Sean Bruno
2015-05-09 17:20:46 UTC
Permalink
tl;dr What are the kernel config options to get good output of kgdb?


I'm trying to get the ability to debug and display internal variables,
as one does, with kgdb. I'm *must* be doing this wrong as I cannot get
any useful output from accessing variables that were JUST accessed in
order to invoke a panic() that I have inserted. This is a GENERIC
kernel without INVARIANTS and without WITNESS:

https://people.freebsd.org/~sbruno/wtf_kgdb.txt

I seem to have debug enabled and am able to browse source, but I
obviously haven't compiled correctly as things are optimized out.

sean
Sean Bruno
2015-05-09 18:02:57 UTC
Permalink
Post by Sean Bruno
tl;dr What are the kernel config options to get good output of kgdb?
I'm trying to get the ability to debug and display internal
variables, as one does, with kgdb. I'm *must* be doing this wrong
as I cannot get any useful output from accessing variables that
were JUST accessed in order to invoke a panic() that I have
inserted. This is a GENERIC kernel without INVARIANTS and without
https://people.freebsd.org/~sbruno/wtf_kgdb.txt
I seem to have debug enabled and am able to browse source, but I
obviously haven't compiled correctly as things are optimized out.
Just to see what the arguments to make are when building, I touched
if_em.c and did a NOCLEAN rebuild.

What I see is this:
https://people.freebsd.org/~sbruno/wtf_make.txt

Should -02 and -g do anything useful here when compiling? Do I need
to drop the optimization flags?

sean
Alexander Kabaev
2015-05-09 23:03:47 UTC
Permalink
On Sat, 09 May 2015 11:30:59 -0700
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
You could try -O0, but also if you just care about txr, go up to
frame 12 where txr initialized to the context arg to em_handle_tx()
and print it out there.
-j
I'm guessing that the place to change -O2 -> -O0 is in kern.pre.mk ?
sean
No, it means you need to iescover DEBUG and how it affects optimization
level :)

.if defined(DEBUG)
_MINUS_O= -O
CTFFLAGS+= -g
.else
...

Say, I have 'makeoptions DEBUG="-g -gdwarf-2"' in my kernel config
file. -gdwarf-2 is probably not required anymore.

--
Alexander Kabaev
Sean Bruno
2015-05-10 02:00:05 UTC
Permalink
Post by Alexander Kabaev
I'm guessing that the place to change -O2 -> -O0 is in
kern.pre.mk ?
sean
No, it means you need to iescover DEBUG and how it affects
optimization level :)
.if defined(DEBUG) _MINUS_O= -O CTFFLAGS+= -g .else ...
Say, I have 'makeoptions DEBUG="-g -gdwarf-2"' in my kernel
config file. -gdwarf-2 is probably not required anymore.
Indeed! :-)

I was directed to go a slightly different way:

make.conf:
COPTFLAGS=-O0
CFLAGS=-O0


This makes unbootable kernels and they panic unless you defined the
following in your kernel config:
options KSTACK_PAGES=6


sean
Sean Bruno
2015-05-15 15:27:17 UTC
Permalink
Post by Sean Bruno
tl;dr What are the kernel config options to get good output of kgdb?
I'm trying to get the ability to debug and display internal
variables, as one does, with kgdb. I'm *must* be doing this wrong
as I cannot get any useful output from accessing variables that
were JUST accessed in order to invoke a panic() that I have
inserted. This is a GENERIC kernel without INVARIANTS and without
https://people.freebsd.org/~sbruno/wtf_kgdb.txt
I seem to have debug enabled and am able to browse source, but I
obviously haven't compiled correctly as things are optimized out.
sean
A little bit of progress and learning on my side, so that's good.
I'll be patching gdb with a couple of 10-year old upstream commits
later today to squash some type missing errors, "#0 doadump
(textdump=Unhandled dwarf expression opcode 0x93". Review is here
https://reviews.freebsd.org/D2534

sean
John Baldwin
2015-05-15 15:50:38 UTC
Permalink
Post by Sean Bruno
tl;dr What are the kernel config options to get good output of kgdb?
I'm trying to get the ability to debug and display internal variables,
as one does, with kgdb. I'm *must* be doing this wrong as I cannot get
any useful output from accessing variables that were JUST accessed in
order to invoke a panic() that I have inserted. This is a GENERIC
https://people.freebsd.org/~sbruno/wtf_kgdb.txt
I seem to have debug enabled and am able to browse source, but I
obviously haven't compiled correctly as things are optimized out.
1) gdb7 does a better job. I hope to get the kgdb patches into the
port soon. If you are feeling brave:

# add texinfo for HEAD
% pkg install gmake bison

% git clone https://github.com/bsdjhb/gdb.git
% cd gdb
% git checkout freebsd-7.9.0-kgdb
% fetch http://people.freebsd.org/~jhb/gdb/build
% sh ./build
% cd obj # or obj.i386 for i386
% gmake

Then use /path/to/git/gdb/obj/gdb/kgdb

2) Even with gdb7 it can't figure out variables that it should figure out
sometimes. Other options are either to find the variable in a higher
frame (e.g. if it is something like 'td' or a driver softc) or to start
poking around in the dissassembly to work out which register it is in
(or which register points to a structure that contains it) and go from
there.
--
John Baldwin
Sean Bruno
2015-05-15 16:00:03 UTC
Permalink
Post by John Baldwin
Post by Sean Bruno
tl;dr What are the kernel config options to get good output of kgdb?
I'm trying to get the ability to debug and display internal
variables, as one does, with kgdb. I'm *must* be doing this
wrong as I cannot get any useful output from accessing variables
that were JUST accessed in order to invoke a panic() that I have
inserted. This is a GENERIC kernel without INVARIANTS and
https://people.freebsd.org/~sbruno/wtf_kgdb.txt
I seem to have debug enabled and am able to browse source, but I
obviously haven't compiled correctly as things are optimized
out.
1) gdb7 does a better job. I hope to get the kgdb patches into
# add texinfo for HEAD % pkg install gmake bison
% git clone https://github.com/bsdjhb/gdb.git % cd gdb % git
checkout freebsd-7.9.0-kgdb % fetch
http://people.freebsd.org/~jhb/gdb/build % sh ./build % cd obj #
or obj.i386 for i386 % gmake
Then use /path/to/git/gdb/obj/gdb/kgdb
2) Even with gdb7 it can't figure out variables that it should
figure out sometimes. Other options are either to find the
variable in a higher frame (e.g. if it is something like 'td' or a
driver softc) or to start poking around in the dissassembly to work
out which register it is in (or which register points to a
structure that contains it) and go from there.
So one non-obvious thing that I'd like an explanation about: If I
manually break to the debbugger and cause a dump (doadump), how do I
poke around in the crash dump later on to find a thread that I'm
looking for, e.g. I want to poke at various bits inside em(4).

sean
John Baldwin
2015-05-15 16:08:09 UTC
Permalink
Post by Sean Bruno
Post by John Baldwin
Post by Sean Bruno
tl;dr What are the kernel config options to get good output of kgdb?
I'm trying to get the ability to debug and display internal
variables, as one does, with kgdb. I'm *must* be doing this
wrong as I cannot get any useful output from accessing variables
that were JUST accessed in order to invoke a panic() that I have
inserted. This is a GENERIC kernel without INVARIANTS and
https://people.freebsd.org/~sbruno/wtf_kgdb.txt
I seem to have debug enabled and am able to browse source, but I
obviously haven't compiled correctly as things are optimized out.
1) gdb7 does a better job. I hope to get the kgdb patches into
# add texinfo for HEAD % pkg install gmake bison
% git clone https://github.com/bsdjhb/gdb.git % cd gdb % git
checkout freebsd-7.9.0-kgdb % fetch
http://people.freebsd.org/~jhb/gdb/build % sh ./build % cd obj #
or obj.i386 for i386 % gmake
Then use /path/to/git/gdb/obj/gdb/kgdb
2) Even with gdb7 it can't figure out variables that it should
figure out sometimes. Other options are either to find the
variable in a higher frame (e.g. if it is something like 'td' or a
driver softc) or to start poking around in the dissassembly to work
out which register it is in (or which register points to a
structure that contains it) and go from there.
So one non-obvious thing that I'd like an explanation about: If I
manually break to the debbugger and cause a dump (doadump), how do I
poke around in the crash dump later on to find a thread that I'm
looking for, e.g. I want to poke at various bits inside em(4).
The current thread in a crashdump is the one that calls dump or panic.
If you want to find a different thread you have a couple of options.
First, kgdb exposes each kernel thread as a thread to gdb. You can
switch to individual kernel threads by using 'tid N' (where N
is td_tid) or 'proc N' (where N is a pid). You can also use 'thread N',
but that requires using gdb's internal thread number which has no
relation to tids or pids. There are several ways to enumerate the
list of threads in a core:

1) You can use 'info threads' in kgdb to get a list of threads.

2) You can run 'ps' against the crashdump using -N and -M to get the
tid and pid values (at least until someone like glebius@ goes on more
of an anti-libkvm tirade and removes crashdump support from ps).
crashinfo runs ps by default, so if you have that enabled you can look
in core.txt.N to get the list of tids / pids.

3) You can grab my gdb scripts at ~jhb/gdb/ and source 'gdb6'. That
gives you a 'ps' command kind of like DDB's ps.
--
John Baldwin
Sean Bruno
2015-05-15 16:23:54 UTC
Permalink
Post by John Baldwin
Post by Sean Bruno
Post by John Baldwin
Post by Sean Bruno
tl;dr What are the kernel config options to get good output of kgdb?
I'm trying to get the ability to debug and display internal
variables, as one does, with kgdb. I'm *must* be doing this
wrong as I cannot get any useful output from accessing
variables that were JUST accessed in order to invoke a
panic() that I have inserted. This is a GENERIC kernel
https://people.freebsd.org/~sbruno/wtf_kgdb.txt
I seem to have debug enabled and am able to browse source,
but I obviously haven't compiled correctly as things are
optimized out.
1) gdb7 does a better job. I hope to get the kgdb patches
# add texinfo for HEAD % pkg install gmake bison
% git clone https://github.com/bsdjhb/gdb.git % cd gdb % git
checkout freebsd-7.9.0-kgdb % fetch
http://people.freebsd.org/~jhb/gdb/build % sh ./build % cd obj
# or obj.i386 for i386 % gmake
Then use /path/to/git/gdb/obj/gdb/kgdb
2) Even with gdb7 it can't figure out variables that it should
figure out sometimes. Other options are either to find the
variable in a higher frame (e.g. if it is something like 'td'
or a driver softc) or to start poking around in the
dissassembly to work out which register it is in (or which
register points to a structure that contains it) and go from
there.
So one non-obvious thing that I'd like an explanation about: If
I manually break to the debbugger and cause a dump (doadump), how
do I poke around in the crash dump later on to find a thread that
I'm looking for, e.g. I want to poke at various bits inside
em(4).
The current thread in a crashdump is the one that calls dump or
panic. If you want to find a different thread you have a couple of
options. First, kgdb exposes each kernel thread as a thread to gdb.
You can switch to individual kernel threads by using 'tid N' (where
N is td_tid) or 'proc N' (where N is a pid). You can also use
'thread N', but that requires using gdb's internal thread number
which has no relation to tids or pids. There are several ways to
1) You can use 'info threads' in kgdb to get a list of threads.
2) You can run 'ps' against the crashdump using -N and -M to get
on more of an anti-libkvm tirade and removes crashdump support from
ps). crashinfo runs ps by default, so if you have that enabled you
can look in core.txt.N to get the list of tids / pids.
3) You can grab my gdb scripts at ~jhb/gdb/ and source 'gdb6'.
That gives you a 'ps' command kind of like DDB's ps.
OMG ... thank you. :-)

I'm guessing, that from the list here, no useful information is going
to be available. It looks like all my interrupt threads are idle and
all taskqueue threads are asleep:

59 Thread 100049 (PID=12: intr/irq257: em0:rx0) sched_switch
(td=0xfffff80002c47940, newtd=<value optimized out>, flags=<value
optimized out>) at /home/sbruno/bsd/em_mq/sys/kern/sched_ule.c:1956
58 Thread 100051 (PID=12: intr/irq258: em0:rx1) sched_switch
(td=0xfffff80002c47000, newtd=<value optimized out>, flags=<value
optimized out>) at /home/sbruno/bsd/em_mq/sys/kern/sched_ule.c:1956
57 Thread 100053 (PID=12: intr/irq259: em0:tx0) sched_switch
(td=0xfffff80002c424a0, newtd=<value optimized out>, flags=<value
optimized out>) at /home/sbruno/bsd/em_mq/sys/kern/sched_ule.c:1956
56 Thread 100055 (PID=12: intr/irq260: em0:tx1) sched_switch
(td=0xfffff80002c5e940, newtd=<value optimized out>, flags=<value
optimized out>) at /home/sbruno/bsd/em_mq/sys/kern/sched_ule.c:1956
55 Thread 100057 (PID=12: intr/irq261: em0:link) sched_switch
(td=0xfffff80002c5e000, newtd=<value optimized out>, flags=<value
optimized out>) at /home/sbruno/bsd/em_mq/sys/kern/sched_ule.c:1956
54 Thread 100058 (PID=12: intr/irq262: em1:rx0) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
53 Thread 100060 (PID=12: intr/irq263: em1:rx1) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
52 Thread 100062 (PID=12: intr/irq264: em1:tx0) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
51 Thread 100064 (PID=12: intr/irq265: em1:tx1) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
50 Thread 100066 (PID=12: intr/irq266: em1:link) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
Ryan Stone
2015-05-15 16:45:10 UTC
Permalink
Post by Sean Bruno
54 Thread 100058 (PID=12: intr/irq262: em1:rx0) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
53 Thread 100060 (PID=12: intr/irq263: em1:rx1) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
52 Thread 100062 (PID=12: intr/irq264: em1:tx0) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
51 Thread 100064 (PID=12: intr/irq265: em1:tx1) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
50 Thread 100066 (PID=12: intr/irq266: em1:link) cpustop_handler ()
at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
These ones were all running at the time.
Sean Bruno
2015-05-15 17:07:56 UTC
Permalink
On Fri, May 15, 2015 at 12:23 PM, Sean Bruno
Post by Sean Bruno
54 Thread 100058 (PID=12: intr/irq262: em1:rx0) cpustop_handler
() at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987 53 Thread
100060 (PID=12: intr/irq263: em1:rx1) cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987 52 Thread 100062
(PID=12: intr/irq264: em1:tx0) cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987 51 Thread 100064
(PID=12: intr/irq265: em1:tx1) cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987 50 Thread 100066
(PID=12: intr/irq266: em1:link) cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
These ones were all running at the time.
Hrm, when I look at them directly in the crashdump, I don't see
anything useful.

(kgdb) tid 100058
[Switching to thread 54 (Thread 100058)]#0 cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
987 CPU_SET_ATOMIC(cpu, &stopped_cpus);
Current language: auto; currently minimal
(kgdb) whe
#0 cpustop_handler () at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:98
7
#1 0xffffffff80f76f7a in ipi_nmi_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:969
#2 0xffffffff80e3657a in trap (frame=0xffffffff817eb910) at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/trap.c:188
#3 0xffffffff80e1b273 in nmi_calltrap () at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/exception.S:509
#4 0x0000000800841841 in ?? ()
Previous frame inner to this frame (corrupt stack?)


Is "what I want to do" not really possible?

sean
Ryan Stone
2015-05-15 17:57:03 UTC
Permalink
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Hrm, when I look at them directly in the crashdump, I don't see
anything useful.
(kgdb) tid 100058
[Switching to thread 54 (Thread 100058)]#0 cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987
987 CPU_SET_ATOMIC(cpu, &stopped_cpus);
Current language: auto; currently minimal
(kgdb) whe
#0 cpustop_handler () at /home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:98
7
#1 0xffffffff80f76f7a in ipi_nmi_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:969
#2 0xffffffff80e3657a in trap (frame=0xffffffff817eb910) at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/trap.c:188
#3 0xffffffff80e1b273 in nmi_calltrap () at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/exception.S:509
#4 0x0000000800841841 in ?? ()
Previous frame inner to this frame (corrupt stack?)
*Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to
figure out where it was running:

frame 2
info line *frame->tf_rip

That gives you the top of the callstack at the time that the core was
taken. To get the rest of it, try:

define trace_stack
set $frame_ptr=$arg0
set $iters=0
while $frame_ptr != 0 && $iters < $arg1
set $ret_addr=((char*)$frame_ptr) + sizeof(void*)
printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr,
*(void**)$ret_addr
printf " "
info line **(void***)$ret_addr
set $frame_ptr=*(void**)$frame_ptr
set $iters=$iters+1
end
end

trace_stack frame->tf_rbp 20
Sean Bruno
2015-05-15 19:20:17 UTC
Permalink
On Fri, May 15, 2015 at 1:07 PM, Sean Bruno
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Hrm, when I look at them directly in the crashdump, I don't see
anything useful.
(kgdb) tid 100058 [Switching to thread 54 (Thread 100058)]#0
cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:987 987
CPU_SET_ATOMIC(cpu, &stopped_cpus); Current language: auto;
currently minimal (kgdb) whe #0 cpustop_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:98 7 #1
0xffffffff80f76f7a in ipi_nmi_handler () at
/home/sbruno/bsd/em_mq/sys/x86/x86/mp_x86.c:969 #2
0xffffffff80e3657a in trap (frame=0xffffffff817eb910) at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/trap.c:188 #3
0xffffffff80e1b273 in nmi_calltrap () at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/exception.S:509 #4
0x0000000800841841 in ?? () Previous frame inner to this frame
(corrupt stack?)
*Sigh*, kgdb isn't unwinding the trap frame properly. You can try
frame 2 info line *frame->tf_rip
I'm guessing that we are just at the limit of what the intree kgdb is
capable of doing with out crashdumps.

#2 0xffffffff80e3657a in trap (frame=0xffffffff817eb910) at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/trap.c:188
188 if (ipi_nmi_handler() == 0)
(kgdb) p frame
$5 = (struct trapframe *) 0xffffffff817eb910
(kgdb) p *frame
$6 = {tf_rdi = 34389196884, tf_rsi = 34389192960, tf_rdx = 0, tf_rcx =
360, tf_r8 = 0, tf_r9 = -8795456263872, tf_rax = 0, tf_rbx =
34393489408, tf_rbp = 140736951475936, tf_r10 = 17232, tf_r11 = 583,
tf_r12 = 1882455366,
tf_r13 = 34389196880, tf_r14 = 0, tf_r15 = 6358856, tf_trapno = 19,
tf_fs = 19, tf_gs = 27, tf_addr = 0, tf_flags = 1, tf_es = 59, tf_ds =
59, tf_err = 0, tf_rip = 34368395329, tf_cs = 67, tf_rflags = 518,
tf_rsp = 140736951475912,
tf_ss = 59}
(kgdb) p *frame->tf_rip
Cannot access memory at address 0x800841841
(kgdb) info line *frame->tf_rip
No line number information available for address 0x800841841
(kgdb)
Andriy Gapon
2015-10-01 21:56:44 UTC
Permalink
Post by Sean Bruno
I'm guessing that we are just at the limit of what the intree kgdb is
capable of doing with out crashdumps.
It's just that a userland process was running on a CPU when it got an NMI.
Post by Sean Bruno
#2 0xffffffff80e3657a in trap (frame=0xffffffff817eb910) at
/home/sbruno/bsd/em_mq/sys/amd64/amd64/trap.c:188
188 if (ipi_nmi_handler() == 0)
(kgdb) p frame
$5 = (struct trapframe *) 0xffffffff817eb910
(kgdb) p *frame
$6 = {tf_rdi = 34389196884, tf_rsi = 34389192960, tf_rdx = 0, tf_rcx =
360, tf_r8 = 0, tf_r9 = -8795456263872, tf_rax = 0, tf_rbx =
34393489408, tf_rbp = 140736951475936, tf_r10 = 17232, tf_r11 = 583,
tf_r12 = 1882455366,
tf_r13 = 34389196880, tf_r14 = 0, tf_r15 = 6358856, tf_trapno = 19,
tf_fs = 19, tf_gs = 27, tf_addr = 0, tf_flags = 1, tf_es = 59, tf_ds =
59, tf_err = 0, tf_rip = 34368395329, tf_cs = 67, tf_rflags = 518,
tf_rsp = 140736951475912,
tf_ss = 59}
(kgdb) p *frame->tf_rip
Cannot access memory at address 0x800841841
(kgdb) info line *frame->tf_rip
No line number information available for address 0x800841841
(kgdb)
--
Andriy Gapon
Andriy Gapon
2015-10-02 06:26:23 UTC
Permalink
Post by Ryan Stone
*Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to
I wonder, what is a reason for this?
Can that be fixed in kgdb itself?
It seems that usually kgdb handles trap frames just fine, but not always?
Post by Ryan Stone
That gives you the top of the callstack at the time that the core was
define trace_stack
set $frame_ptr=$arg0
set $iters=0
while $frame_ptr != 0 && $iters < $arg1
set $ret_addr=((char*)$frame_ptr) + sizeof(void*)
printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr, *(void**)$ret_addr
printf " "
info line **(void***)$ret_addr
set $frame_ptr=*(void**)$frame_ptr
set $iters=$iters+1
end
end
trace_stack frame->tf_rbp 20
Thank you for this script.
Here is an example from my practice.

(kgdb) bt
#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291
#1 0xffffffff8063453f in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:359
#2 0xffffffff80634ba4 in vpanic (fmt=<value optimized out>, ap=<value optimized
out>) at /usr/src/sys/kern/kern_shutdown.c:635
#3 0xffffffff806348a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:568
#4 0xffffffff8041bba7 in db_panic (addr=<value optimized out>, have_addr=false,
count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473
#5 0xffffffff8041b67b in db_command (cmd_table=0x0) at
/usr/src/sys/ddb/db_command.c:440
#6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#7 0xffffffff8041de0b in db_trap (type=<value optimized out>, code=0) at
/usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0) at
/usr/src/sys/kern/subr_kdb.c:653
#9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at
/usr/src/sys/amd64/amd64/trap.c:381
#10 0xffffffff80809623 in nmi_calltrap () at
/usr/src/sys/libkern/explicit_bzero.c:28
#11 0xffffffff80619e1f in __mtx_assert (c=<value optimized out>, what=<value
optimized out>, file=<value optimized out>, line=<value optimized out>) at
/usr/src/sys/kern/kern_mutex.c:842
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal

(kgdb) fr 9
#9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at
/usr/src/sys/amd64/amd64/trap.c:381
381 kdb_trap(type, 0, frame);

(kgdb) trace_stack frame->tf_rbp 20
frameptr=0xfffffe02b8356e90, ret_addr=0xffffffff807fef86
Line 833 of "/usr/src/sys/vm/vm_reserv.c" starts at address
0xffffffff807fef86 <vm_reserv_free_page+38> and ends at 0xffffffff807fef90
<vm_reserv_free_page+48>.
frameptr=0xfffffe02b8356eb0, ret_addr=0xffffffff807f2b96
Line 2432 of "/usr/src/sys/vm/vm_page.c" starts at address
0xffffffff807f2b96 <vm_page_free_toq+262> and ends at 0xffffffff807f2b9c
<vm_page_free_toq+268>.
frameptr=0xfffffe02b8356ed0, ret_addr=0xffffffff807f2e4d
Line 963 of "/usr/src/sys/vm/vm_page.c" starts at address 0xffffffff807f2e4d
<vm_page_free+13> and ends at 0xffffffff807f2e50 <vm_page_free_zero>.
frameptr=0xfffffe02b8356ee0, ret_addr=0xffffffff821c28e2
Line 268 of
"/usr/src/sys/modules/drm2/drm2/../../../dev/drm2/ttm/ttm_bo_vm.c" starts at
address 0xffffffff821c28e2 <ttm_bo_vm_fault+1010> and ends at 0xffffffff821c28ee
<ttm_bo_vm_fault+1022>.
frameptr=0xfffffe02b8356f50, ret_addr=0xffffffff807d4fd3
Line 321 of "/usr/src/sys/vm/device_pager.c" starts at address
0xffffffff807d4fce <dev_pager_getpages+94> and ends at 0xffffffff807d4fdb
<dev_pager_getpages+107>.
frameptr=0xfffffe02b8356fa0, ret_addr=0xffffffff807f9d67
Line 291 of "/usr/src/sys/vm/vm_pager.c" starts at address
0xffffffff807f9d58 <vm_pager_get_pages+40> and ends at 0xffffffff807f9d6a
<vm_pager_get_pages+58>.
frameptr=0xfffffe02b8356fd0, ret_addr=0xffffffff807e0d84
Line 675 of "/usr/src/sys/vm/vm_fault.c" starts at address
0xffffffff807e0d84 <vm_fault_hold+1860> and ends at 0xffffffff807e0d8d
<vm_fault_hold+1869>.
frameptr=0xfffffe02b83578f0, ret_addr=0xffffffff807e05ee
Line 277 of "/usr/src/sys/vm/vm_fault.c" starts at address
0xffffffff807e05d9 <vm_fault+121> and ends at 0xffffffff807e05f1 <vm_fault+145>.
frameptr=0xfffffe02b8357930, ret_addr=0xffffffff80821342
Line 735 of "/usr/src/sys/amd64/amd64/trap.c" starts at address
0xffffffff80821342 <trap_pfault+290> and ends at 0xffffffff80821346
<trap_pfault+294>.
frameptr=0xfffffe02b83579c0, ret_addr=0xffffffff80820bda
Line 326 of "/usr/src/sys/amd64/amd64/trap.c" starts at address
0xffffffff80820bc6 <trap+1366> and ends at 0xffffffff80820bdf <trap+1391>.
frameptr=0xfffffe02b8357bd0, ret_addr=0xffffffff8082154a
Line 629 of "/usr/src/sys/amd64/amd64/trap.c" starts at address
0xffffffff8082154a <trap_check+42> and ends at 0xffffffff80821560
<dblfault_handler>.
frameptr=0xfffffe02b8357bf0, ret_addr=0xffffffff808091e3
Line 28 of "/usr/src/sys/libkern/explicit_bzero.c" starts at address
0xffffffff806e74dd <explicit_bzero+29> and ends at 0xffffffff8088a2d0
<__explicit_bzero_hook>.
frameptr=0x7fffffffe8f0Cannot access memory at address 0x7fffffffe8f8

Output of trace_stack looks perfectly sane for me up to the next trap frame.
--
Andriy Gapon
John Baldwin
2015-10-02 16:12:58 UTC
Permalink
Post by Andriy Gapon
Post by Ryan Stone
*Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to
I wonder, what is a reason for this?
Can that be fixed in kgdb itself?
It seems that usually kgdb handles trap frames just fine, but not always?
It should be fixable. If this doesn't work under newer kgdb let me know and I'll
try to fix it. I did fix a few edge cases with special frame handling in the
newer kgdb though those mostly had to do with fork_trampoline and possibly
Xtimerint (and aside from fork_trampoline I think the fixes were mostly for i386
where different handlers setup trapframes differently)
Post by Andriy Gapon
Post by Ryan Stone
That gives you the top of the callstack at the time that the core was
define trace_stack
set $frame_ptr=$arg0
set $iters=0
while $frame_ptr != 0 && $iters < $arg1
set $ret_addr=((char*)$frame_ptr) + sizeof(void*)
printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr, *(void**)$ret_addr
printf " "
info line **(void***)$ret_addr
set $frame_ptr=*(void**)$frame_ptr
set $iters=$iters+1
end
end
trace_stack frame->tf_rbp 20
Thank you for this script.
Here is an example from my practice.
(kgdb) bt
#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291
#1 0xffffffff8063453f in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:359
#2 0xffffffff80634ba4 in vpanic (fmt=<value optimized out>, ap=<value optimized
out>) at /usr/src/sys/kern/kern_shutdown.c:635
#3 0xffffffff806348a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:568
#4 0xffffffff8041bba7 in db_panic (addr=<value optimized out>, have_addr=false,
count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473
#5 0xffffffff8041b67b in db_command (cmd_table=0x0) at
/usr/src/sys/ddb/db_command.c:440
#6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#7 0xffffffff8041de0b in db_trap (type=<value optimized out>, code=0) at
/usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0) at
/usr/src/sys/kern/subr_kdb.c:653
#9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at
/usr/src/sys/amd64/amd64/trap.c:381
#10 0xffffffff80809623 in nmi_calltrap () at
/usr/src/sys/libkern/explicit_bzero.c:28
This may be part of the problem. The trapframe unwinder depends on function names
to know when it is crossing a trapframe. nmi_calltrap() is not the function at
explicit_bzero.c:28. Usually debugging this sort of thing starts by going to frame 11
and comparing its registers with the values in the trapframe. They should match, but
sometimes you will find them shifted by one or two, etc.
--
John Baldwin
Andriy Gapon
2015-10-02 21:29:30 UTC
Permalink
Post by John Baldwin
Post by Andriy Gapon
Post by Ryan Stone
*Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to
I wonder, what is a reason for this?
Can that be fixed in kgdb itself?
It seems that usually kgdb handles trap frames just fine, but not always?
It should be fixable. If this doesn't work under newer kgdb let me know and I'll
try to fix it.
Okay, letting you know :-)
The backtraces from the in-tree kgdb and the newer kgdb both abort at the same
frame (output from the newer kgdb is in my message in another kgdb related thread).
Post by John Baldwin
I did fix a few edge cases with special frame handling in the
newer kgdb though those mostly had to do with fork_trampoline and possibly
Xtimerint (and aside from fork_trampoline I think the fixes were mostly for i386
where different handlers setup trapframes differently)
Post by Andriy Gapon
Post by Ryan Stone
That gives you the top of the callstack at the time that the core was
define trace_stack
set $frame_ptr=$arg0
set $iters=0
while $frame_ptr != 0 && $iters < $arg1
set $ret_addr=((char*)$frame_ptr) + sizeof(void*)
printf "frameptr=%p, ret_addr=%p\n", (void*)$frame_ptr, *(void**)$ret_addr
printf " "
info line **(void***)$ret_addr
set $frame_ptr=*(void**)$frame_ptr
set $iters=$iters+1
end
end
trace_stack frame->tf_rbp 20
Thank you for this script.
Here is an example from my practice.
(kgdb) bt
#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291
#1 0xffffffff8063453f in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:359
#2 0xffffffff80634ba4 in vpanic (fmt=<value optimized out>, ap=<value optimized
out>) at /usr/src/sys/kern/kern_shutdown.c:635
#3 0xffffffff806348a3 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:568
#4 0xffffffff8041bba7 in db_panic (addr=<value optimized out>, have_addr=false,
count=0, modif=0x0) at /usr/src/sys/ddb/db_command.c:473
#5 0xffffffff8041b67b in db_command (cmd_table=0x0) at
/usr/src/sys/ddb/db_command.c:440
#6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#7 0xffffffff8041de0b in db_trap (type=<value optimized out>, code=0) at
/usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0) at
/usr/src/sys/kern/subr_kdb.c:653
#9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0) at
/usr/src/sys/amd64/amd64/trap.c:381
#10 0xffffffff80809623 in nmi_calltrap () at
/usr/src/sys/libkern/explicit_bzero.c:28
This may be part of the problem. The trapframe unwinder depends on function names
to know when it is crossing a trapframe. nmi_calltrap() is not the function at
explicit_bzero.c:28. Usually debugging this sort of thing starts by going to frame 11
and comparing its registers with the values in the trapframe. They should match, but
sometimes you will find them shifted by one or two, etc.
And it seems that nmi_calltrap being a label within an assembler-defined
procedure confuses the in-tree kgdb quite a lot:

(kgdb) list *0xffffffff80809623
0xffffffff80809623 is at /usr/src/sys/libkern/explicit_bzero.c:28.
23 void
24 explicit_bzero(void *buf, size_t len)
25 {
26 memset(buf, 0, len);
27 __explicit_bzero_hook(buf, len);
28 }
(kgdb) list nmi_calltrap
23 void
24 explicit_bzero(void *buf, size_t len)
25 {
26 memset(buf, 0, len);
27 __explicit_bzero_hook(buf, len);
28 }
(kgdb) disassemble nmi_calltrap
Dump of assembler code for function nmi_calltrap:
0xffffffff8080961b <nmi_calltrap+0>: mov %rsp,%rdi
0xffffffff8080961e <nmi_calltrap+3>: callq 0xffffffff80820670 <trap>
0xffffffff80809623 <nmi_calltrap+8>: test %ebx,%ebx
0xffffffff80809625 <nmi_calltrap+10>: je 0xffffffff80809695 <nocallchain>
0xffffffff80809627 <nmi_calltrap+12>: mov %gs:0x0,%rax
0xffffffff80809630 <nmi_calltrap+21>: or %rax,%rax
0xffffffff80809633 <nmi_calltrap+24>: je 0xffffffff80809695 <nocallchain>
0xffffffff80809635 <nmi_calltrap+26>: testl $0x400000,0xec(%rax)
0xffffffff8080963f <nmi_calltrap+36>: je 0xffffffff80809695 <nocallchain>
0xffffffff80809641 <nmi_calltrap+38>: mov %rsp,%rsi
0xffffffff80809644 <nmi_calltrap+41>: mov $0xc0,%rcx
0xffffffff8080964b <nmi_calltrap+48>: mov %gs:0x220,%rdx
0xffffffff80809654 <nmi_calltrap+57>: sub %rcx,%rdx
0xffffffff80809657 <nmi_calltrap+60>: mov %rdx,%rdi
0xffffffff8080965a <nmi_calltrap+63>: shr $0x3,%rcx
0xffffffff8080965e <nmi_calltrap+67>: cld
0xffffffff8080965f <nmi_calltrap+68>: rep movsq %ds:(%rsi),%es:(%rdi)
0xffffffff80809662 <nmi_calltrap+71>: mov %ss,%eax
0xffffffff80809664 <nmi_calltrap+73>: push %rax
0xffffffff80809665 <nmi_calltrap+74>: push %rdx
0xffffffff80809666 <nmi_calltrap+75>: pushfq
0xffffffff80809667 <nmi_calltrap+76>: mov %cs,%eax
0xffffffff80809669 <nmi_calltrap+78>: push %rax
0xffffffff8080966a <nmi_calltrap+79>: pushq $0xffffffff80809671
0xffffffff8080966f <nmi_calltrap+84>: iretq
End of assembler dump.
(kgdb) disassemble explicit_bzero
Dump of assembler code for function explicit_bzero:
0xffffffff806e74c0 <explicit_bzero+0>: push %rbp
0xffffffff806e74c1 <explicit_bzero+1>: mov %rsp,%rbp
0xffffffff806e74c4 <explicit_bzero+4>: push %r14
0xffffffff806e74c6 <explicit_bzero+6>: push %rbx
0xffffffff806e74c7 <explicit_bzero+7>: mov %rsi,%r14
0xffffffff806e74ca <explicit_bzero+10>: mov %rdi,%rbx
0xffffffff806e74cd <explicit_bzero+13>: callq 0xffffffff806e74f0 <memset>
0xffffffff806e74d2 <explicit_bzero+18>: mov %rbx,%rdi
0xffffffff806e74d5 <explicit_bzero+21>: mov %r14,%rsi
0xffffffff806e74d8 <explicit_bzero+24>: callq 0xffffffff8088a2d0
<__explicit_bzero_hook>
0xffffffff806e74dd <explicit_bzero+29>: pop %rbx
0xffffffff806e74de <explicit_bzero+30>: pop %r14
0xffffffff806e74e0 <explicit_bzero+32>: pop %rbp
0xffffffff806e74e1 <explicit_bzero+33>: retq
End of assembler dump.


The newer kgdb is smarter about this situation:

(kgdb) list *0xffffffff80809623
0xffffffff80809623 is at /usr/src/sys/amd64/amd64/exception.S:527.
522 * - Check if the thread requires a user call chain to be
523 * captured.
524 *
525 * We are still in NMI mode at this point.
526 */
527 testl %ebx,%ebx
528 jz nocallchain /* not from userspace */
529 movq PCPU(CURTHREAD),%rax
530 orq %rax,%rax /* curthread present? */
531 jz nocallchain

However, that does not seem to help with stack unwinding.
--
Andriy Gapon
John Baldwin
2016-01-08 20:05:59 UTC
Permalink
Post by Andriy Gapon
Post by John Baldwin
Post by Andriy Gapon
Post by Ryan Stone
*Sigh*, kgdb isn't unwinding the trap frame properly. You can try this to
I wonder, what is a reason for this?
Can that be fixed in kgdb itself?
It seems that usually kgdb handles trap frames just fine, but not always?
It should be fixable. If this doesn't work under newer kgdb let me know and I'll
try to fix it.
Okay, letting you know :-)
The backtraces from the in-tree kgdb and the newer kgdb both abort at the same
frame (output from the newer kgdb is in my message in another kgdb related thread).
So I figured out why newer kgdb wasn't unwinding through NULL function pointer
traps yesterday. I'm not sure if the fix will help with your case as well,
but it might be worth trying. The changes are in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206044
--
John Baldwin
Andriy Gapon
2016-01-21 12:07:51 UTC
Permalink
Post by John Baldwin
So I figured out why newer kgdb wasn't unwinding through NULL function pointer
traps yesterday. I'm not sure if the fix will help with your case as well,
but it might be worth trying. The changes are in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206044
Thank you very much!
It does help in my case:

(kgdb) bt
#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291
#1 0xffffffff8063453f in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:359
#2 0xffffffff80634ba4 in vpanic (fmt=<optimized out>, ap=<optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:635
#3 0xffffffff806348a3 in panic (fmt=<unavailable>) at
/usr/src/sys/kern/kern_shutdown.c:568
#4 0xffffffff8041bba7 in db_panic (addr=<optimized out>,
have_addr=<unavailable>, count=<unavailable>, modif=<unavailable>) at
/usr/src/sys/ddb/db_command.c:473
#5 0xffffffff8041b67b in db_command (last_cmdp=<optimized out>, cmd_table=0x0,
dopager=<optimized out>) at /usr/src/sys/ddb/db_command.c:440
#6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#7 0xffffffff8041de0b in db_trap (type=<optimized out>, code=<unavailable>) at
/usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0
<nmi0_stack+3888>) at /usr/src/sys/kern/subr_kdb.c:653
#9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0 <nmi0_stack+3888>) at
/usr/src/sys/amd64/amd64/trap.c:381
#10 <signal handler called>
#11 0xffffffff80619e1f in __mtx_assert (c=<optimized out>, what=<optimized out>,
file=<optimized out>, line=<optimized out>) at /usr/src/sys/kern/kern_mutex.c:842
#12 0xffffffff807fef86 in vm_reserv_free_page (m=0xfffff80229e07cc0) at
/usr/src/sys/vm/vm_reserv.c:832
#13 0xffffffff807f2b96 in vm_page_free_toq (m=0xfffff80229e07cc0) at
/usr/src/sys/vm/vm_page.c:2432
#14 0xffffffff807f2e4d in vm_page_free (m=0xffffffff81129198
<vm_page_queue_free_mtx+24>) at /usr/src/sys/vm/vm_page.c:962
#15 0xffffffff821c28e2 in ttm_bo_vm_fault (vm_obj=0xfffff8021fcdeb00,
offset=5304320, prot=<optimized out>, mres=0xfffffe02b8357050) at
/usr/src/sys/modules/drm2/drm2/../../../dev/drm2/ttm/ttm_bo_vm.c:269
#16 0xffffffff807d4fd3 in dev_pager_getpages (object=0xfffff8021fcdeb00,
ma=0xfffffe02b8357050, count=1, reqpage=0) at /usr/src/sys/vm/device_pager.c:321
#17 0xffffffff807f9d67 in vm_pager_get_pages (object=0xfffff8021fcdeb00,
m=0xfffffe02b8357050, count=1, reqpage=0) at /usr/src/sys/vm/vm_pager.c:291
#18 0xffffffff807e0d84 in vm_fault_hold (map=0xfffff8001dd57000,
vaddr=34729947136, fault_type=2 '\002', fault_flags=0, m_hold=0x0) at
/usr/src/sys/vm/vm_fault.c:672
#19 0xffffffff807e05ee in vm_fault (map=0xfffff8001dd57000, vaddr=<optimized
out>, fault_type=2 '\002', fault_flags=0) at /usr/src/sys/vm/vm_fault.c:277
#20 0xffffffff80821342 in trap_pfault (frame=0xfffffe02b8357c00, usermode=1) at
/usr/src/sys/amd64/amd64/trap.c:734
#21 0xffffffff80820bda in trap (frame=0xfffffe02b8357c00) at
/usr/src/sys/amd64/amd64/trap.c:326
#22 0xffffffff8082154a in trap_check (frame=0xfffffe02b8357c00) at
/usr/src/sys/amd64/amd64/trap.c:628
#23 <signal handler called>
#24 0x00000008022b4046 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffe7c8

See http://article.gmane.org/gmane.os.freebsd.devel.hackers/56410 for comparison.
One small regression is that previously there was nmi_calltrap in the trace, now
it's just "<signal handler called>".
--
Andriy Gapon
John Baldwin
2016-01-21 17:18:54 UTC
Permalink
Post by Andriy Gapon
Post by John Baldwin
So I figured out why newer kgdb wasn't unwinding through NULL function pointer
traps yesterday. I'm not sure if the fix will help with your case as well,
but it might be worth trying. The changes are in
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=206044
Thank you very much!
(kgdb) bt
#0 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:291
#1 0xffffffff8063453f in kern_reboot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:359
#2 0xffffffff80634ba4 in vpanic (fmt=<optimized out>, ap=<optimized out>) at
/usr/src/sys/kern/kern_shutdown.c:635
#3 0xffffffff806348a3 in panic (fmt=<unavailable>) at
/usr/src/sys/kern/kern_shutdown.c:568
#4 0xffffffff8041bba7 in db_panic (addr=<optimized out>,
have_addr=<unavailable>, count=<unavailable>, modif=<unavailable>) at
/usr/src/sys/ddb/db_command.c:473
#5 0xffffffff8041b67b in db_command (last_cmdp=<optimized out>, cmd_table=0x0,
dopager=<optimized out>) at /usr/src/sys/ddb/db_command.c:440
#6 0xffffffff8041b524 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#7 0xffffffff8041de0b in db_trap (type=<optimized out>, code=<unavailable>) at
/usr/src/sys/ddb/db_main.c:251
#8 0xffffffff80669de8 in kdb_trap (type=19, code=0, tf=0xffffffff80f976d0
<nmi0_stack+3888>) at /usr/src/sys/kern/subr_kdb.c:653
#9 0xffffffff80820d26 in trap (frame=0xffffffff80f976d0 <nmi0_stack+3888>) at
/usr/src/sys/amd64/amd64/trap.c:381
#10 <signal handler called>
#11 0xffffffff80619e1f in __mtx_assert (c=<optimized out>, what=<optimized out>,
file=<optimized out>, line=<optimized out>) at /usr/src/sys/kern/kern_mutex.c:842
#12 0xffffffff807fef86 in vm_reserv_free_page (m=0xfffff80229e07cc0) at
/usr/src/sys/vm/vm_reserv.c:832
#13 0xffffffff807f2b96 in vm_page_free_toq (m=0xfffff80229e07cc0) at
/usr/src/sys/vm/vm_page.c:2432
#14 0xffffffff807f2e4d in vm_page_free (m=0xffffffff81129198
<vm_page_queue_free_mtx+24>) at /usr/src/sys/vm/vm_page.c:962
#15 0xffffffff821c28e2 in ttm_bo_vm_fault (vm_obj=0xfffff8021fcdeb00,
offset=5304320, prot=<optimized out>, mres=0xfffffe02b8357050) at
/usr/src/sys/modules/drm2/drm2/../../../dev/drm2/ttm/ttm_bo_vm.c:269
#16 0xffffffff807d4fd3 in dev_pager_getpages (object=0xfffff8021fcdeb00,
ma=0xfffffe02b8357050, count=1, reqpage=0) at /usr/src/sys/vm/device_pager.c:321
#17 0xffffffff807f9d67 in vm_pager_get_pages (object=0xfffff8021fcdeb00,
m=0xfffffe02b8357050, count=1, reqpage=0) at /usr/src/sys/vm/vm_pager.c:291
#18 0xffffffff807e0d84 in vm_fault_hold (map=0xfffff8001dd57000,
vaddr=34729947136, fault_type=2 '\002', fault_flags=0, m_hold=0x0) at
/usr/src/sys/vm/vm_fault.c:672
#19 0xffffffff807e05ee in vm_fault (map=0xfffff8001dd57000, vaddr=<optimized
out>, fault_type=2 '\002', fault_flags=0) at /usr/src/sys/vm/vm_fault.c:277
#20 0xffffffff80821342 in trap_pfault (frame=0xfffffe02b8357c00, usermode=1) at
/usr/src/sys/amd64/amd64/trap.c:734
#21 0xffffffff80820bda in trap (frame=0xfffffe02b8357c00) at
/usr/src/sys/amd64/amd64/trap.c:326
#22 0xffffffff8082154a in trap_check (frame=0xfffffe02b8357c00) at
/usr/src/sys/amd64/amd64/trap.c:628
#23 <signal handler called>
#24 0x00000008022b4046 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fffffffe7c8
See http://article.gmane.org/gmane.os.freebsd.devel.hackers/56410 for comparison.
One small regression is that previously there was nmi_calltrap in the trace, now
it's just "<signal handler called>".
Yeah, listing trapframes as that is fallout from the fix. I've thought about
adding a custom frame type so I can install a custom frame printer and output
things like the trap number the way ddb does, but that would be a far more
invasive change that might be harder to maintain going forward.

Glad that this helped though!
--
John Baldwin
Continue reading on narkive:
Loading...