From: Linus Torvalds
|
From: Linus Torvalds
|
|
|
How to track down an Oops.. [originally a mail to linux-kernel]
|
How to track down an Oops.. [originally a mail to linux-kernel]
|
|
|
The main trick is having 5 years of experience with those pesky oops
|
The main trick is having 5 years of experience with those pesky oops
|
messages ;-)
|
messages ;-)
|
|
|
Actually, there are things you can do that make this easier. I have two
|
Actually, there are things you can do that make this easier. I have two
|
separate approaches:
|
separate approaches:
|
|
|
gdb /usr/src/linux/vmlinux
|
gdb /usr/src/linux/vmlinux
|
gdb> disassemble
|
gdb> disassemble
|
|
|
That's the easy way to find the problem, at least if the bug-report is
|
That's the easy way to find the problem, at least if the bug-report is
|
well made (like this one was - run through ksymoops to get the
|
well made (like this one was - run through ksymoops to get the
|
information of which function and the offset in the function that it
|
information of which function and the offset in the function that it
|
happened in).
|
happened in).
|
|
|
Oh, it helps if the report happens on a kernel that is compiled with the
|
Oh, it helps if the report happens on a kernel that is compiled with the
|
same compiler and similar setups.
|
same compiler and similar setups.
|
|
|
The other thing to do is disassemble the "Code:" part of the bug report:
|
The other thing to do is disassemble the "Code:" part of the bug report:
|
ksymoops will do this too with the correct tools (and new version of
|
ksymoops will do this too with the correct tools (and new version of
|
ksymoops), but if you don't have the tools you can just do a silly
|
ksymoops), but if you don't have the tools you can just do a silly
|
program:
|
program:
|
|
|
char str[] = "\xXX\xXX\xXX...";
|
char str[] = "\xXX\xXX\xXX...";
|
main(){}
|
main(){}
|
|
|
and compile it with gcc -g and then do "disassemble str" (where the "XX"
|
and compile it with gcc -g and then do "disassemble str" (where the "XX"
|
stuff are the values reported by the Oops - you can just cut-and-paste
|
stuff are the values reported by the Oops - you can just cut-and-paste
|
and do a replace of spaces to "\x" - that's what I do, as I'm too lazy
|
and do a replace of spaces to "\x" - that's what I do, as I'm too lazy
|
to write a program to automate this all).
|
to write a program to automate this all).
|
|
|
Finally, if you want to see where the code comes from, you can do
|
Finally, if you want to see where the code comes from, you can do
|
|
|
cd /usr/src/linux
|
cd /usr/src/linux
|
make fs/buffer.s # or whatever file the bug happened in
|
make fs/buffer.s # or whatever file the bug happened in
|
|
|
and then you get a better idea of what happens than with the gdb
|
and then you get a better idea of what happens than with the gdb
|
disassembly.
|
disassembly.
|
|
|
Now, the trick is just then to combine all the data you have: the C
|
Now, the trick is just then to combine all the data you have: the C
|
sources (and general knowledge of what it _should_ do, the assembly
|
sources (and general knowledge of what it _should_ do, the assembly
|
listing and the code disassembly (and additionally the register dump you
|
listing and the code disassembly (and additionally the register dump you
|
also get from the "oops" message - that can be useful to see _what_ the
|
also get from the "oops" message - that can be useful to see _what_ the
|
corrupted pointers were, and when you have the assembler listing you can
|
corrupted pointers were, and when you have the assembler listing you can
|
also match the other registers to whatever C expressions they were used
|
also match the other registers to whatever C expressions they were used
|
for).
|
for).
|
|
|
Essentially, you just look at what doesn't match (in this case it was the
|
Essentially, you just look at what doesn't match (in this case it was the
|
"Code" disassembly that didn't match with what the compiler generated).
|
"Code" disassembly that didn't match with what the compiler generated).
|
Then you need to find out _why_ they don't match. Often it's simple - you
|
Then you need to find out _why_ they don't match. Often it's simple - you
|
see that the code uses a NULL pointer and then you look at the code and
|
see that the code uses a NULL pointer and then you look at the code and
|
wonder how the NULL pointer got there, and if it's a valid thing to do
|
wonder how the NULL pointer got there, and if it's a valid thing to do
|
you just check against it..
|
you just check against it..
|
|
|
Now, if somebody gets the idea that this is time-consuming and requires
|
Now, if somebody gets the idea that this is time-consuming and requires
|
some small amount of concentration, you're right. Which is why I will
|
some small amount of concentration, you're right. Which is why I will
|
mostly just ignore any panic reports that don't have the symbol table
|
mostly just ignore any panic reports that don't have the symbol table
|
info etc looked up: it simply gets too hard to look it up (I have some
|
info etc looked up: it simply gets too hard to look it up (I have some
|
programs to search for specific patterns in the kernel code segment, and
|
programs to search for specific patterns in the kernel code segment, and
|
sometimes I have been able to look up those kinds of panics too, but
|
sometimes I have been able to look up those kinds of panics too, but
|
that really requires pretty good knowledge of the kernel just to be able
|
that really requires pretty good knowledge of the kernel just to be able
|
to pick out the right sequences etc..)
|
to pick out the right sequences etc..)
|
|
|
_Sometimes_ it happens that I just see the disassembled code sequence
|
_Sometimes_ it happens that I just see the disassembled code sequence
|
from the panic, and I know immediately where it's coming from. That's when
|
from the panic, and I know immediately where it's coming from. That's when
|
I get worried that I've been doing this for too long ;-)
|
I get worried that I've been doing this for too long ;-)
|
|
|
Linus
|
Linus
|
|
|
|
|
---------------------------------------------------------------------------
|
---------------------------------------------------------------------------
|
Notes on Oops tracing with klogd:
|
Notes on Oops tracing with klogd:
|
|
|
In order to help Linus and the other kernel developers there has been
|
In order to help Linus and the other kernel developers there has been
|
substantial support incorporated into klogd for processing protection
|
substantial support incorporated into klogd for processing protection
|
faults. In order to have full support for address resolution at least
|
faults. In order to have full support for address resolution at least
|
version 1.3-pl3 of the sysklogd package should be used.
|
version 1.3-pl3 of the sysklogd package should be used.
|
|
|
When a protection fault occurs the klogd daemon automatically
|
When a protection fault occurs the klogd daemon automatically
|
translates important addresses in the kernel log messages to their
|
translates important addresses in the kernel log messages to their
|
symbolic equivalents. This translated kernel message is then
|
symbolic equivalents. This translated kernel message is then
|
forwarded through whatever reporting mechanism klogd is using. The
|
forwarded through whatever reporting mechanism klogd is using. The
|
protection fault message can be simply cut out of the message files
|
protection fault message can be simply cut out of the message files
|
and forwarded to the kernel developers.
|
and forwarded to the kernel developers.
|
|
|
Two types of address resolution are performed by klogd. The first is
|
Two types of address resolution are performed by klogd. The first is
|
static translation and the second is dynamic translation. Static
|
static translation and the second is dynamic translation. Static
|
translation uses the System.map file in much the same manner that
|
translation uses the System.map file in much the same manner that
|
ksymoops does. In order to do static translation the klogd daemon
|
ksymoops does. In order to do static translation the klogd daemon
|
must be able to find a system map file at daemon initialization time.
|
must be able to find a system map file at daemon initialization time.
|
See the klogd man page for information on how klogd searches for map
|
See the klogd man page for information on how klogd searches for map
|
files.
|
files.
|
|
|
Dynamic address translation is important when kernel loadable modules
|
Dynamic address translation is important when kernel loadable modules
|
are being used. Since memory for kernel modules is allocated from the
|
are being used. Since memory for kernel modules is allocated from the
|
kernel's dynamic memory pools there are no fixed locations for either
|
kernel's dynamic memory pools there are no fixed locations for either
|
the start of the module or for functions and symbols in the module.
|
the start of the module or for functions and symbols in the module.
|
|
|
The kernel supports system calls which allow a program to determine
|
The kernel supports system calls which allow a program to determine
|
which modules are loaded and their location in memory. Using these
|
which modules are loaded and their location in memory. Using these
|
system calls the klogd daemon builds a symbol table which can be used
|
system calls the klogd daemon builds a symbol table which can be used
|
to debug a protection fault which occurs in a loadable kernel module.
|
to debug a protection fault which occurs in a loadable kernel module.
|
|
|
At the very minimum klogd will provide the name of the module which
|
At the very minimum klogd will provide the name of the module which
|
generated the protection fault. There may be additional symbolic
|
generated the protection fault. There may be additional symbolic
|
information available if the developer of the loadable module chose to
|
information available if the developer of the loadable module chose to
|
export symbol information from the module.
|
export symbol information from the module.
|
|
|
Since the kernel module environment can be dynamic there must be a
|
Since the kernel module environment can be dynamic there must be a
|
mechanism for notifying the klogd daemon when a change in module
|
mechanism for notifying the klogd daemon when a change in module
|
environment occurs. There are command line options available which
|
environment occurs. There are command line options available which
|
allow klogd to signal the currently executing daemon that symbol
|
allow klogd to signal the currently executing daemon that symbol
|
information should be refreshed. See the klogd manual page for more
|
information should be refreshed. See the klogd manual page for more
|
information.
|
information.
|
|
|
A patch is included with the sysklogd distribution which modifies the
|
A patch is included with the sysklogd distribution which modifies the
|
modules-2.0.0 package to automatically signal klogd whenever a module
|
modules-2.0.0 package to automatically signal klogd whenever a module
|
is loaded or unloaded. Applying this patch provides essentially
|
is loaded or unloaded. Applying this patch provides essentially
|
seamless support for debugging protection faults which occur with
|
seamless support for debugging protection faults which occur with
|
kernel loadable modules.
|
kernel loadable modules.
|
|
|
The following is an example of a protection fault in a loadable module
|
The following is an example of a protection fault in a loadable module
|
processed by klogd:
|
processed by klogd:
|
---------------------------------------------------------------------------
|
---------------------------------------------------------------------------
|
Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
|
Aug 29 09:51:01 blizard kernel: Unable to handle kernel paging request at virtual address f15e97cc
|
Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
|
Aug 29 09:51:01 blizard kernel: current->tss.cr3 = 0062d000, %cr3 = 0062d000
|
Aug 29 09:51:01 blizard kernel: *pde = 00000000
|
Aug 29 09:51:01 blizard kernel: *pde = 00000000
|
Aug 29 09:51:01 blizard kernel: Oops: 0002
|
Aug 29 09:51:01 blizard kernel: Oops: 0002
|
Aug 29 09:51:01 blizard kernel: CPU: 0
|
Aug 29 09:51:01 blizard kernel: CPU: 0
|
Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868]
|
Aug 29 09:51:01 blizard kernel: EIP: 0010:[oops:_oops+16/3868]
|
Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
|
Aug 29 09:51:01 blizard kernel: EFLAGS: 00010212
|
Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c
|
Aug 29 09:51:01 blizard kernel: eax: 315e97cc ebx: 003a6f80 ecx: 001be77b edx: 00237c0c
|
Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c
|
Aug 29 09:51:01 blizard kernel: esi: 00000000 edi: bffffdb3 ebp: 00589f90 esp: 00589f8c
|
Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
|
Aug 29 09:51:01 blizard kernel: ds: 0018 es: 0018 fs: 002b gs: 002b ss: 0018
|
Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
|
Aug 29 09:51:01 blizard kernel: Process oops_test (pid: 3374, process nr: 21, stackpage=00589000)
|
Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
|
Aug 29 09:51:01 blizard kernel: Stack: 315e97cc 00589f98 0100b0b4 bffffed4 0012e38e 00240c64 003a6f80 00000001
|
Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
|
Aug 29 09:51:01 blizard kernel: 00000000 00237810 bfffff00 0010a7fa 00000003 00000001 00000000 bfffff00
|
Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
|
Aug 29 09:51:01 blizard kernel: bffffdb3 bffffed4 ffffffda 0000002b 0007002b 0000002b 0000002b 00000036
|
Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
|
Aug 29 09:51:01 blizard kernel: Call Trace: [oops:_oops_ioctl+48/80] [_sys_ioctl+254/272] [_system_call+82/128]
|
Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3
|
Aug 29 09:51:01 blizard kernel: Code: c7 00 05 00 00 00 eb 08 90 90 90 90 90 90 90 90 89 ec 5d c3
|
---------------------------------------------------------------------------
|
---------------------------------------------------------------------------
|
|
|
Dr. G.W. Wettstein Oncology Research Div. Computing Facility
|
Dr. G.W. Wettstein Oncology Research Div. Computing Facility
|
Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com
|
Roger Maris Cancer Center INTERNET: greg@wind.rmcc.com
|
820 4th St. N.
|
820 4th St. N.
|
Fargo, ND 58122
|
Fargo, ND 58122
|
Phone: 701-234-7556
|
Phone: 701-234-7556
|
|
|
