hold tracking information for mutexes. Now, the "core show locks" output
will output information about who is holding a rwlock when a thread is
waiting on it.
(closes issue #11279)
Reported by: ys
Patches:
trunk_lock_utils.v8.diff uploaded by ys (license 281)
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@120064 65c4cc65-6c06-0410-ace0-fbb531ad65f3
If a deadlock is detected, then the typical lock information will be
printed along with a backtrace of the stack for the offending threads.
Use of this requires compiling with DETECT_DEADLOCKS and having glibc
installed.
Furthermore, issuing the "core show locks" CLI command will print the
normal lock information as well as a backtraces for each lock. This
requires that DEBUG_THREADS is enabled and that glibc is installed.
All the backtrace features may be disabled by running the configure
script with --without-execinfo as an argument
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@118173 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r116088 | mmichelson | 2008-05-13 18:47:49 -0500 (Tue, 13 May 2008) | 12 lines
A change to the way channel locks are handled when DEBUG_CHANNEL_LOCKS is defined.
After debugging a deadlock, it was noticed that when DEBUG_CHANNEL_LOCKS
is enabled in menuselect, the actual origin of channel locks is obscured
by the fact that all channel locks appear to happen in the function
ast_channel_lock(). This code change redefines ast_channel_lock to be a
macro which maps to __ast_channel_lock(), which then relays the proper
file name, line number, and function name information to the core lock
functions so that this information will be displayed in the case that
there is some sort of locking error or core show locks is issued.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@116089 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r105116 | russell | 2008-02-28 16:23:05 -0600 (Thu, 28 Feb 2008) | 8 lines
Fix a bug in the lock tracking code that was discovered by mmichelson. The issue
is that if the lock history array was full, then the functions to mark a lock as
acquired or not would adjust the stats for whatever lock is at the end of the array,
which may not be itself. So, do a sanity check to make sure that we're updating
lock info for the proper lock.
(This explains the bizarre stats on lock #63 in BE-396, thanks Mark!)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@105144 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r92875 | mmichelson | 2007-12-13 19:24:06 -0600 (Thu, 13 Dec 2007) | 7 lines
When compiling with DETECT_DEADLOCKS, don't spam the CLI with messages
about possible deadlocks. Instead just print the intended single message every
five seconds.
(closes issue 11537, reported and patched by dimas)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@92876 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r91828 | russell | 2007-12-07 15:17:24 -0600 (Fri, 07 Dec 2007) | 6 lines
Fix another bug in the DEBUG_THREADS code. The ast_mutex_init() function had
the mutex attribute object marked as static. This means that multiple threads
initializing locks at the same time could step on each other and end up with
improperly initialized locks.
(found when tracking down locking issues related to issue #11080)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@91829 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r91826 | russell | 2007-12-07 15:11:08 -0600 (Fri, 07 Dec 2007) | 6 lines
I love fixing lock related errors in the lock debugging code. That's about as
ironic as it gets in Asterisk programming land. Anyway, I spotted this bug while
trying to track down why systems are locking up and acting weird in issue #11080.
The mutex attribute object was marked as static in this function when it should
not have been.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@91827 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r91070 | russell | 2007-12-04 18:35:31 -0600 (Tue, 04 Dec 2007) | 11 lines
Fix some crashes in chan_iax2 that were reported as happening on Mac systems.
It turns out that the problem was the Mac version of the ast_atomic_fetchadd_int()
function. The Mac atomic add function returns the _new_ value, while this function
is supposed to return the old value. So, the crashes happened on unreferencing
objects. If the reference count was decreased to 1, ao2_ref() thought that it
had been decreased to zero, and called the destructor. However, there was still
an outstanding reference around.
(closes issue #11176)
(closes issue #11289)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@91114 65c4cc65-6c06-0410-ace0-fbb531ad65f3
to compat.h so it is always available - hopefully this will let
us reduce the number of inclusions of channel.h and frame.h
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@89426 65c4cc65-6c06-0410-ace0-fbb531ad65f3
comments on the content of these two files.
utils.h (which is included in over 150 files) contains a lot of
unrelated functions which require the inclusion of a large number
of other headers. At some point we should partition its content
in a better way.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@89341 65c4cc65-6c06-0410-ace0-fbb531ad65f3
build times - tested, there is no measureable difference before and
after this commit.
In this change:
use asterisk/compat.h to include a small set of system headers:
inttypes.h, unistd.h, stddef.h, stddint.h, sys/types.h, stdarg.h,
stdlib.h, alloca.h, stdio.h
Where available, the inclusion is conditional on HAVE_FOO_H as determined
by autoconf.
Normally, source files should not include any of the above system headers,
and instead use either "asterisk.h" or "asterisk/compat.h" which does it
better.
For the time being I have left alone second-level directories
(main/db1-ast, etc.).
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@89333 65c4cc65-6c06-0410-ace0-fbb531ad65f3
through ast_mutex primitives.
To detect all occurrences, I have renamed the lock field in struct ast_channel
so it is clear that it shouldn't be used directly.
There are some uses in res/res_features.c (see details of the diff)
that are error prone as they try and lock two channels without
caring about the order (or without explaining why it is safe).
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@89293 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r88931 | russell | 2007-11-06 07:50:15 -0600 (Tue, 06 Nov 2007) | 8 lines
Remove some checks to see if locks are initialized from the non-DEBUG_THREADS
versions of the lock routines. These are incorrect for a number of reasons:
- It breaks the build on mac.
- If there is a problem with locks not getting initialized, then the proper
fix is to find that place and fix the code so that it does get initialized.
- If additional debug code is needed to help find the problem areas, then this
type of things should _only_ be put in the DEBUG_THREADS wrappers.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@88932 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r86836 | russell | 2007-10-22 16:36:12 -0500 (Mon, 22 Oct 2007) | 9 lines
If lock tracking is not enabled, then we can not attempt to log any mutex
failures. If so, we could end up in infinite recursion. The only lock that
is affected by this is a mutex in astmm.c used when MALLOC_DEBUG is enabled.
(closes issue #11044)
Reported by: ys
Patches:
lock.h.diff uploaded by ys (license 281)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@86839 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r85994 | russell | 2007-10-16 17:14:36 -0500 (Tue, 16 Oct 2007) | 16 lines
Some locking errors exposed the fact that the lock debugging code itself was
not thread safe. How ironic! Anyway, these changes ensure that the code that
is accessing the lock debugging data is thread-safe.
Many thanks to Ivan for finding and fixing the core issue here, and also
thanks to those that tested the patch and provided test results.
(closes issue #10571)
(closes issue #10886)
(closes issue #10875)
(might close some others, as well ...)
Patches: (from issue #10571)
ivan_ast_1_4_12_rel_patch_lock.h.diff uploaded by Ivan (license 229)
- a few small changes by me
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@85995 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r78995 | russell | 2007-08-10 10:20:09 -0500 (Fri, 10 Aug 2007) | 4 lines
The last set of changes that I made to "core show locks" made it not able to
track mutexes unless they were declared using AST_MUTEX_DEFINE_STATIC. Locks
initialized with ast_mutex_init() were not tracked. It should work now.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@79005 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r78095 | russell | 2007-08-03 14:39:49 -0500 (Fri, 03 Aug 2007) | 28 lines
Add some improvements to lock debugging. These changes take effect
with DEBUG_THREADS enabled and provide the following:
* This will keep track of which locks are held by which thread as well as
which lock a thread is waiting for in a thread-local data structure. A
reference to this structure is available on the stack in the dummy_start()
function, which is the common entry point for all threads. This information
can be easily retrieved using gdb if you switch to the dummy_start() stack
frame of any thread and print the contents of the lock_info variable.
* All of the thread-local structures for keeping track of this lock information
are also stored in a list so that the information can be dumped to the CLI
using the "core show locks" CLI command. This introduces a little bit of a
performance hit as it requires additional underlying locking operations
inside of every lock/unlock on an ast_mutex. However, the benefits of
having this information available at the CLI is huge, especially considering
this is only done in DEBUG_THREADS mode. It means that in most cases where
we debug deadlocks, we no longer have to request access to the machine to
analyze the contents of ast_mutex_t structures. We can now just ask them
to get the output of "core show locks", which gives us all of the information
we needed in most cases.
I also had to make some additional changes to astmm.c to make this work when
both MALLOC_DEBUG and DEBUG_THREADS are enabled. I disabled tracking of one
of the locks in astmm.c because it gets used inside the replacement memory
allocation routines, and the lock tracking code allocates memory. This caused
infinite recursion.
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@78096 65c4cc65-6c06-0410-ace0-fbb531ad65f3
