When DEBUG_THREADS is defined, lock.h uses strerror(), which is defined
in the libc string.h file, to print warning messages. If the including
source file doesn't include string.h then strerror() won't be found and
and compile errors will be thrown. Since lock.h depends on this, string.h
is now included from there if DEBUG_THREADS is defined. This way, including
source files don't have to worry about it.
A few tweaks needed to be done to some existing header files to allow them to
be compiled when included from C++ source files.
logger.h had declarations for ast_register_verbose() and
ast_unregister_verbose() which caused C++ issues but those functions were
actually removed from logger.c many years ago so the declarations were just
removed from logger.h.
(cherry picked from commit e8c5c3380a)
Previously, DETECT_DEADLOCKS depended on DEBUG_THREADS.
Unfortunately, DEBUG_THREADS adds a lot of lock tracking overhead
to all of the lock lifecycle calls whereas DETECT_DEADLOCKS just
causes the lock calls to loop over trylock in 200us intervals until
the lock is obtained and spits out log messages if it takes more
than 5 seconds. From a code perspective, the only reason they were
tied together was for logging. So... The ifdefs in lock.c were
refactored to allow DETECT_DEADLOCKS to be enabled without
also enabling DEBUG_THREADS.
Resolves: #321
UserNote: You no longer need to select DEBUG_THREADS to use
DETECT_DEADLOCKS. This removes a significant amount of overhead
if you just want to detect possible deadlocks vs needing full
lock tracing.
(cherry picked from commit 7e2243f9e1)
__ast_mutex_logger used the variable `canlog` without accepting it as a
argument. Replace with internal macro `log_mutex_error` which takes
canlog as the first arguement. This will prevent confusion when working
with lock.c code, many of the function declare the canlog variable and
in some cases it previously appeared to be unused.
Change-Id: I83b372cb0654c5c18eadc512f65a57fa6c2e9853
Add a volatile flag to lock tracking structures so we only need to use
the global lock when first initializing tracking.
Additionally add support for DEBUG_THREADS_LOOSE_ABI. This is used by
astobj2.c to eliminate storage for tracking fields when DEBUG_THREADS is
not defined.
Change-Id: Iabd650908901843e9fff47ef1c539f0e1b8cb13b
Create ast_atomic macro's to provide a consistent interface to the
common functionality of __atomic and __sync built-in functions.
ASTERISK-27619
Change-Id: Ieba3f81832a0e25c5725ea067e5d6f742d33eb5b
This change causes the configure script to throw an error if neither
__sync nor __atomic builtin functions are available.
ASTERISK-27619
Change-Id: Ie01a281e0f5c41dfeeb5f250c1ccea8752f56ef9
Add a check to configure.ac for __atomic_fetch_add support. If found
use the __atomic built-in operators for ast_atomic_dec_and_test and
ast_atomic_fetchadd_int.
ASTERISK~27619
Change-Id: I65b4feb02bae368904ed0fb03f585c05f50a690e
This makes the 'bt' parameter unconditional for ast_store_lock_info and
ast_remove_lock_info. The 'bt' parameter is unused when HAVE_BKTR is
undefined.
Change-Id: Ieced0e920928b735a39c3b5952b806c473d67453
This allows asterisk to compiled with LOW_MEMORY to load modules built
without LOW_MEMORY.
ASTERISK-26398 #close
Change-Id: I24b78ac9493ab933b11087a8b6794f3c96d4872d
Users upgrading from asterisk 13.5 to a later version and who use
realtime with peers that have mailboxes were experiencing runaway
situations that manifested as a continuous stream of taskprocessor
congestion errors, memory leaks and an unresponsive chan_sip.
A related issue was that setting rtcachefriends=no NEVER worked in
asterisk 13 (since the move to stasis). In 13.5 and earlier, when a
peer tried to register, all of the stasis threads would block and
chan_sip would again become unresponsive. After 13.5, the runaway
would happen.
There were a number of causes...
* mwi_event_cb was (indirectly) calling build_peer even though calls to
mwi_event_cb are often caused by build_peer.
* In an effort to prevent chan_sip from being unloaded while messages
were still in flight, destroy_mailboxes was calling
stasis_unsubscribe_and_join but in some cases waited forever for the
final message.
* add_peer_mailboxes wasn't properly marking the existing mailboxes
on a peer as "keep" so build_peer would always delete them all.
* add_peer_mwi_subs was unsubscribing existing mailbox subscriptions
then just creating them again.
All of this was causing a flood of subscribes and unsubscribes on
multiple threads all for the same peer and mailbox.
Fixes...
* add_peer_mailboxes now marks mailboxes correctly and build_peer only
deletes the ones that really are no longer needed by the peer.
* add_peer_mwi_subs now only adds subscriptions marked as "new" instead
of unsubscribing and resubscribing everything. It also adds the peer
object's address to the mailbox instead of its name to the subscription
userdata so mwi_event_cb doesn't have to call build_peer.
With these changes, with rtcachefriends=yes (the most common setting),
there are no leaks, locks, loops or crashes at shutdown.
rtcachefriends=no still causes leaks but at least it doesn't lock, loop
or crash. Since making rtcachefriends=no work wasnt in scope for this
issue, further work will have to be deferred to a separate patch.
Side fixes...
* The ast_lock_track structure had a member named "thread" which gdb
doesn't like since it conflicts with it's "thread" command. That
member was renamed to "thread_id".
ASTERISK-25468 #close
Change-Id: I07519ef7f092629e1e844f855abd279d6475cdd0
This patch started with David Lee's patch at
https://reviewboard.asterisk.org/r/2826/ and includes a regression fix
introduced by the ASTERISK-22455 patch.
The initialization of a mutex's lock tracking structure was not protected
in a critical section. This is fine for any mutex that is explicitly
initialized, but a static mutex may have its lock tracking double
initialized if multiple threads attempt the first lock simultaneously.
* Added a global mutex to properly serialize initialization of the lock
tracking structure. The painful global lock can be mitigated by adding a
double checked lock flag as discussed on the original review request.
* Defer lock tracking initialization until first use.
* Don't be "helpful" and initialize an uninitialized lock when
DEBUG_THREADS is enabled. Debug code is not supposed to fix or change
normal code behavior. We don't need a lock initialization race that would
force a re-setup of lock tracking. Lock tracking already handles
initialization on first use.
* Properly handle allocation failures of the lock tracking structure.
* No need to initialize tracking data in __ast_pthread_mutex_destroy()
just to turn around and destroy it.
The regression introduced by ASTERISK-22455 is the result of manipulating
a pthread_mutex_t struct outside of the pthread library code. The
pthread_mutex_t struct seems to have a global linked list pointer member
that can get changed by other threads. Therefore, saving and restoring
the contents of a pthread_mutex_t struct is a bad thing.
Thanks to Thomas Airmont for finding this obscure regression.
* Don't overwrite the struct ast_lock_track.reentr_mutex member to restore
tracking data in __ast_cond_wait() and __ast_cond_timedwait(). The
pthread_mutex_t struct must be treated as a read-only opaque variable.
Miscellaneous other items fixed by this patch:
* Match ast_suspend_lock_info() with ast_restore_lock_info() in
__ast_cond_timedwait().
* Made some uninitialized lock sanity checks return EINVAL and try a
DO_THREAD_CRASH.
* Fix bad canlog initialization expressions.
ASTERISK-24614 #close
Reported by: Thomas Airmont
Review: https://reviewboard.asterisk.org/r/4247/
Review: https://reviewboard.asterisk.org/r/2826/
........
Merged revisions 429539 from http://svn.asterisk.org/svn/asterisk/branches/11
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@429541 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Made the vector macro API be more like linked lists.
1) Added a name parameter to ast_vector() to name the vector struct.
2) Made the API take a pointer to the vector struct instead of the struct
itself.
3) Added an element cleanup macro/function parameter when removing an
element from the vector for ast_vector_remove_cmp_unordered() and
ast_vector_remove_elem_unordered().
4) Added ast_vector_get_addr() in case the vector element is not a simple
pointer.
* Converted an inline vector usage in stasis_message_router to use the
vector API. It needed the API improvements so it could be converted.
* Fixed topic reference leak in router_dtor() when the
stasis_message_router is destroyed.
* Fixed deadlock potential in stasis_forward_all() and
stasis_forward_cancel(). Locking two topics at the same time requires
deadlock avoidance.
* Made internal_stasis_subscribe() tolerant of a NULL topic.
* Made stasis_message_router_add(),
stasis_message_router_add_cache_update(), stasis_message_router_remove(),
and stasis_message_router_remove_cache_update() tolerant of a NULL
message_type.
* Promoted a LOG_DEBUG message to LOG_ERROR as intended in
dispatch_message().
Review: https://reviewboard.asterisk.org/r/2903/
........
Merged revisions 402429 from http://svn.asterisk.org/svn/asterisk/branches/12
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@402430 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch fixes some long-standing bugs in debug threads that were
exacerbated with recent Optional API work in Asterisk 12.
With debug threads enabled, on some systems, there's a lock ordering
problem between our mutex and glibc's mutex protecting its module list
(Ubuntu Lucid, glibc 2.11.1 in this instance). In one thread, the module
list will be locked before acquiring our mutex. In another thread, our
mutex will be locked before locking the module list (which happens in
the depths of calling backtrace()).
This patch fixes this issue by moving backtrace() calls outside of
critical sections that have the mutex acquired. The bigger change was to
reentrancy tracking for ast_cond_{timed,}wait, which wrongly assumed
that waiting on the mutex was equivalent to a single unlock (it actually
suspends all recursive locks on the mutex).
(closes issue ASTERISK-22455)
Review: https://reviewboard.asterisk.org/r/2824/
........
Merged revisions 398648 from http://svn.asterisk.org/svn/asterisk/branches/1.8
........
Merged revisions 398649 from http://svn.asterisk.org/svn/asterisk/branches/11
........
Merged revisions 398651 from http://svn.asterisk.org/svn/asterisk/branches/12
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@398652 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Review https://reviewboard.asterisk.org/r/2580/ tried to fix the mismatch
in memory pools but had a math error determining the buffer size and
didn't address other similar memory pool mismatches.
* Effectively reverted the previous patch to go in the same direction as
trunk for the returned memory pool of ast_bt_get_symbols().
* Fixed memory leak in ast_bt_get_symbols() when BETTER_BACKTRACES is
defined.
* Fixed some formatting in ast_bt_get_symbols().
* Fixed sig_pri.c freeing memory allocated by libpri when MALLOC_DEBUG is
enabled.
* Fixed __dump_backtrace() freeing memory from ast_bt_get_symbols() when
MALLOC_DEBUG is enabled.
* Moved __dump_backtrace() because of compile issues with the utils
directory.
(closes issue ASTERISK-22221)
Reported by: Matt Jordan
Review: https://reviewboard.asterisk.org/r/2778/
........
Merged revisions 397525 from http://svn.asterisk.org/svn/asterisk/branches/1.8
........
Merged revisions 397528 from http://svn.asterisk.org/svn/asterisk/branches/11
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@397570 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch allows astmm to access the backtrace generation code in Asterisk.
When memory is allocated, a backtrace is created and stored with the memory
region that tracks the allocation. If a memory corruption is detected, the
backtrace is printed to the astmm log. The backtrace will make use of the
BETTER_BACKTRACES build option if available.
As a result, this patch moves the backtrace generation code into its own file
and uses the non-wrapped versions of the C library memory allocation routines.
This allows the memory allocation code to safely use the backtrace generation
routines without infinitely recursing.
Review: https://reviewboard.asterisk.org/r/2567
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@391012 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Similar to the patch that moved the fork earlier in the startup sequence to
prevent mutex errors in the recursive mutex surrounding the read/write thread
registration lock, this patch re-initializes the logmsgs mutex. Part of the
start up sequence before forking the process into the background includes
reading asterisk.conf; this has to occur prior to the call to daemon in order
to read startup parameters. When reading in a conf file, log statements can
be generated. Since this can't be avoided, the mutex instead is
re-initialized to ensure a reset of any thread tracking information.
This patch also includes some additional debugging to catch errors when
locking or unlocking the recursive mutex that surrounds locks when the
DEBUG_THREADS build option is enabled. DO_CRASH or THREAD_CRASH will
cause an abort() if a mutex error is detected.
(issue ASTERISK-19463)
Reported by: mjordan
Tesetd by: mjordan
........
Merged revisions 376586 from http://svn.asterisk.org/svn/asterisk/branches/1.8
........
Merged revisions 376587 from http://svn.asterisk.org/svn/asterisk/branches/10
........
Merged revisions 376588 from http://svn.asterisk.org/svn/asterisk/branches/11
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@376589 65c4cc65-6c06-0410-ace0-fbb531ad65f3
With the SCOPED_LOCK macro, you can create a variable
that locks a specific lock and unlocks the lock when the
variable goes out of scope. This is useful for situations
where many breaks, continues, returns, or other interruptions
would require separate unlock statements. With a scoped lock,
these aren't necessary.
There are specializations for mutexes, read locks, write locks,
ao2 locks, ao2 read locks, ao2 write locks, and channel locks.
Each of these is a SCOPED_LOCK at heart though.
Review: https://reviewboard.asterisk.org/r/2060
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@371582 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Add the ability to specify what kind of locking an ao2 object has when it
is allocated. The locking could be one of: MUTEX, RWLOCK, or none.
New API:
ao2_t_alloc_options()
ao2_alloc_options()
ao2_t_container_alloc_options()
ao2_container_alloc_options()
ao2_rdlock()
ao2_wrlock()
ao2_tryrdlock()
ao2_trywrlock()
The OBJ_NOLOCK and AO2_ITERATOR_DONTLOCK flags have a slight meaning
change. They no longer mean that the object is protected by an external
mechanism. They mean the lock associated with the object has already been
manually obtained by one of the ao2_lock calls. This change is necessary
for RWLOCK support since they are not reentrant. Also an operation on an
ao2 container may require promoting a read lock to a write lock by
releasing the already held read lock to re-acquire as a write lock.
Replaced API calls:
ao2_t_link_nolock()
ao2_link_nolock()
ao2_t_unlink_nolock()
ao2_unlink_nolock()
with the respective
ao2_t_link_flags()
ao2_link_flags()
ao2_t_unlink_flags()
ao2_unlink_flags()
API calls to be more flexible and to allow an anticipated enhancement to
control linking duplicate objects into a container.
The changes to format.c and format_cap.c are taking advantange of the new
ao2 locking options to simplify the use of the format capabilities
containers.
Review: https://reviewboard.asterisk.org/r/1554/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@357272 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.8
........
r304950 | tilghman | 2011-01-31 00:41:36 -0600 (Mon, 31 Jan 2011) | 18 lines
Change mutex tracking so that it only consumes memory in the core mutex object when it's actually being used.
This reduces the overall size of a mutex which was 3016 bytes before this back
down to 216 bytes (this is on 64-bit Linux with a glibc-implemented mutex).
The exactness of the numbers here may vary slightly based upon how mutexes are
implemented on a platform, but the long and short of it is that prior to this
commit, chan_iax2 held down 98MB of memory on a 64-bit system for nothing more
than a table of 32767 locks. After this commit, the same table occupies a mere
7MB of memory.
(closes issue #18194)
Reported by: job
Patches:
20110124__issue18194.diff.txt uploaded by tilghman (license 14)
Tested by: tilghman
Review: https://reviewboard.asterisk.org/r/1066
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@304951 65c4cc65-6c06-0410-ace0-fbb531ad65f3
"Bad Things" would happen if Asterisk was compiled with DEBUG_THREADS, but a
loaded module was not (or vice versa). This also immensely simplifies the
lock code, since there are no longer 2 separate versions of them.
Review: https://reviewboard.asterisk.org/r/508/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@258557 65c4cc65-6c06-0410-ace0-fbb531ad65f3
This patch finishes the implementation of OBJ_MULTIPLE in astobj2 (the
case where multiple results need to be returned; OBJ_NODATA mode
already was supported). In addition, it converts ast_channel_iterators
(only the targeted versions, not the ones that iterate over all
channels) to use this method.
During this work, I removed the 'ao2_flags' arguments to the
ast_channel_iterator constructor functions; there were no uses of that
argument yet, there is only one possible flag to pass, and it made the
iterators less 'opaque'. If at some point in the future someone really
needs an ast_channel_iterator that does not lock the container, we can
provide constructor(s) for that purpose.
Review: https://reviewboard.asterisk.org/r/379/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@225244 65c4cc65-6c06-0410-ace0-fbb531ad65f3
There is a lot that could be said about this, but the patch is a big
improvement for performance, stability, code maintainability,
and ease of future code development.
The channel list is no longer an unsorted linked list. The main container
for channels is an astobj2 hash table. All of the code related to searching
for channels or iterating active channels has been rewritten. Let n be
the number of active channels. Iterating the channel list has gone from
O(n^2) to O(n). Searching for a channel by name went from O(n) to O(1).
Searching for a channel by extension is still O(n), but uses a new method
for doing so, which is more efficient.
The ast_channel object is now a reference counted object. The benefits
here are plentiful. Some benefits directly related to issues in the
previous code include:
1) When threads other than the channel thread owning a channel wanted
access to a channel, it had to hold the lock on it to ensure that it didn't
go away. This is no longer a requirement. Holding a reference is
sufficient.
2) There are places that now require less dealing with channel locks.
3) There are places where channel locks are held for much shorter periods
of time.
4) There are places where dealing with more than one channel at a time becomes
_MUCH_ easier. ChanSpy is a great example of this. Writing code in the
future that deals with multiple channels will be much easier.
Some additional information regarding channel locking and reference count
handling can be found in channel.h, where a new section has been added that
discusses some of the rules associated with it.
Mark Michelson also assisted with the development of this patch. He did the
conversion of ChanSpy and introduced a new API, ast_autochan, which makes it
much easier to deal with holding on to a channel pointer for an extended period
of time and having it get automatically updated if the channel gets masqueraded.
Mark was also a huge help in the code review process.
Thanks to David Vossel for his assistance with this branch, as well. David
did the conversion of the DAHDIScan application by making it become a wrapper
for ChanSpy internally.
The changes come from the svn/asterisk/team/russell/ast_channel_ao2 branch.
Review: http://reviewboard.digium.com/r/203/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@190423 65c4cc65-6c06-0410-ace0-fbb531ad65f3
When dvossel and I were doing some load testing last week, we noticed that we
could make Asterisk trunk lock up instantly when we started generating a bunch
of calls. The backtraces of locked threads were bizarre, and many were stuck
on an _unlock_ of an rwlock.
The changes are:
1) Fix a number of places where a backtrace would be loaded into an invalid
index of the backtrace array. It's an off by one error, which ends up
writing over the rwlock itself.
2) Ensure that in the array of held locks, we NULL out an index once it is
not being used so that it's not confusing when analyzing its contents.
3) Remove a bunch of logging referring to an rwlock operating being done
with "deep reentrancy". It is normal for _many_ threads to hold a
read lock on an rwlock.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@184531 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Copied from my review board description:
This is a continuation of the API changes documentation started for describing
changes between releases. Most of the API changes were pretty simple needing
only to be brought to attention via the new "Asterisk API Changes" list.
However, if you see anything that needs further explanation feel free to
supplement what is there. The current method of documenting is to add (in the
header file): \version <ver number> <description of changes> and then to add
the function to the change list in doxyref.h on the AstAPIChanges page. I also
made sure all the functions that were newly added were tagged with \since
1.6.1. I think this is a good habit to start both for the historical aspect as
well as for the future ability to easily add a "New Asterisk API" page.
Review: http://reviewboard.digium.com/r/190/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@180719 65c4cc65-6c06-0410-ace0-fbb531ad65f3
------------------------------------------------------------------------
r159808 | kpfleming | 2008-11-29 10:58:29 -0600 (Sat, 29 Nov 2008) | 7 lines
update dev-mode compiler flags to match the ones used by default on Ubuntu Intrepid, so all developers will see the same warnings and errors
since this branch already had some printf format attributes, enable checking for them and tag functions that didn't have them
format attributes in a consistent way
------------------------------------------------------------------------
in addition:
move some format attributes from main/utils.c to the header files they belong in, and fix up references to the relevant functions based on new compiler warnings
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@159818 65c4cc65-6c06-0410-ace0-fbb531ad65f3
Reported by: murf
Most of this bug was already fixed by Tilghman before
I opened it; Many thanks to Tilghman for his fix
in svn version 125794. That fix cleared up some of the
fields in the lock_info.
This commit changes the address that is stored for the
lock in the lock_info struct, so that it is the same
as that passed into the locking macros. This makes
searching for a lock_info (as in log_show_lock())
by its lock addr possible. The lock_addr field is
infinitely more useful if it is the same as what
is 'publicly' available outside the lock_info code.
Many thanks to kpfleming, putnopvut, and Russell for their
invaluable insights earlier today.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@131570 65c4cc65-6c06-0410-ace0-fbb531ad65f3
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r126573 | russell | 2008-06-30 11:05:08 -0500 (Mon, 30 Jun 2008) | 10 lines
Fix a typo in the non-DEBUG_THREADS version of the recently added DEADLOCK_AVOIDANCE()
macro. This caused the lock to not actually be released, and as a result, not
avoid deadlocks at all. This resolves the issues reported in the last while about
Asterisk locking up all over the place (and most commonly, in chan_iax2).
(closes issue #12927)
(closes issue #12940)
(closes issue #12925)
(potentially closes others ...)
........
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@126574 65c4cc65-6c06-0410-ace0-fbb531ad65f3
account that multiple threads could hold the same rdlock at the same time.
As such, it expected that when a thread released a lock that it must have
been the last to acquire the lock as well. Erroneous error messages would
be sent to the console stating that a thread was attempting to unlock a lock
it did not own.
Now all threads are examined to be sure that the message is only printed
when it is supposed to be printed.
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@125133 65c4cc65-6c06-0410-ace0-fbb531ad65f3
