There is a LOT of work in this commit but the TL;DR is that it takes
CEL processing from using 38% of the CPU instructions used by a call,
which is more than that used by the call processing itself, down to less
than 10% of the instructions.
So here's the deal... cdr_custom, cdr_sqlite3_custom, cel_custom
and cel_sqlite3_custom all shared one ugly trait...they all used
ast_str_substitute_variables() or pbx_substitute_variables_helper()
to resolve the dialplan functions used in their config files. Not only
are they both extraordinarily expensive, they both require a dummy
channel to be allocated and destroyed for each record written. For CDRs,
that's not too bad because we only write one CDR per call. For CELs however,
it's a disaster.
As far as source code goes, the modules basically all did the same thing.
Unfortunately, they did it badly. The config files simply contained long
opaque strings which were intepreted by ast_str_substitute_variables() or
pbx_substitute_variables_helper(), the very functions that ate all the
instructions. This meant introducing a new "advanced" config format much
like the advanced manager event filtering added to manager.conf in 2024.
Fortunately however, if the legacy config was recognizable, we were able to
parse it as an advanced config and gain the benefit. If not, then it
goes the legacy, and very expensive, route.
Given the commonality among the modules, instead of making the same
improvements to 4 modules then trying to maintain them over time, a single
module "res_cdrel_custom" was created that contains all of the common code.
A few bonuses became possible in the process...
* The cdr_custom and cel_custom modules now support JSON formatted output.
* The cdr_sqlite_custom and cel_sqlite3_custom modules no longer have
to share an Sqlite3 database.
Summary of changes:
A new module "res/res_cdrel_custom.c" has been created and the existing
cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom modules
are now just stubs that call the code in res_cdrel_custom.
res_cdrel_custom contains:
* A common configuration facility.
* Getters for both CDR and CEL fields that share the same abstraction.
* Formatters for all data types found in the ast_cdr and ast_event
structures that share the same abstraction.
* Common writers for the text file and database backends that, you guessed it,
share the same abstraction.
The result is that while there is certainly a net increase in the number of
lines in the code base, most of it is in the configuration handling at
load-time. The run-time instruction path length is now significanty shorter.
```
Scenario Instructions Latency
=====================================================
CEL pre changes 38.49% 37.51%
CEL Advanced 9.68% 6.06%
CEL Legacy (auto-conv to adv) 9.95% 6.13%
CEL Sqlite3 pre changes 39.41% 39.90%
CEL Sqlite3 Advanced 25.68% 24.24%
CEL Sqlite3 Legacy (auto conv) 25.88% 24.53%
CDR pre changes 4.79% 2.95%
CDR Advanced 0.79% 0.47%
CDR Legacy (auto conv to adv) 0.86% 0.51%
CDR Sqlite3 pre changes 4.47% 2.89%
CEL Sqlite3 Advanced 2.16% 1.29%
CEL Sqlite3 Legacy (auto conv) 2.19% 1.30%
```
Notes:
* We only write one CDR per call but every little bit helps.
* Sqlite3 still takes a fair amount of resources but the new config
makes a decent improvement.
* Legacy configs that we can't auto convert will still take the
"pre changes" path.
If you're interested in more implementation details, see the comments
at the top of the res_cdrel_custom.c file.
One minor fix to CEL is also included...Although TenantID was added to the
ast_event structure, it was always rendered as an empty string. It's now
properly rendered.
UserNote: Significant performance improvements have been made to the
cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom modules.
See the new sample config files for those modules to see how to benefit
from them.
A follow up to #893 that brings the same functionality to
cdr_custom. Also update the sample configuration files to note support
for absolute paths.
(cherry picked from commit 16a5adf02a)
If a filename starts with a '/' in cel_custom [mappings] assume it is
a absolute file path and not relative filename/path to
AST_LOG_DIR/cel_custom/
(cherry picked from commit 2ec3a0973e)
* acl (named_acl.c)
* cdr
* cel
* ccss
* dnsmgr
* dsp
* enum
* extconfig (config.c)
* features
* http
* indications
* logger
* manager
* plc
* sounds
* udptl
These modules are now loaded at appropriate time by the module loader.
Unlike loadable modules these use AST_MODULE_LOAD_FAILURE on error so
the module loader will abort startup on failure of these modules.
Some of these modules are still initialized or shutdown from outside the
module loader. logger.c is initialized very early and shutdown very
late, manager.c is initialized by the module loader but is shutdown by
the Asterisk core (too much uses it without holding references).
Change-Id: I371a9a45064f20026c492623ea8062d02a1ab97f
In all non-pbx modules, AST_MODULE_LOAD_FAILURE has been changed
to AST_MODULE_LOAD_DECLINE. This prevents asterisk from exiting
if a module can't be loaded. If the user wishes to retain the
FAILURE behavior for a specific module, they can use the "require"
or "preload-require" keyword in modules.conf.
A new API was added to logger: ast_is_logger_initialized(). This
allows asterisk.c/check_init() to print to the error log once the
logger subsystem is ready instead of just to stdout. If something
does fail before the logger is initialized, we now print to stderr
instead of stdout.
Change-Id: I5f4b50623d9b5a6cb7c5624a8c5c1274c13b2b25
ASTERISK_REGISTER_FILE no longer has any purpose so this commit removes
all traces of it.
Previously exported symbols removed:
* __ast_register_file
* __ast_unregister_file
* ast_complete_source_filename
This also removes the mtx_prof static variable that was declared when
MTX_PROFILE was enabled. This variable was only used in lock.c so it
is now initialized in that file only.
ASTERISK-26480 #close
Change-Id: I1074af07d71f9e159c48ef36631aa432c86f9966
The stringfields refactor to allow adding stringfields to the end of a
structure (f6f4cf459f) exposed some
incomplete cleanup code by some stringfield users.
The most noticeable leaker is the logging system where there is a leak for
every log message generated.
ASTERISK-26078 #close
Reported by: Etienne Lessard
Patches:
jira_asterisk_26078_v13.patch (license #5621) patch uploaded
by Richard Mudgett
Change-Id: If6a08b31336b492c3de6f9dfd07c447f8d5a8782
Git does not support the ability to replace a token with a version
string during check-in. While it does have support for replacing a
token on clone, this is somewhat sub-optimal: the token is replaced
with the object hash, which is not particularly easy for human
consumption. What's more, in practice, the source file version was often
not terribly useful. Generally, when triaging bugs, the overall version
of Asterisk is far more useful than an individual SVN version of a file. As a
result, this patch removes Asterisk's support for showing source file
versions.
Specifically, it does the following:
* Rename ASTERISK_FILE_VERSION macro to ASTERISK_REGISTER_FILE, and
remove passing the version in with the macro. Other facilities
than 'core show file version' make use of the file names, such as
setting a debug level only on a specific file. As such, the act of
registering source files with the Asterisk core still has use. The
macro rename now reflects the new macro purpose.
* main/asterisk:
- Refactor the file_version structure to reflect that it no longer
tracks a version field.
- Remove the "core show file version" CLI command. Without the file
version, it is no longer useful.
- Remove the ast_file_version_find function. The file version is no
longer tracked.
- Rename ast_register_file_version/ast_unregister_file_version to
ast_register_file/ast_unregister_file, respectively.
* main/manager: Remove value from the Version key of the ModuleCheck
Action. The actual key itself has not been removed, as doing so would
absolutely constitute a backwards incompatible change. However, since
the file version is no longer tracked, there is no need to attempt to
include it in the Version key.
* UPGRADE: Add notes for:
- Modification to the ModuleCheck AMI Action
- Removal of the "core show file version" CLI command
Change-Id: I6cf0ff280e1668bf4957dc21f32a5ff43444a40e
https://origsvn.digium.com/svn/asterisk/branches/10
................
r337974 | rmudgett | 2011-09-26 14:35:23 -0500 (Mon, 26 Sep 2011) | 37 lines
Merged revisions 337973 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.8
........
r337973 | rmudgett | 2011-09-26 14:30:39 -0500 (Mon, 26 Sep 2011) | 30 lines
Fix deadlock when using dummy channels.
Dummy channels created by ast_dummy_channel_alloc() should be destoyed by
ast_channel_unref(). Using ast_channel_release() needlessly grabs the
channel container lock and can cause a deadlock as a result.
* Analyzed use of ast_dummy_channel_alloc() and made use
ast_channel_unref() when done with the dummy channel. (Primary reason for
the reported deadlock.)
* Made app_dial.c:dial_exec_full() not call ast_call() holding any channel
locks. Chan_local could not perform deadlock avoidance correctly.
(Potential deadlock exposed by this issue. Secondary reason for the
reported deadlock since the held lock was part of the deadlock chain.)
* Fixed some uses of ast_dummy_channel_alloc() not checking the returned
channel pointer for failure.
* Fixed some potential chan=NULL pointer usage in func_odbc.c. Protected
by testing the bogus_chan value.
* Fixed needlessly clearing a 1024 char auto array when setting the first
char to zero is enough in manager.c:action_getvar().
(closes issue ASTERISK-18613)
Reported by: Thomas Arimont
Patches:
jira_asterisk_18613_v1.8.patch (license #5621) patch uploaded by rmudgett
Tested by: Thomas Arimont
........
................
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@337975 65c4cc65-6c06-0410-ace0-fbb531ad65f3
CEL is the new system for logging channel events. This was inspired after
facing many problems trying to represent what is possible to happen to a call
in Asterisk using CDR records. For more information on CEL, see the built in
HTML or PDF documentation generated from the files in doc/tex/.
Many thanks to Steve Murphy (murf) and Brian Degenhardt (bmd) for their hard
work developing this code. Also, thanks to Matt Nicholson (mnicholson) and
Sean Bright (seanbright) for their assistance in the final push to get this
code ready for Asterisk trunk.
Review: https://reviewboard.asterisk.org/r/239/
git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@203638 65c4cc65-6c06-0410-ace0-fbb531ad65f3
