mirror of
https://github.com/asterisk/asterisk.git
synced 2026-04-28 01:32:57 +00:00
9c23df049d
There is a LOT of work in this commit but the TL;DR is that it takes CEL processing from using 38% of the CPU instructions used by a call, which is more than that used by the call processing itself, down to less than 10% of the instructions. So here's the deal... cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom all shared one ugly trait...they all used ast_str_substitute_variables() or pbx_substitute_variables_helper() to resolve the dialplan functions used in their config files. Not only are they both extraordinarily expensive, they both require a dummy channel to be allocated and destroyed for each record written. For CDRs, that's not too bad because we only write one CDR per call. For CELs however, it's a disaster. As far as source code goes, the modules basically all did the same thing. Unfortunately, they did it badly. The config files simply contained long opaque strings which were intepreted by ast_str_substitute_variables() or pbx_substitute_variables_helper(), the very functions that ate all the instructions. This meant introducing a new "advanced" config format much like the advanced manager event filtering added to manager.conf in 2024. Fortunately however, if the legacy config was recognizable, we were able to parse it as an advanced config and gain the benefit. If not, then it goes the legacy, and very expensive, route. Given the commonality among the modules, instead of making the same improvements to 4 modules then trying to maintain them over time, a single module "res_cdrel_custom" was created that contains all of the common code. A few bonuses became possible in the process... * The cdr_custom and cel_custom modules now support JSON formatted output. * The cdr_sqlite_custom and cel_sqlite3_custom modules no longer have to share an Sqlite3 database. Summary of changes: A new module "res/res_cdrel_custom.c" has been created and the existing cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom modules are now just stubs that call the code in res_cdrel_custom. res_cdrel_custom contains: * A common configuration facility. * Getters for both CDR and CEL fields that share the same abstraction. * Formatters for all data types found in the ast_cdr and ast_event structures that share the same abstraction. * Common writers for the text file and database backends that, you guessed it, share the same abstraction. The result is that while there is certainly a net increase in the number of lines in the code base, most of it is in the configuration handling at load-time. The run-time instruction path length is now significanty shorter. ``` Scenario Instructions Latency ===================================================== CEL pre changes 38.49% 37.51% CEL Advanced 9.68% 6.06% CEL Legacy (auto-conv to adv) 9.95% 6.13% CEL Sqlite3 pre changes 39.41% 39.90% CEL Sqlite3 Advanced 25.68% 24.24% CEL Sqlite3 Legacy (auto conv) 25.88% 24.53% CDR pre changes 4.79% 2.95% CDR Advanced 0.79% 0.47% CDR Legacy (auto conv to adv) 0.86% 0.51% CDR Sqlite3 pre changes 4.47% 2.89% CEL Sqlite3 Advanced 2.16% 1.29% CEL Sqlite3 Legacy (auto conv) 2.19% 1.30% ``` Notes: * We only write one CDR per call but every little bit helps. * Sqlite3 still takes a fair amount of resources but the new config makes a decent improvement. * Legacy configs that we can't auto convert will still take the "pre changes" path. If you're interested in more implementation details, see the comments at the top of the res_cdrel_custom.c file. One minor fix to CEL is also included...Although TenantID was added to the ast_event structure, it was always rendered as an empty string. It's now properly rendered. UserNote: Significant performance improvements have been made to the cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom modules. See the new sample config files for those modules to see how to benefit from them.
253 lines
13 KiB
Plaintext
253 lines
13 KiB
Plaintext
;
|
|
; Asterisk Channel Event Logging (CEL) - Custom DSV Backend
|
|
;
|
|
; This is the configuration file for the customizable DSV backend for CEL
|
|
; logging.
|
|
;
|
|
; DSV?? Delimiter Separated Values because the field delimiter doesn't have
|
|
; to be a comma.
|
|
;
|
|
; Legacy vs Advanced Mappings
|
|
;
|
|
; Legacy mappings are those that are defined using dialplan functions like
|
|
; CALLERID and CSV_QUOTE and require a VERY expensive function replacement
|
|
; process at runtime for every record output. In performance testing, 38%
|
|
; of the CPU instructions executed to handle a call were executed on behalf
|
|
; of legacy CEL logging. That's more than the instructions actually used
|
|
; to process the call itself.
|
|
;
|
|
; Advanced mappings are those that are defined by a list of field names and
|
|
; parameters that define the field separator and quote character you want to
|
|
; use. This type of mapping is uses significantly less resources at runtime.
|
|
; Performance testing showed advanced CEL logging accounting for less than
|
|
; 10% of the CPU instructions executed which is well below the instructions
|
|
; needed to process the call itself.
|
|
;
|
|
; There are several "special" variables created by this module that can be used
|
|
; in a mapping, both legacy and advanced:
|
|
;
|
|
; eventtype - The name of the CEL event.
|
|
; eventtime - The timestamp of the CEL event.
|
|
; eventenum - Like eventtype but is "USER_DEFINED" for a user defined event.
|
|
; userdeftype - User defined event type name from CELGenUserEvent().
|
|
; eventextra - Extra data included with this CEL event, typically along with
|
|
; an event of type USER_DEFINED from CELGenUserEvent().
|
|
; BRIDGEPEER - Bridged peer channel name at the time of the CEL event.
|
|
; CHANNEL(peer) could also be used.
|
|
;
|
|
; Legacy Mappings
|
|
;
|
|
; Within a legacy mapping, use the CALLERID(), CHANNEL() and CSV_QUOTE()
|
|
; functions to retrieve values from the CEL event (and pay the price).
|
|
;
|
|
; NOTE: If your legacy mapping uses commas as field separators and only the
|
|
; CSV_QUOTE, CALLERID and CHANNEL dialplan functions or one of the special
|
|
; variables, the module will attempt to strip the functions and create a much
|
|
; faster advanced mapping for it. However, we urge you to create a real
|
|
; advanced mapping and not rely on this process. If the mapping contains
|
|
; something not recognized it will go the slower legacy route.
|
|
;
|
|
; Each entry in the "mappings" category represents a separate output file.
|
|
; A filename that starts with a forward-slash '/' will be treated as an absolute
|
|
; path name. If it doesn't, it must be a file name with no directory separators
|
|
; which will be placed in the /var/log/asterisk/cel-custom directory.
|
|
;
|
|
;[mappings]
|
|
;Master.csv => ${CSV_QUOTE(${eventtype})},${CSV_QUOTE(${eventtime})},${CSV_QUOTE(${CALLERID(name)})},${CSV_QUOTE(${CALLERID(num)})},${CSV_QUOTE(${CALLERID(ANI)})},${CSV_QUOTE(${CALLERID(RDNIS)})},${CSV_QUOTE(${CALLERID(DNID)})},${CSV_QUOTE(${CHANNEL(exten)})},${CSV_QUOTE(${CHANNEL(context)})},${CSV_QUOTE(${CHANNEL(channame)})},${CSV_QUOTE(${CHANNEL(appname)})},${CSV_QUOTE(${CHANNEL(appdata)})},${CSV_QUOTE(${CHANNEL(amaflags)})},${CSV_QUOTE(${CHANNEL(accountcode)})},${CSV_QUOTE(${CHANNEL(uniqueid)})},${CSV_QUOTE(${CHANNEL(linkedid)})},${CSV_QUOTE(${BRIDGEPEER})},${CSV_QUOTE(${CHANNEL(userfield)})},${CSV_QUOTE(${userdeftype})},${CSV_QUOTE(${eventextra})}
|
|
|
|
;
|
|
; Advanced Mappings
|
|
;
|
|
; Each category in this file other than "mappings" represents a separate output file.
|
|
; A filename that starts with a forward-slash '/' will be treated as an absolute
|
|
; path name. If it doesn't, it must be a file name with no directory separators
|
|
; which will be placed in the /var/log/asterisk/cel-custom directory.
|
|
;
|
|
;[cel_master.csv] ; Output file or path name.
|
|
|
|
;format = dsv ; Advanced mappings can actually have two types
|
|
; "dsv" or "json". This example uses "dsv", which
|
|
; is the default, but see the example below for more
|
|
; info on the json format.
|
|
|
|
;separator_character = , ; The character to use for the field separator,
|
|
; It defaults to a comma but you can use any
|
|
; character you want. For instance, specify
|
|
; \t to separate the fields with a tab.
|
|
|
|
;quote_character = " ; The character to use as the quote character.
|
|
; It defaults to the double quote but again, it
|
|
; can be any character you want although the
|
|
; single-quote is the only other one that makes
|
|
; sense.
|
|
|
|
;quote_escape_character = " ; The character to use to escape the quote_character
|
|
; should the quote character actually appear in the
|
|
; field output. A good example of this is a caller
|
|
; id string like "My Name" <1000>. For true CSV
|
|
; compatibility, it must be the same as the quote
|
|
; character but a backslash might be acceptable for
|
|
; other formats.
|
|
|
|
;quoting_method = all ; Nothing says you _have_ to quote anything. The only
|
|
; time a field MUST be quoted is if it contains the
|
|
; separator character and this is handled automatically.
|
|
; Additionally, the following options are available:
|
|
; "all": Quote all fields. The default.
|
|
; "non_numeric": Quote all non numeric fields.
|
|
; "none": Don't quote any field (unless it contains
|
|
; a separator character).
|
|
|
|
;fields = EventType,eventenum,userdeftype,"Some Literal",EventTime,Name,Num,ani,rdnis,dnid,Exten,Context,ChanName,AppName,AppData,AMAFlags(amaflags),AccountCode,UniqueID(noquote),LinkedID(noquote),Peer,PeerAccount,UserField,EventExtra,TenantID
|
|
; This is the list of fields to include in the record. The field names are the
|
|
; same as in the legacy mapping but without any enclosing dialplan functions.
|
|
; You can specify literals to be placed in the output record by double-quoting
|
|
; them. There is also some special notation available in the form of "qualifiers".
|
|
; A qualifier is a list of tags, separated by the '^' character and placed
|
|
; directly after the field name and enclosed in parentheses.
|
|
;
|
|
; All fields can accept the "quote" or "noquote" qualifier when you want to exempt
|
|
; a field from the "quoting_method" policy. For example, when quoting_method=all,
|
|
; you can force the uniqueid field to not be quoted with `uniqueid(noquote)`. The
|
|
; example in fields above shows this.
|
|
;
|
|
; The default output format for the "EventTime" timestamp field is the "%Y-%m-%d %T"
|
|
; strftime string format however you can also format the field as an int64 or a
|
|
; float: `eventtime(int64)` or `eventtime(float)`.
|
|
;
|
|
; Unlike CDRs, the "amaflags" field is output as its numerical value by default
|
|
; for historical reasons. You can output it as its friendly string with
|
|
; `amaflags(amaflags)`. This will print "DOCUMENTATION" instead of "3" for instance.
|
|
;
|
|
; If you need to combine flags, use the caret '^' symbol: `eventtime(int64^noquote)`
|
|
;
|
|
; Final notes about "fields":
|
|
; Field names and qualifiers aren't case sensitive.
|
|
; You MUST use the comma to separate the fields, not the "separator_character".
|
|
; You MUST use the double-quote to indicate literal fields, not the "quote_character".
|
|
; Whitespace in "fields" is ignored except in literals.
|
|
|
|
;
|
|
; An Advanced JSON example:
|
|
;
|
|
;[cdr_master.json]
|
|
;format = json
|
|
;fields = EventType,eventenum,userdeftype,"My: Literal",EventTime(float),Name,Num,ani,rdnis,dnid,Exten,Context,ChanName,AppName,AppData,AMAFlags(amaflags),AccountCode,UniqueID,LinkedID,Peer,PeerAccount,UserField,EventExtra,TenantID
|
|
;
|
|
; In order to ensure valid JSON, the following settings are forced:
|
|
; The separator character is always the comma.
|
|
; The quote character is always the double-quote.
|
|
; The quote escape character is always the backslash.
|
|
; The quoting method is always "non_numeric".
|
|
;
|
|
; Since JSON requires both a name and value, the name is always
|
|
; the field name. For literals however you must specify the literal
|
|
; as a "name: value" pair as demonstrated above. The output record
|
|
; would then look something like:
|
|
; {"eventtype":"HANGUP","eventenum":"HANGUP","userdeftype":"","My":"Literal","eventtime":1771359872.0, ...}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
;
|
|
; Advanced Mappings
|
|
;
|
|
; Advanced mappings use SIGNIFICANTLY less resources than legacy mappings
|
|
; because we don't need to use dialplan function replacement.
|
|
;
|
|
;[cel_master_advanced.csv] ; The destination file name.
|
|
; Can be a name relative to ASTLOGDIR or an
|
|
; absolute path.
|
|
|
|
;format = csv ; Sets the output format. The default is "csv"
|
|
; but here "csv" really means "character-separated-values"
|
|
; because the separator doesn't have to be a comma.
|
|
; The other alternative is "json" (see example below).
|
|
|
|
;separator_character = , ; Set the character to use between fields.
|
|
; Defaults to a comma but other characters
|
|
; can be used. For example, if you want
|
|
; tab-separated fields, use \t as the separator.
|
|
|
|
;quote_character = " ; Set the quoting character.
|
|
; Defaults to double-quote (") but any character
|
|
; can be used although only the double and single
|
|
; quote (') characters make sense.
|
|
|
|
;quote_escape_character = " ; Sets the character used to escape quotes that
|
|
; may be in the field values. The default is the
|
|
; same character as the quote_character so an
|
|
; embedded JSON blob would look like this:
|
|
; "{""extra"":""somextratext""}"
|
|
; You could also use a backslash (\) in which case
|
|
; the blob would look like this:
|
|
; "{\"extra\":\"somextratext\"}"
|
|
|
|
;quoting_method = all ; Sets what/when to quote.
|
|
; all - Quote all fields. (the default)
|
|
; minimal - Only quote fields that have the
|
|
; separator character in them.
|
|
; non_numeric - Quote all non-numeric fields.
|
|
; none - Don't quote anything.
|
|
; Probably not a good idea but could
|
|
; be useful in special circumstances.
|
|
|
|
; The fields to output. These names correspond to the internal CEL event
|
|
; field names which is how some of the performance gains are realized.
|
|
; Anything not recognized as a field name will be printed as a literal in
|
|
; the output.
|
|
;
|
|
; CEL Event Field Names:
|
|
;
|
|
; eventtype - Could be a standard event name or a user event name
|
|
; eventenum - Will always be a standard event name or 'USER_DEFINED'
|
|
; eventtime - Uses the dateformat set in cel.conf.
|
|
; usereventname - Will be the user event name if set.
|
|
; cidname
|
|
; cidnum
|
|
; exten
|
|
; context
|
|
; channame
|
|
; appname
|
|
; appdata
|
|
; amaflags
|
|
; acctcode
|
|
; uniqueid
|
|
; userfield
|
|
; cidani
|
|
; cidrdnis
|
|
; ciddnid
|
|
; peer
|
|
; linkedid
|
|
; peeracct
|
|
; extra
|
|
; tenantid
|
|
;
|
|
; You MUST use the comma and double-quote here, not the separator or
|
|
; or quote characters specified above. The names are case-insensitive.
|
|
;
|
|
;fields = EventType,EventEnum,"My Literal",EventTime,UserEventName,CIDName,CIDNum,Exten,Context,ChanName,AppName,AppData,AMAFlags,AcctCode,UniqueID,UserField,CIDani,CIDrdnis,CIDdnid,Peer,LinkedID,PeerAcct,Extra,TenantID
|
|
|
|
;
|
|
; A tab-separated-value example:
|
|
;
|
|
;[cel_master.tsv]
|
|
;separator_character = \t
|
|
;fields = EventType,EventEnum,"My Literal",EventTime,UserEventName,CIDName,CIDNum,Exten,Context,ChanName,AppName,AppData,AMAFlags,AcctCode,UniqueID,UserField,CIDani,CIDrdnis,CIDdnid,Peer,LinkedID,PeerAcct,Extra,TenantID
|
|
|
|
;
|
|
; A JSON example:
|
|
;
|
|
; The separator and quoting options don't apply to JSON.
|
|
; Literals must be specified as a "name: value" pair or they'll be ignored.
|
|
;
|
|
;[cel_master.json]
|
|
;format = json
|
|
;fields = EventType,eventenum,userdeftype,"My: Literal",EventTime,CIDName,CIDNum,CIDani,CIDrdnis,CIDdnid,Exten,Context,ChanName,AppName,AppData,AMAFlags,AcctCode,UniqueID,LinkedID,Peer,PeerAcct,UserField,Extra,TenantID
|