# API and build change and fix summaries. Doc corrections
# and/or changes are not mentioned here; see the commit messages.
-2021 February 08; Version 4.6.0
+2021 February 22; Version 4.6.0
Enhanced to use millisecond based timestamps when writing log messages.
(RIC-627)
be used as if it were supplied via the environment variable. This should
make debugging in existing containers easier.
+ Enhanced to send an alarm manager when the user application is not
+ receiving messages fast enough. The original messages written to
+ the standard error are still generated. (RIC-204)
+
2021 January 21; Version 4.5.2
Fixes the excessive TCP session bug when sending to a slow receiver
and a related segment fault because of too many open file descriptors.
which RMR recognises:
&half_space
+.if false
&indent
&beg_dlist( 1.25i &ditext )
&ditem(RMR_BIND_IF) The interface to bind to listen ports to. If not defined 0.0.0.0 (all interfaces) is assumed.
&ditem(RMR_SRC_NAMEONLY) If the value of this variable is greater than 0, RMR will not permit the IP address to be
sent as the message source. Only the host name will be sent as the source in the message header.
&end_dlist
+
&uindent
.st &textsize
+.fi
+
+.** pull from man pages to avoid 2 places for maint
+.im ../man/env_var_list.im
&space
-&h2(Logging)
-RMR does &bold(not) use any logging libraries; any error or warning messages are written to standard error.
+There are other, non-RMR, variables which may exist and are used by RMR.
+These variable names are not under the control of RMR, so they are subject to change without potentiallyb being
+reflected in either RMR's code, or this document.
+The following is a list of these environment variables.
+
+&half_space
+&indent
+&beg_dlist( 1.25i &ditext )
+ &ditem(ALARM_MANAGER_SERVICE_NAME)
+ This is the DNS name, or IP address, of the process which is listening for RMR alarm messages.
+ If this variable is missing, &cw(service-ricplt-alarmmanager-rmr) is assumed.
+
+ &ditem(ALARM_MANAGER_SERVICE_PORT)
+ This is the port that the alarm manager is using to accept RMR messages. If the environment
+ variable is missing the value &cw(4560) is assumed.
+&end_dlist
+&uindent
+.st &textsize
+
+
+&h2(Logging and Alarms)
+As with nearly all UNIX libraries, errors, warnings and informational messages are written in plain text to the
+standard error device (stderr).
+All RMR messages are prefixed with the current time (in milliseconds past the standard UNIX epoch), the process ID,
+and a severity indicator.
.if false
+RMR does &bold(not) use any logging libraries; any error or warning messages are written to standard error.
¬e .sm .
.cn l=&cn_line_len i=0 start &atbot Times-roman 8p .7i
This is standard practice for container based applications as it makes error output easily available to operations.
.cn end
.fi
-RMR messages are written with one of three prefix strings:
+RMR messages are written with one of three severity strings:
&half_space
&end_dlist
&space
&uindent
+
+&h3(Log message supression)
+For the most part, the &ital(fast path) code in RMR does no logging; even when messages are squelched, there is a non-zero
+cosst to check for the setting each time a potential message is to be written.
+To that end, RMRM will log only severe errors once initialisation has completed.
+An exception to this policy exists in the route table collection thread.
+The thread of execution which collects route table updates does not need to be concerned with performance, and as such
+has the potential to log its actions in a very verbose manner.
+The environment variable &cw( RMR_VCTL_FILE ) can be used to define a file where the desired verbosity level (0 to 4
+where 0 is off) can be placed.
+If the environment variable is not set when the process starts, RMR will assume that the file &cw(/tmp/rmr.v) will be used.
+Beginning with version 4.6.0 this file does &bold(not) need to exist when the process is started.
+To change the verbosity level, the desired value is written to the file on the first line.
+
+&h3(Alarms)
+The route table colleciton thread is also responsible for watching for situations which need to be reported as alarms
+to the platform's alarm management service.
+When a state exists RMR will create and send an alarm (via RMR message) to the alarm service, and will send a &ital(clear)
+message when the state no longer exists.
+Currently RMR will alarm only when the application is not removing messages from the receive ring quicklye enough causing
+RMR to drop messages as they are received.
.ix synonym `round robin` round-robin
.ix snare `route table`
+.ix snare `standard error`
+.ix synonym `standard error` stderr
+.ix snare `standard out`
+.ix synonym `standard out` stdout
+
.ix snare `subscription ID`
.ix synonym `subscription ID` `subscription ID,`
.ix synonym `subscription ID` `subID`
.if false
==================================================================================
- Copyright (c) 2019-2020 Nokia
- Copyright (c) 2019-2020 AT&T Intellectual Property.
+ Copyright (c) 2019-2021 Nokia
+ Copyright (c) 2019-2021 AT&T Intellectual Property.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
.ln
.ll -.5i
.in +.25i
-Copyright (c) 2019-2020 Nokia
+Copyright (c) 2019-2021 Nokia
.br
-Copyright (c) 2019-2020 AT&T Intellectual Property.
+Copyright (c) 2019-2021 AT&T Intellectual Property.
.sp 1
Licensed under the Apache License, Version 2.0 (the "License");
.dv reverse_titles 1
.** setup will do the right thing with the index configuration
+.dv toc_file user.toc
.dv index_snare_file index_snatch.im
.im ./setup.im
version 4.0.0, the RMR versions should no longer skip.
-2021 January21; Version 4.5.2
------------------------------
+2021 February 22; Version 4.6.0
+-------------------------------
+
+Enhanced to use millisecond based timestamps when writing log
+messages. (RIC-627)
+
+Enhanced to eliminate the reqirement that the file specified
+by the RMR_VCTL_FILE environment variable exist when RMR is
+initialised. RMR will correctly find this file if it is
+created after initialisation. Further, if the environment
+variable is not given, the user may create and populate
+/tmp/rmr.v at any time and the contents of this file will be
+used as if it were supplied via the environment variable.
+This should make debugging in existing containers easier.
+
+Enhanced to send an alarm manager when the user application
+is not receiving messages fast enough. The original messages
+written to the standard error are still generated. (RIC-204)
+
+
+
+2021 January 21; Version 4.5.2
+------------------------------
Fixes the excessive TCP session bug when sending to a slow
receiver and a related segment fault because of too many open
:header-rows: 0
:class: borderless
- * - **RMR_BIND_IF**
+ * - **RMR_ASYNC_CONN**
-
- The interface to bind to listen ports to. If not defined
- 0.0.0.0 (all interfaces) is assumed.
+ Allows the async connection mode to be turned off (by setting
+ the value to 0). When set to 1, or missing from the
+ environment, RMR will invoke the connection interface in the
+ transport mechanism using the non-blocking (async) mode. This
+ will likely result in many "soft failures" (retry) until the
+ connection is established, but allows the application to
+ continue unimpeded should the connection be slow to set up.
- * - **RMR_RTG_SVC**
+ * - **RMR_BIND_IF**
-
- This variabe supplies the host:port (or address:port) of the
- Route Manager (route table generator) process. RMR will
- attempt to connect to this address port combination and
- request a route table. If it is desired to prevent RMR from
- attempting to request a dynamic route table, the value of
- this variable should be set to "-1." If not set
- ``routemgr`` is assumed.
+ This provides the interface that RMR will bind listen ports
+ to, allowing for a single interface to be used rather than
+ listening across all interfaces. This should be the IP
+ address assigned to the interface that RMR should listen on,
+ and if not defined RMR will listen on all interfaces.
* - **RMR_CTL_PORT**
-
- This is the port which RMR's route table collector thread
- will use to listen for RMR messages from the route manager
- (route table generator). By default this is 4561, and must be
- unique for each RMR process running on the host/container.
+ This variable defines the port that RMR should open for
+ communications with Route Manager, and other RMR control
+ applications. If not defined, the port 4561 is assumed.
+
+ Previously, the ``RMR_RTG_SVC`` (route table generator
+ service port) was used to define this port. However, a future
+ version of Route Manager will require RMR to connect and
+ request tables, thus that variable is now used to supply the
+ Route Manager's well-known address and port.
+
+ To maintain backwards compatibility with the older Route
+ Manager versions, the presence of this variable in the
+ environment will shift RMR's behaviour with respect to the
+ default value used when ``RMR_RTG_SVC`` is **not** defined.
+
+ When ``RMR_CTL_PORT`` is **defined:** RMR assumes that Route
+ Manager requires RMR to connect and request table updates is
+ made, and the default well-known address for Route manager is
+ used (routemgr:4561).
+
+ When ``RMR_CTL_PORT`` is **undefined:** RMR assumes that
+ Route Manager will connect and push table updates, thus the
+ default listen port (4561) is used.
+
+ To avoid any possible misinterpretation and/or incorrect
+ assumptions on the part of RMR, it is recommended that both
+ the ``RMR_CTL_PORT`` and ``RMR_RTG_SVC`` be defined. In the
+ case where both variables are defined, RMR will behave
+ exactly as is communicated with the variable's values.
* - **RMR_RTREQ_FREQ**
-
- When a new route table is needed, the frequency that RMR
- sends a route table request to the Route Manager defaults to
- 5 seconds. This variable can be used to set the frequency to
- a value between 1 and 300 seconds inclusive.
+ When RMR needs a new route table it will send a request once
+ every ``n`` seconds. The default value for ``n`` is 5, but
+ can be changed if this variable is set prior to invoking the
+ process. Accepted values are between 1 and 300 inclusive.
- * - **RMR_SEED_RT**
+ * - **RMR_RTG_SVC**
+ -
+ The value of this variable depends on the Route Manager in
+ use.
+
+ When the Route Manager is expecting to connect to an xAPP and
+ push route tables, this variable must indicate the
+ ``port`` which RMR should use to listen for these
+ connections.
+
+ When the Route Manager is expecting RMR to connect and
+ request a table update during initialisation, the variable
+ should be the ``host`` of the Route Manager process.
+
+ The ``RMR_CTL_PORT`` variable (added with the support of
+ sending table update requests to Route manager), controls the
+ behaviour if this variable is not set. See the description of
+ that variable for details.
+
+ * - **RMR_HR_LOG**
+ -
+ By default RMR writes messages to standard error (incorrectly
+ referred to as log messages) in human readable format. If
+ this environment variable is set to 0, the format of standard
+ error messages might be written in some format not easily
+ read by humans. If missing, a value of 1 is assumed.
+
+ * - **RMR_LOG_VLEVEL**
-
- Where RMR expects to find the name of the seed (static) route
- table. If not defined no static table is read.
+ This is a numeric value which corresponds to the verbosity
+ level used to limit messages written to standard error. The
+ lower the number the less chatty RMR functions are during
+ execution. The following is the current relationship between
+ the value set on this variable and the messages written:
+
+
+ .. list-table::
+ :widths: auto
+ :header-rows: 0
+ :class: borderless
+
+ * - **0**
+ -
+ Off; no messages of any sort are written.
+
+ * - **1**
+ -
+ Only critical messages are written (default if this variable
+ does not exist)
+
+ * - **2**
+ -
+ Errors and all messages written with a lower value.
+
+ * - **3**
+ -
+ Warnings and all messages written with a lower value.
+
+ * - **4**
+ -
+ Informational and all messages written with a lower value.
+
+ * - **5**
+ -
+ Debugging mode -- all messages written, however this requires
+ RMR to have been compiled with debugging support enabled.
+
+
* - **RMR_RTG_ISRAW**
-
- If the value set to 0, RMR expects the route table manager
- messages to be messages with and RMR header. If this is not
- defined messages are assumed to be "raw" (without an RMR
- header.
+ **Deprecated.** Should be set to 1 if the route table
+ generator is sending "plain" messages (not using RMR to send
+ messages), 0 if the RTG is using RMR to send. The default is
+ 1 as we don't expect the RTG to use RMR.
+
+ This variable is only recognised when using the NNG transport
+ library as it is not possible to support NNG "raw"
+ communications with other transport libraries. It is also
+ necessary to match the value of this variable with the
+ capabilities of the Route Manager; at some point in the
+ future RMR will assume that all Route Manager messages will
+ arrive via an RMR connection and will ignore this variable.
+
+ * - **RMR_SEED_RT**
+ -
+ This is used to supply a static route table which can be used
+ for debugging, testing, or if no route table generator
+ process is being used to supply the route table. If not
+ defined, no static table is used and RMR will not report
+ *ready* until a table is received. The static route table may
+ contain both the route table (between newrt start and end
+ records), and the MEID map (between meid_map start and end
+ records).
+
+ * - **RMR_SRC_ID**
+ -
+ This is either the name or IP address which is placed into
+ outbound messages as the message source. This will used when
+ an RMR based application uses the rmr_rts_msg() function to
+ return a response to the sender. If not supplied RMR will use
+ the hostname which in some container environments might not
+ be routable.
+
+ The value of this variable is also used for Route Manager
+ messages which are sent via an RMR connection.
* - **RMR_VCTL_FILE**
-
- Provides a file which is used to set the verbose level of the
- route table collection thread. The first line of the file is
- read and expected to contain an integer value to set the
- verbose level. The value may be changed at any time and the
- route table thread will adjust accordingly.
+ This supplies the name of a verbosity control file. The core
+ RMR functions do not produce messages unless there is a
+ critical failure. However, the route table collection thread,
+ not a part of the main message processing component, can
+ write additional messages to standard error. If this variable
+ is set, RMR will extract the verbosity level for these
+ messages (0 is silent) from the first line of the file.
+ Changes to the file are detected and thus the level can be
+ changed dynamically, however RMR will only suss out this
+ variable during initialisation, so it is impossible to enable
+ verbosity after startup.
- * - **RMR_SRC_NAMEONLY**
+ * - **RMR_WARNINGS**
-
- If the value of this variable is greater than 0, RMR will not
- permit the IP address to be sent as the message source. Only
- the host name will be sent as the source in the message
- header.
+ If set to 1, RMR will write some warnings which are
+ non-performance impacting. If the variable is not defined, or
+ set to 0, RMR will not write these additional warnings.
+There are other, non-RMR, variables which may exist and are
+used by RMR. These variable names are not under the control
+of RMR, so they are subject to change without potentiallyb
+being reflected in either RMR's code, or this document. The
+following is a list of these environment variables.
-Logging
--------
+ .. list-table::
+ :widths: auto
+ :header-rows: 0
+ :class: borderless
+
+ * - **ALARM_MANAGER_SERVICE_NAME**
+ -
+ This is the DNS name, or IP address, of the process which is
+ listening for RMR alarm messages. If this variable is
+ missing, ``service-ricplt-alarmmanager-rmr`` is assumed.
+
+ * - **ALARM_MANAGER_SERVICE_PORT**
+ -
+ This is the port that the alarm manager is using to accept
+ RMR messages. If the environment variable is missing the
+ value ``4560`` is assumed.
-RMR does **not** use any logging libraries; any error or
-warning messages are written to standard error. RMR messages
-are written with one of three prefix strings:
+
+
+
+Logging and Alarms
+------------------
+
+As with nearly all UNIX libraries, errors, warnings and
+informational messages are written in plain text to the
+standard error device (stderr). All RMR messages are prefixed
+with the current time (in milliseconds past the standard UNIX
+epoch), the process ID, and a severity indicator. RMR
+messages are written with one of three severity strings:
.. list-table::
+Log message supression
+----------------------
+
+For the most part, the *fast path* code in RMR does no
+logging; even when messages are squelched, there is a
+non-zero cosst to check for the setting each time a potential
+message is to be written. To that end, RMRM will log only
+severe errors once initialisation has completed. An exception
+to this policy exists in the route table collection thread.
+The thread of execution which collects route table updates
+does not need to be concerned with performance, and as such
+has the potential to log its actions in a very verbose
+manner. The environment variable `` RMR_VCTL_FILE `` can be
+used to define a file where the desired verbosity level (0 to
+4 where 0 is off) can be placed. If the environment variable
+is not set when the process starts, RMR will assume that the
+file ``/tmp/rmr.v`` will be used. Beginning with version
+4.6.0 this file does **not** need to exist when the process
+is started. To change the verbosity level, the desired value
+is written to the file on the first line.
+
+
+Alarms
+------
+
+The route table colleciton thread is also responsible for
+watching for situations which need to be reported as alarms
+to the platform's alarm management service. When a state
+exists RMR will create and send an alarm (via RMR message) to
+the alarm service, and will send a *clear* message when the
+state no longer exists. Currently RMR will alarm only when
+the application is not removing messages from the receive
+ring quicklye enough causing RMR to drop messages as they are
+received.
+
+
Notes
=====
#define ENV_CTL_PORT "RMR_CTL_PORT" // route collector will listen here for control messages (4561 default)
#define ENV_RTREQ_FREA "RMR_RTREQ_FREQ" // frequency we will request route table updates when we want one (1-300 inclusive)
+
+#define ENV_AM_NAME "ALARM_MGR_SERVICE_NAME" // alarm manager env vars that we need
+#define ENV_AM_PORT "ALARM_MGR_SERVICE_PORT"
+
#define NO_FLAGS 0 // no flags to pass to a function
#define FL_NOTHREAD 0x01 // do not start an additional thread (must be 'user land' to support rtg
#define HFL_SUBID 0x02 // subscription ID is populated
#define HFL_CALL_MSG 0x04 // msg sent via blocking call
+/*
+ Alarm action constants describe the type (e.g. dropping messages) and whether or not
+ this is a "raise" or "clear" action. Raise/clear is determined by the least significant
+ bit; 1 == raise.
+*/
+#define ALARM_RAISE 0x01
+#define ALARM_CLEAR 0x00
+#define ALARM_KIND(a) (a&ALARM_MASK)
+#define ALARM_DROPS 0x02
+#define ALARM_MASK 0xfffe
+
/*
Message header; interpreted by the other side, but never seen by
the user application.
static int ie_test( void* r, int i_factor, long inserts );
+// --- internal alarm generation ---------------------
+static void uta_alarm( void* vctx, int kind, int prob_id, char* info );
+static void uta_alarm_send( void* vctx, rmr_mbuf_t* msg );
+
// ----- route table generic static things ---------
static inline uint64_t build_rt_key( int32_t sub_id, int32_t mtype );
static void collect_things( void* st, void* entry, char const* name, void* thing, void* vthing_list );
// ----- prototypes ------------------------------
+extern long long mstime( );
extern void rmr_vlog( int write_level, char* fmt, ... );
extern void rmr_vlog_force( int write_level, char* fmt, ... );
extern int rmr_vlog_init();
--- /dev/null
+// : vi ts=4 sw=4 noet :
+/*
+==================================================================================
+ Copyright (c) 2021 Nokia
+ Copyright (c) 2021 AT&T Intellectual Property.
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+==================================================================================
+*/
+
+/*
+------------------------------------------------------------------------------
+Mnemonic: alarm.c
+Abstract: The RMR housekeeping thread (route table collection) is also responsible
+ for sending a small number of alarm messages to the alarm system.
+ To avoid extra overhead, the alarm library is NOT included; this
+ module provides the alarm generation support for the housekeeping
+ thread.
+
+Date: 19 February 2021
+ HBD-EKD
+Author: E. Scott Daniels
+
+------------------------------------------------------------------------------
+*/
+
+#ifndef alarm_static_c
+#define alarm_static_c
+
+#include <errno.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <string.h>
+#include <stdlib.h>
+#include <memory.h>
+#include <netdb.h>
+#include <pthread.h>
+
+#include <rmr.h>
+#include <RIC_message_types.h>
+#include <rmr_logging.h>
+
+
+extern const char* __progname; // runtime supplies (we have no acccess to argv[0]
+
+/*
+ Generate the endpoint for the arlarm system. We assume that it is defined
+ by two environment variables: ALARM_MGR_SERVICE_NAME and ALARM_MGR_SERVICE_PORT.
+ If the name is not given, we assume 127.0.0.1 (we don't use localhost for
+ to avoid the double meaning within some flavours' hosts files). If the port
+ is not given, we assume the default RMR port of 4560.
+
+ The string returned must be freed by the caller.
+*/
+static char* uta_alarm_endpt( ) {
+ char wbuf[300]; // enough room for max name of 256 + :port
+ const char* et; // environment token
+ const char* addr = "service-ricplt-alarmmanager-rmr"; // defaults
+ const char* port = "4560";
+
+ if( (et = getenv( ENV_AM_NAME )) != NULL ) {
+ addr = et;
+ }
+
+ if( (et = getenv( ENV_AM_PORT )) != NULL ) {
+ port = et;
+ }
+
+ wbuf[sizeof(wbuf)-1] = 0; // guarentee it is nil terminated if too long
+ snprintf( wbuf, sizeof( wbuf ) -1, "%s:%s", addr, port );
+ return strdup( wbuf );
+}
+
+/*
+ Send the message as an alarm. We will create a wormhole as we cannot depend
+ on having a routing table that is valid. This send does NOT allocate a new
+ message to return to the caller.
+*/
+static void uta_alarm_send( void* vctx, rmr_mbuf_t* msg ) {
+ static int whid = -1;
+ static int ok2log = 0; // if we must log to stderr, do it seldomly; this is the next spot in time where logging is ok
+ char* ept;
+ int now;
+
+ if( whid < 0 ) {
+ ept = uta_alarm_endpt();
+ if( (whid = rmr_wh_open( vctx, ept )) < 0 ) {
+ if( (now = time( NULL )) > ok2log ) {
+ ok2log = now + 60;
+ }
+ rmr_vlog( RMR_VL_WARN, "unable to open wormhole for alarms: %s\n", ept );
+ free( ept );
+ return;
+ }
+ rmr_vlog( RMR_VL_INFO, "alarm wormhole opened to: %s\n", ept );
+ free( ept );
+ }
+
+ if( msg != NULL ) {
+ msg = rmr_wh_send_msg( vctx, whid, msg );
+ rmr_free_msg( msg );
+ }
+}
+
+
+/*
+ Builds an alarm message for the kind of action. Alarm message is palced in
+ the given buffer using the buffer length (blen) to truncate. A convenience
+ pointer to the buffer is returned and will be nil on error.
+
+ Kind is an ALARM_* constant ANDed with ALARM_RAISE or ALARM_CLEAR.
+ Problem ID (prob_id) is an integer assigned by the caller and should
+ match when calling to clear the alarm.
+
+ Info is the message text that is added to the alarm; nil terminated string.
+*/
+static void uta_alarm( void* vctx, int kind, int prob_id, char* info ) {
+ rmr_mbuf_t* msg;
+ int used;
+ char* maction = "CLEAR";
+
+ if( (msg = rmr_alloc_msg( vctx, 2048 )) == NULL ) {
+ return;
+ }
+
+ if( kind & ALARM_RAISE ) {
+ maction = "RAISE";
+ }
+
+ if( info == NULL ) {
+ info = "unspecific RMR issue";
+ }
+
+ used = snprintf( msg->payload, 2047,
+ "{ "
+ "\"managedObjectId\": null, " // there is no object for rmr alarms
+ "\"applicationId\": \"%s\", "
+ "\"specificProblem\": %d, " // problem id
+ "\"perceivedSeverity\": \"WARNING\", " // all RMR messages are warnings
+ "\"identifyingInfo\": \"%s\", "
+ "\"additionalInfo\": null, "
+ "\"AlarmAction\": \"%s\", "
+ "\"AlarmTime\": %lld"
+ " }",
+
+ __progname, // see comment above
+ prob_id,
+ info,
+ maction, // raise/clear
+ mstime() // use logging module's timestamp generator
+ );
+
+ msg->len = strlen( msg->payload );
+ msg->state = RMR_OK;
+ msg->sub_id = -1;
+ msg->mtype = RIC_ALARM;
+ uta_alarm_send( vctx, msg );
+}
+
+
+
+#endif
If time() returns 515300400, this function will add three didgets which
represent the milliseconds: 515300400123 (515300400.123).
*/
-static long long mstime( ) {
+extern long long mstime( ) {
struct timespec now;
long long rv = 0;
}
}
+// ---- alarm generation --------------------------------------------------------------------------
+
+/*
+ Given the user's context (not the thread private context) look to see if the application isn't
+ working fast enough and we're dropping messages. If the drop counter has changed since the last
+ peeked, and we have not raised an alarm, then we will alarm. If the counter hasn't changed, then we
+ set a timer and if the counter still hasn't changed when it expires we will clear the alarm.
+
+ The private context is what we use to send so as not to interfere with the user flow.
+*/
+static void alarm_if_drops( uta_ctx_t* uctx, uta_ctx_t* pctx ) {
+ static int alarm_raised = 0;
+ static int ok2clear = 0; // time that we can clear
+ static int lastd = 0; // the last counter value so we can compute delta
+ static int prob_id = 0; // problem ID we assume alarm manager handles dups between processes
+
+ rmr_vlog( RMR_VL_DEBUG, "checking for drops... raised=%d 0k2clear=%d lastd=%d probid=%d\n", alarm_raised, ok2clear, lastd, prob_id );
+ if( ! alarm_raised ) {
+ if( uctx->dcount - lastd == 0 ) { // not actively dropping, ok to do nothing
+ return;
+ }
+
+ alarm_raised = 1;
+ uta_alarm( pctx, ALARM_DROPS | ALARM_RAISE, prob_id, "application running slow; RMR is dropping messages" );
+ rmr_vlog( RMR_VL_INFO, "drop alarm raised" );
+ } else {
+ if( uctx->dcount - lastd != 0 ) { // still dropping or dropping again; we've alarmed so nothing to do
+ lastd = uctx->dcount;
+ ok2clear = 0; // reset the timer
+ return;
+ }
+
+ if( ok2clear == 0 ) { // first round where not dropping
+ ok2clear = time( NULL ) + 60; // we'll clear the alarm in 60s
+ } else {
+ if( time( NULL ) > ok2clear ) { // things still stable after expiry
+ rmr_vlog( RMR_VL_INFO, "drop alarm cleared\n" );
+ alarm_raised = 0;
+ uta_alarm( pctx, ALARM_DROPS | ALARM_CLEAR, prob_id++, "RMR message dropping has stopped" );
+ }
+ }
+ }
+}
+
// ---- utility -----------------------------------------------------------------------------------
/*
Little diddy to trim whitespace and trailing comments. Like shell, trailing comments
if( ctx->shutdown != 0 ) {
break; // mostly for unit test, but allows a forced stop
}
+
+ // extra housekeeping chores can be added here...
+ alarm_if_drops( ctx, pvt_cx ); // send an alarm if message are dropping, clear if we set one and thtings are better
}
vlevel = refresh_vlevel( 0 ); // ensure it's fresh when we get a message
rt_epcounts( ctx->rtable, ctx->my_name );
}
}
+
+ alarm_if_drops( ctx ); // send an alarm if message are dropping, clear if we set one and thtings are better
}
vlevel = refresh_vlevel( 0 ); // ensure it's fresh when we get a message
int d2_len; // extra header data 2 length (future)
int nn_sock; // our general listen socket
int rtable_ready; // set to true when rt is received or loaded
+ int dcount; // drop counter when app is slow
route_table_t* rtable; // the active route table
route_table_t* old_rtable; // the previously used rt, sits here to allow for draining
route_table_t* new_rtable; // route table under construction
static inline void queue_normal( uta_ctx_t* ctx, rmr_mbuf_t* mbuf ) {
static time_t last_warning = 0;
- static long dcount = 0;
+ //static long dcount = 0;
chute_t* chute;
if( ! uta_ring_insert( ctx->mring, mbuf ) ) {
rmr_free_msg( mbuf ); // drop if ring is full
- dcount++;
+ //dcount++;
+ ctx->dcount++;
if( time( NULL ) > last_warning + 60 ) { // issue warning no more frequently than every 60 sec
- rmr_vlog( RMR_VL_ERR, "rmr_mt_receive: application is not receiving fast enough; %ld msgs dropped since last warning\n", dcount );
+ rmr_vlog( RMR_VL_ERR, "rmr_mt_receive: application is not receiving fast enough; %ld msgs dropped since last warning\n", ctx->dcount );
last_warning = time( NULL );
- dcount = 0;
+ ctx->dcount = 0;
}
return;
#include "ring_static.c" // message ring support
#include "rt_generic_static.c" // route table things not transport specific
#include "rtable_si_static.c" // route table things -- transport specific
+#include "alarm.c"
#include "rtc_static.c" // route table collector (thread code)
#include "tools_static.c"
#include "sr_si_static.c" // send/receive static functions
--- /dev/null
+// : vi ts=4 sw=4 noet :
+/*
+==================================================================================
+ Copyright (c) 2021 Nokia
+ Copyright (c) 2021 AT&T Intellectual Property.
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+==================================================================================
+*/
+
+/*
+ Mmemonic: alarm_test.c
+ Abstract: Unit test for common/src/alarm.c functions.
+
+ Author: E. Scott Daniels
+ Date: 22 February 2021
+*/
+
+#ifdef KEEP
+
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <strings.h>
+#include <errno.h>
+#include <string.h>
+#include <stdint.h>
+#include <pthread.h>
+#include <semaphore.h>
+
+
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <strings.h>
+#include <errno.h>
+#include <string.h>
+#include <stdint.h>
+#include <ctype.h>
+//#include <sys/epoll.h>
+//#include <pthread.h>
+//#include <semaphore.h>
+
+#define DEBUG 1 // must define before pulling in rmr header files
+#define PARANOID_CHECKS 1 // must have parinoid testing on to not fail on nil pointer tests
+
+
+ // specific test tools in this directory
+#undef NNG_UNDER_TEST
+#include "test_support.c" // things like fail_if()
+#include "test_msg_support.c"
+#include "test_gen_rt.c"
+
+
+#include "rmr.h" // things the users see
+#include "rmr_agnostic.h" // rmr private things
+
+#include "rmr_symtab.h" // must pull in for context setup
+#include "rmr_agnostic.h" // transport agnostic header
+
+#include "logging.c"
+#include "rt_generic_static.c"
+#include "tools_static.c"
+#include "symtab.c"
+//#include "rmr_si.c"
+//#include "mbuf_api.c"
+
+#include "test_ctx_support.c" // dummy context support (needs symtab defs, so not with others above)
+
+
+//static void gen_rt( uta_ctx_t* ctx ); // defined in sr_si_static_test, but used by a few others (eliminate order requirement below)
+
+ // and finally.... the things under test
+#include "alarm.c"
+//#include "tools_static_test.c" // local test functions pulled directly because of static nature of things
+//#include "symtab_static_test.c"
+//#include "ring_static_test.c"
+//#include "rt_static_test.c"
+//#include "wormhole_static_test.c"
+//#include "mbuf_api_static_test.c"
+//#include "sr_si_static_test.c"
+//#include "lg_buf_static_test.c"
+// do NOT include the receive test static must be stand alone
+
+
+#endif
+
+/*
+ These tests assume there is a dummy process listening on 127.0.0.1:1986; if it's not there
+ the tests still pass, but coverage is reduced because the sends never happen.
+*/
+static int alarm_test( ) {
+ int errors = 0; // number errors found
+ uta_ctx_t* ctx;
+ char* endpt = NULL;
+
+ ctx = mk_dummy_ctx();
+ gen_rt( ctx );
+
+ ctx->my_name = strdup( "private" );
+ ctx->my_ip = strdup( "30.4.19.86:2750" );
+
+ endpt = uta_alarm_endpt(); // check defaults are generated
+ if( fail_if_nil( endpt, "alarm endpoint did not generate a string for defaults" ) ) {
+ errors++;
+ } else {
+ errors += fail_if_false( strcmp( endpt, "service-ricplt-alarmmanager-rmr:4560" ) == 0, "alarm endpoint default string not expected" );
+ free( endpt );
+ }
+
+ setenv( "ALARM_MGR_SERVICE_NAME", "127.0.0.1", 1 ); // test to ensure digging from env is good too
+ setenv( "ALARM_MGR_SERVICE_PORT", "999", 1 ); // connect should fail
+ endpt = uta_alarm_endpt(); // check defaults are generated
+ uta_alarm( ctx, ALARM_RAISE, 0, "some info for the alarm" ); // this should fail as the service isn't running
+
+ setenv( "ALARM_MGR_SERVICE_NAME", "127.0.0.1", 1 ); // test to ensure digging from env is good too
+ setenv( "ALARM_MGR_SERVICE_PORT", "1986", 1 );
+ endpt = uta_alarm_endpt(); // check defaults are generated
+ if( fail_if_nil( endpt, "alarm endpoint did not generate a string when name/port are set in env" ) ) {
+ errors++;
+ } else {
+ errors += fail_if_false( strcmp( endpt, "127.0.0.1:1986" ) == 0, "alarm endpoint string not expected when values are in env" );
+ free( endpt );
+ }
+
+
+ // these functions do not return values; driving for coverage and crash testing
+ uta_alarm( ctx, ALARM_RAISE, 0, "some info for the alarm" );
+ uta_alarm( ctx, ALARM_CLEAR, 0, NULL );
+ uta_alarm_send( ctx, NULL ); // ensure nil message doesn't crash us
+
+
+ if( ctx ) {
+ free( ctx->my_name );
+ free( ctx->my_ip );
+ free( ctx );
+ }
+
+ return !!errors; // 1 or 0 regardless of count
+}
+/*
+
+int main( ) {
+ int errors = 0;
+
+ errors += alarm_test();
+ exit( !!errors );
+}
+*/
#include "mbuf_api_static_test.c"
#include "sr_si_static_test.c"
#include "lg_buf_static_test.c"
+#include "alarm_static_test.c"
// do NOT include the receive test static must be stand alone
#include "rmr_si_api_static_test.c"
rmr_set_vlevel( 5 ); // enable all debugging
+ fprintf( stderr, "\n<INFO> starting alarm tests (%d)\n", errors );
+ errors += alarm_test();
+ fprintf( stderr, "<INFO> error count: %d\n", errors );
+
fprintf( stderr, "\n<INFO> starting lg buffer tests (%d)\n", errors );
errors += rmr_lgbuf_test();
fprintf( stderr, "<INFO> error count: %d\n", errors );
/*
If em_send_failures is true, this will fail a small part of the time
to simualte connection failures.
+
+ If the port number is < 1000 it will fail -- allowing for specific,single
+ failure cases.
*/
static int em_next_fd = 0;
static int em_siconnect( struct ginfo_blk *gptr, char *abuf ) {
static int count = 0;
+ char* tok;
if( em_send_failures && (count++ % 15 == 14) ) {
//fprintf( stderr, "<SIEM> siem is failing connect attempt\n\n" );
return -1;
}
+ if( (tok = strchr( abuf, ':' )) != NULL && atoi( tok+1 ) < 1000 ) {
+ fprintf( stderr, "<SIEM> siem is emulating connect to (%s) with a failure; port <1000\n", abuf );
+ return -1;
+ }
+
fprintf( stderr, "<SIEM> siem is emulating connect to (%s) attempt return fd=%d\n", abuf, em_next_fd );
if( em_next_fd < 50 ) {
em_next_fd++;
Code Review / ric-plt / lib / rmr.git / commitdiff