1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
3 .. CAUTION: this document is generated from source in doc/src/rtd.
4 .. To make changes edit the source and recompile the document.
5 .. Do NOT make changes directly to .rst or .md files.
7 ============================================================================================
9 ============================================================================================
10 --------------------------------------------------------------------------------------------
11 RIC Message Router -- RMR
12 --------------------------------------------------------------------------------------------
18 The RIC Message Router (RMR) is a library for peer-to-peer
19 communication. Applications use the library to send and
20 receive messages where the message routing and endpoint
21 selection is based on the message type rather than DNS host
22 name-IP port combinations. The library provides the following
26 * Routing and endpoint selection is based on *message type.*
28 * Application is insulated from the underlying transport
29 mechanism and/or protocols.
31 * Message distribution (round robin or fanout) is selectable
34 * Route management updates are received and processed
35 asynchronously and without overt application involvement.
43 RMR's main purpose is to provide an application with the
44 ability to send and receive messages to/from other peer
45 applications with minimal effort on the application's part.
46 To achieve this, RMR manages all endpoint information,
47 connections, and routing information necessary to establish
48 and maintain communication. From the application's point of
49 view, all that is required to send a message is to allocate
50 (via RMR) a message buffer, add the payload data, and set the
51 message type. To receive a message, the application needs
52 only to invoke the receive function; when a message arrives a
53 message buffer will be returned as the function result.
59 Applications are required to place a message type into a
60 message before sending, and may optionally add a subscription
61 ID when appropriate. The combination of message type, and
62 subscription ID are refered to as the *message key,* and is
63 used to match an entry in a routing table which provides the
64 possible endpoints expecting to receive messages with the
71 An endpoint from RMR's perspective is an application to which
72 RMR may establish a connection, and expect to send messages
73 with one or more defined message keys. Each entry in the
74 route table consists of one or more endpoint groups, called
75 round robin groups. When a message matches a specific entry,
76 the entry's groups are used to select the destination of the
77 message. A message is sent once to each group, with messages
78 being *balanced* across the endpoints of a group via round
79 robin selection. Care should be taken when defining multiple
80 groups for a message type as there is extra overhead required
81 and thus the overall message latency is somewhat increased.
87 Route table information is made available to RMR a static
88 file (loaded once), or by updates sent from a separate route
89 manager application. If a static table is provided, it is
90 loaded during RMR initialization and will remain in use until
91 an external process connects and delivers a route table
92 update (often referred to as a dynamic update). Dynamic
93 updates are listened for in a separate process thread and
94 applied automatically; the application does not need to allow
95 for, or trigger, updates.
98 Latency And Throughput
99 ----------------------
101 While providing insulation from the underlying message
102 transport mechanics, RMR must also do so in such a manner
103 that message latency and throughput are not impacted. In
104 general, the RMR induced overhead, incurred due to the
105 process of selecting an endpoint for each message, is minimal
106 and should not impact the overall latency or throughput of
107 the application. This impact has been measured with test
108 applications running on the same physical host and the
109 average latency through RMR for a message was on the order of
112 As an application's throughput increases, it becomes easy for
113 the application to overrun the underlying transport mechanism
114 (e.g. NNG), consume all available TCP transmit buffers, or
115 otherwise find itself in a situation where a send might not
116 immediately complete. RMR offers different *modes* which
117 allow the application to manage these states based on the
118 overall needs of the application. These modes are discussed
119 in the *Configuration* section of this document.
125 To use, the RMR based application simply needs to initialise
126 the RMR environment, wait for RMR to have received a routing
127 table (become ready), and then invoke either the send or
128 receive functions. These steps, and some behind the scenes
129 details, are described in the following paragraphs.
135 The RMR function ``rmr_init()`` is used to set up the RMR
136 environment and must be called before messages can be sent or
137 received. One of the few parameters that the application must
138 communicate to RMR is the port number that will be used as
139 the listen port for new connections. The port number is
140 passed on the initialisation function call and a TCP listen
141 socket will be opened with this port. If the port is already
142 in use RMR will report a failure; the application will need
143 to reinitialise with a different port number, abort, or take
144 some other action appropriate for the application.
146 In addition to creating a TCP listen port, RMR will start a
147 process thread which will be responsible for receiving
148 dynamic updates to the route table. This thread also causes a
149 TCP listen port to be opened as it is expected that the
150 process which generates route table updates will connect and
151 send new information when needed. The route table update port
152 is **not** supplied by the application, but is supplied via
153 an environment variable as this value is likely determined by
154 the mechanism which is starting and configuring the
161 On successful initialisation, a void pointer, often called a
162 *handle* by some programming languages, is returned to the
163 application. This is a reference to the RMR control
164 information and must be passed as the first parameter on most
165 RMR function calls. RMR refers to this as the context, or
172 An application which is only receiving messages does not need
173 to wait for RMR to *become ready* after the call to the
174 initialization function. However, before the application can
175 successfully send a message, RMR must have loaded a route
176 table, and the application must wait for RMR to report that
177 it has done so. The RMR function ``rmr_ready()`` will return
178 the value *true* (1) when a complete route table has been
179 loaded and can be used to determine the endpoint for a send
186 The process of receiving is fairly straight forward. The
187 application invokes the RMR ``rmr_rcv_msg()`` function which
188 will block until a message is received. The function returns
189 a pointer to a message block which provides all of the
190 details about the message. Specifically, the application has
191 access to the following information either directly or
195 * The payload (actual data)
197 * The total payload length in bytes
199 * The number of bytes of the payload which contain valid data
201 * The message type and subscription ID values
203 * The hostname and IP address of the source of the message
208 * Tracing data (if provided)
216 The message payload contains the *raw* data that was sent by
217 the peer application. The format will likely depend on the
218 message type, and is expected to be known by the application.
219 A direct pointer to the payload is available from the message
220 buffer (see appendix B for specific message buffer details).
222 Two payload-related length values are also directly
223 available: the total payload length, and the number of bytes
224 actually filled with data. The used length is set by the
225 caller, and may or not be an accurate value. The total
226 payload length is determined when the buffer is created for
227 sending, and is the maximum number of bytes that the
228 application may modify should the buffer be used to return a
232 Message Type and Subscription ID
233 --------------------------------
235 The message type and subscription ID are both directly
236 available from the message buffer, and are the values which
237 were used to by RMR in the sending application to select the
238 endpoint. If the application resends the message, as opposed
239 to returning the message buffer as a response, the message
240 number and/or the subscription ID might need to be changed to
241 avoid potential issues[1].
247 The source, or sender information, is indirectly available to
248 the application via the ``rmr_get_src()`` and
249 ``rmr_get_ip()`` functions. The former returns a string
250 containing ``hostname:port,`` while the string
251 ``ip:port`` is returned by the latter.
257 The message buffer contains a fixed length set of bytes which
258 applications can set to track related messages across the
259 application concept of a transaction. RMR will use the
260 transaction ID for matching a response message when the
261 ``rmr_call()`` function is used to send a message.
267 RMR supports the addition of an optional trace information to
268 any message. The presence and size is controlled by the
269 application, and can vary from message to message if desired.
270 The actual contents of the trace information is determined by
271 the application; RMR provides only the means to set, extract,
272 and obtain a direct reference to the trace bytes. The trace
273 data field in a message buffer is discussed in greater detail
274 in the *Trace Data* section.
280 Sending requires only slightly more work on the part of the
281 application than receiving a message. The application must
282 allocate an RMR message buffer, populate the message payload
283 with data, set the message type and length, and optionally
284 set the subscription ID. Information such as the source IP
285 address, hostname, and port are automatically added to the
286 message buffer by RMR, so there is no need for the
287 application to worry about these.
290 Message Buffer Allocation
291 -------------------------
293 The function ``rmr_msg_alloc()`` allocates a *zero copy*
294 buffer and returns a pointer to the RMR ``rmr_mbuf_t``
295 structure. The message buffer provides direct access to the
296 payload, length, message type and subscription ID fields. The
297 buffer must be preallocated in order to allow the underlying
298 transport mechanism to allocate the payload space from its
299 internal memory pool; this eliminates multiple copies as the
300 message is sent, and thus is more efficient.
302 If a message buffer has been received, and the application
303 wishes to use the buffer to send a response, or to forward
304 the buffer to another application, a new buffer does **not**
305 need to be allocated. The application may set the necessary
306 information (message type, etc.), and adjust the payload, as
307 is necessary and then pass the message buffer to
308 ``rmr_send_msg()`` or ``rmr_rts_msg()`` to be sent or
309 returned to the sender.
312 Populating the Message Buffer
313 -----------------------------
315 The application has direct access to several of the message
316 buffer fields, and should set them appropriately.
326 This is the number of bytes that the application placed into
327 the payload. Setting length to 0 is allowed, and length may
328 be less than the allocated payload size.
332 The message type that RMR will use to determine the endpoint
333 used as the target of the send.
337 The subscription ID if the message is to be routed based on
338 the combination of message type and subscription ID. If no
339 subscription ID is valid for the message, the application
340 should set the field with the RMR constant
345 The application should obtain the reference (pointer) to the
346 payload from the message buffer and place any data into the
347 payload. The application is responsible for ensuring that the
348 maximum payload size is not exceeded. The application may
349 obtain the maximum size via the ``rmr_payload_size()``
354 Optionally, the application may add trace information to the
361 Sending a Message Buffer
362 ------------------------
364 Once the application has populated the necessary bits of a
365 message, it may be sent by passing the buffer to the
366 ``rmr_send_msg()`` function. This function will select an
367 endpoint to receive the message, based on message type and
368 subscription ID, and will pass the message to the underlying
369 transport mechanism for actual transmission on the
370 connection. (Depending on the underlying transport mechanism,
371 the actual connection to the endpoint may happen at the time
372 of the first message sent to the endpoint, and thus the
373 latency of the first send might be longer than expected.)
375 On success, the send function will return a reference to a
376 message buffer; the status within that message buffer will
377 indicate what the message buffer contains. When the status is
378 ``RMR_OK`` the reference is to a **new** message buffer for
379 the application to use for the next send; the payload size is
380 the same as the payload size allocated for the message that
381 was just sent. This is a convenience as it eliminates the
382 need for the application to call the message allocation
383 function at some point in the future, and assumes the
384 application will send many messages which will require the
385 same payload dimensions.
387 If the message contains any status other than ``RMR_OK,``
388 then the message could **not** be sent, and the reference is
389 to the unsent message buffer. The value of the status will
390 indicate whether the nature of the failure was transient (
391 ``RMR_ERR_RETRY``) or not. Transient failures are likely to
392 be successful if the application attempts to send the message
393 at a later time. Unfortunately, it is impossible for RMR to
394 know the exact transient failure (e.g. connection being
395 established, or TCP buffer shortage), and thus it is not
396 possible to communicate how long the application should wait
397 before attempting to resend, if the application wishes to
398 resend the message. (More discussion with respect to message
399 retries can be found in the *Handling Failures* section.)
405 Several forms of usage fall into a more advanced category and
406 are described in the following sections. These include
407 blocking call, return to sender and wormhole functions.
413 The RMR function ``rmr_call()`` sends a message in the exact
414 same manner as the ``rmr_send_msg()()`` function, with the
415 endpoint selection based on the message key. But unlike the
416 send function, ``rmr_call()`` will block and wait for a
417 response from the application that is selected to receive the
418 message. The matching message is determined by the
419 transaction ID which the application must place into the
420 message buffer prior to invoking ``rmr_call()``. Similarly,
421 the responding application must ensure that the same
422 transaction ID is placed into the message buffer before
423 returning its response.
425 The return from the call is a message buffer with the
426 response message; there is no difference between a message
427 buffer returned by the receive function and one returned by
428 the ``rmr_call()`` function. If a response is not received in
429 a reasonable amount of time, a nil message buffer is returned
430 to the calling application.
436 Because of the nature of RMR's routing policies, it is
437 generally not possible for an application to control exactly
438 which endpoint is sent a message. There are cases, such as
439 responding to a message delivered via ``rmr_call()`` that the
440 application must send a message and guarantee that RMR routes
441 it to an exact destination. To enable this, RMR provides the
442 ``rmr_rts_msg(),`` return to sender, function. Upon receipt
443 of any message, an application may alter the payload, and if
444 necessary the message type and subscription ID, and pass the
445 altered message buffer to the ``rmr_rts_msg()`` function to
446 return the altered message to the application which sent it.
447 When this function is used, RMR will examine the message
448 buffer for the source information and use that to select the
449 connection on which to write the response.
455 The basic call mechanism described above is **not** thread
456 safe, as it is not possible to guarantee that a response
457 message is delivered to the correct thread. The RMR function
458 ``rmr_mt_call()`` accepts an additional parameter which
459 identifies the calling thread in order to ensure that the
460 response is delivered properly. In addition, the application
461 must specifically initialise the multi-threaded call
462 environment by passing the ``RMRFL_MTCALL`` flag as an option
463 to the ``rmr_init()`` function.
465 One advantage of the multi-threaded call capability in RMR is
466 the fact that only the calling thread is blocked. Messages
467 received which are not responses to the call are continued to
468 be delivered via normal ``rmr_rcv_msg()`` calls.
470 While the process is blocked waiting for the response, it is
471 entirely possible that asynchronous, non-matching, messages
472 will arrive. When this happens, RMR will queues the messages
473 and return them to the application over the next calls to
480 As was mentioned earlier, the design of RMR is to eliminate
481 the need for an application to know a specific endpoint, even
482 when a response message is being sent. In some rare cases it
483 may be necessary for an application to establish a direct
484 connection to an RMR-based application rather than relying on
485 message type and subscription ID based routing. The
486 *wormhole* functions provide an application with the ability
487 to create a direct connection and then to send and receive
488 messages across the connection. The following are the RMR
489 functions which provide wormhole communications:
499 Open a connection to an endpoint. Name or IP address and port
500 of the endpoint is supplied. Returns a wormhole ID that the
501 application must use when sending a direct message.
503 * - **rmr_wh_send_msg**
505 Sends an RMR message buffer to the connected application. The
506 message type and subscription ID may be set in the message,
507 but RMR will ignore both.
511 Closes the direct connection.
520 The vast majority of states reported by RMR are fatal; if
521 encountered during setup or initialization, then it is
522 unlikely that any message oriented processing should
523 continue, and when encountered on a message operation
524 continued operation on that message should be abandoned.
525 Specifically with regard to message sending, it is very
526 likely that the underlying transport mechanism will report a
527 *soft,* or transient, failure which might be successful if
528 the operation is retried at a later point in time. The
529 paragraphs below discuss the methods that an application
530 might deal with these soft failures.
536 When a soft failure is reported, the returned message buffer
537 returned by the RMR function will be ``RMR_ERR_RETRY.`` These
538 types of failures can occur for various reasons; one of two
539 reasons is typically the underlying cause:
542 * The session to the targeted recipient (endpoint) is not
545 * The transport mechanism buffer pool is full and cannot
546 accept another buffer.
550 Unfortunately, it is not possible for RMR to determine which
551 of these two cases is occurring, and equally as unfortunate
552 the time to resolve each is different. The first, no
553 connection, may require up to a second before a message can
554 be accepted, while a rejection because of buffer shortage is
555 likely to resolve in less than a millisecond.
561 The action which an application takes when a soft failure is
562 reported ultimately depends on the nature of the application
563 with respect to factors such as tolerance to extended message
564 latency, dropped messages, and over all message rate.
570 In an effort to reduce the workload of an application
571 developer, RMR has a default retry policy such that RMR will
572 attempt to retransmit a message up to 1000 times when a soft
573 failure is reported. These retries generally take less than 1
574 millisecond (if all 1000 are attempted) and in most cases
575 eliminates nearly all reported soft failures to the
576 application. When using this mode, it might allow the
577 application to simply treat all bad return values from a send
578 attempt as permanent failures.
580 If an application is so sensitive to any delay in RMR, or the
581 underlying transport mechanism, it is possible to set RMR to
582 return a failure immediately on any kind of error (permanent
583 failures are always reported without retry). In this mode,
584 RMR will still set the state in the message buffer to
585 ``RMR_ERR_RETRY,`` but will **not** make any attempts to
586 resend the message. This zero-retry policy is enabled by
587 invoking the ``rmr_set_stimeout()`` with a value of 0; this
588 can be done once immediately after ``rmr_init()`` is invoked.
590 Regardless of the retry mode which the application sets, it
591 will ultimately be up to the application to handle failures
592 by queuing the message internally for resend, retrying
593 immediately, or dropping the send attempt all together. As
594 stated before, only the application can determine how to best
595 handle send failures.
601 RMR will return the state of processing for message based
602 operations (send/receive) as the status in the message
603 buffer. For non-message operations, state is returned to the
604 caller as the integer return value for all functions which
605 are not expected to return a pointer (e.g.
606 ``rmr_init()``.) The following are the RMR state constants
607 and a brief description of their meaning.
617 state is good; operation finished successfully
619 * - **RMR_ERR_BADARG**
621 argument passed to function was unusable
623 * - **RMR_ERR_NOENDPT**
625 send/call could not find an endpoint based on msg type
627 * - **RMR_ERR_EMPTY**
629 msg received had no payload; attempt to send an empty message
631 * - **RMR_ERR_NOHDR**
633 message didn't contain a valid header
635 * - **RMR_ERR_SENDFAILED**
637 send failed; errno may contain the transport provider reason
639 * - **RMR_ERR_CALLFAILED**
641 unable to send the message for a call function; errno may
642 contain the transport provider reason
644 * - **RMR_ERR_NOWHOPEN**
646 no wormholes are open
650 the wormhole id provided was invalid
652 * - **RMR_ERR_OVERFLOW**
654 operation would have busted through a buffer/field size
656 * - **RMR_ERR_RETRY**
658 request (send/call/rts) failed, but caller should retry
659 (EAGAIN for wrappers)
661 * - **RMR_ERR_RCVFAILED**
663 receive failed (hard error)
665 * - **RMR_ERR_TIMEOUT**
667 response message not received in a reasonable amount of time
669 * - **RMR_ERR_UNSET**
671 the message hasn't been populated with a transport buffer
673 * - **RMR_ERR_TRUNC**
675 length in the received buffer is longer than the size of the
676 allocated payload, received message likely truncated (length
677 set by sender could be wrong, but we can't know that)
679 * - **RMR_ERR_INITFAILED**
681 initialisation of something (probably message) failed
683 * - **RMR_ERR_NOTSUPP**
685 the request is not supported, or RMR was not initialised for
690 Depending on the underlying transport mechanism, and the
691 nature of the call that RMR attempted, the system
692 ``errno`` value might reflect additional detail about the
693 failure. Applications should **not** rely on errno as some
694 transport mechanisms do not set it with any consistency.
697 Configuration and Control
698 =========================
700 With the assumption that most RMR based applications will be
701 executed in a containerised environment, there are some
702 underlying mechanics which the developer may need to know in
703 order to properly provide a configuration specification to
704 the container management system. The following paragraphs
705 briefly discuss these.
712 RMR requires two (2) TCP listen ports: one for general
713 application-to-application communications and one for
714 route-table updates. The general communication port is
715 specified by the application at the time RMR is initialised.
716 The port used to listen for route table updates is likely to
717 be a constant port shared by all applications provided they
718 are running in separate containers. To that end, the port
719 number defaults to 4561, but can be configured with an
720 environment variable (see later paragraph in this section).
726 RMR is typically host name agnostic. Route table entries may
727 contain endpoints defined either by host name or IP address.
728 In the container world the concept of a *service name* might
729 exist, and likely is different than a host name. RMR's only
730 requirement with respect to host names is that a name used on
731 a route table entry must be resolvable via the
732 ``gethostbyname`` system call.
735 Environment Variables
736 ---------------------
738 Several environment variables are recognised by RMR which, in
739 general, are used to define interfaces and listen ports (e.g.
740 the route table update listen port), or debugging
741 information. Generally this information is system controlled
742 and thus RMR expects this information to be defined in the
743 environment rather than provided by the application. The
744 following is a list of the environment variables which RMR
753 * - **RMR_ASYNC_CONN**
755 Allows the async connection mode to be turned off (by setting
756 the value to 0). When set to 1, or missing from the
757 environment, RMR will invoke the connection interface in the
758 transport mechanism using the non-blocking (async) mode. This
759 will likely result in many "soft failures" (retry) until the
760 connection is established, but allows the application to
761 continue unimpeded should the connection be slow to set up.
765 This provides the interface that RMR will bind listen ports
766 to, allowing for a single interface to be used rather than
767 listening across all interfaces. This should be the IP
768 address assigned to the interface that RMR should listen on,
769 and if not defined RMR will listen on all interfaces.
773 This variable defines the port that RMR should open for
774 communications with Route Manager, and other RMR control
775 applications. If not defined, the port 4561 is assumed.
777 Previously, the ``RMR_RTG_SVC`` (route table generator
778 service port) was used to define this port. However, a future
779 version of Route Manager will require RMR to connect and
780 request tables, thus that variable is now used to supply the
781 Route Manager's well-known address and port.
783 To maintain backwards compatibility with the older Route
784 Manager versions, the presence of this variable in the
785 environment will shift RMR's behaviour with respect to the
786 default value used when ``RMR_RTG_SVC`` is **not** defined.
788 When ``RMR_CTL_PORT`` is **defined:** RMR assumes that Route
789 Manager requires RMR to connect and request table updates is
790 made, and the default well-known address for Route manager is
791 used (routemgr:4561).
793 When ``RMR_CTL_PORT`` is **undefined:** RMR assumes that
794 Route Manager will connect and push table updates, thus the
795 default listen port (4561) is used.
797 To avoid any possible misinterpretation and/or incorrect
798 assumptions on the part of RMR, it is recommended that both
799 the ``RMR_CTL_PORT`` and ``RMR_RTG_SVC`` be defined. In the
800 case where both variables are defined, RMR will behave
801 exactly as is communicated with the variable's values.
803 * - **RMR_RTREQ_FREQ**
805 When RMR needs a new route table it will send a request once
806 every ``n`` seconds. The default value for ``n`` is 5, but
807 can be changed if this variable is set prior to invoking the
808 process. Accepted values are between 1 and 300 inclusive.
812 The value of this variable depends on the Route Manager in
815 When the Route Manager is expecting to connect to an xAPP and
816 push route tables, this variable must indicate the
817 ``port`` which RMR should use to listen for these
820 When the Route Manager is expecting RMR to connect and
821 request a table update during initialisation, the variable
822 should be the ``host`` of the Route Manager process.
824 The ``RMR_CTL_PORT`` variable (added with the support of
825 sending table update requests to Route manager), controls the
826 behaviour if this variable is not set. See the description of
827 that variable for details.
831 By default RMR writes messages to standard error (incorrectly
832 referred to as log messages) in human readable format. If
833 this environment variable is set to 0, the format of standard
834 error messages might be written in some format not easily
835 read by humans. If missing, a value of 1 is assumed.
837 * - **RMR_LOG_VLEVEL**
839 This is a numeric value which corresponds to the verbosity
840 level used to limit messages written to standard error. The
841 lower the number the less chatty RMR functions are during
842 execution. The following is the current relationship between
843 the value set on this variable and the messages written:
853 Off; no messages of any sort are written.
857 Only critical messages are written (default if this variable
862 Errors and all messages written with a lower value.
866 Warnings and all messages written with a lower value.
870 Informational and all messages written with a lower value.
874 Debugging mode -- all messages written, however this requires
875 RMR to have been compiled with debugging support enabled.
879 * - **RMR_RTG_ISRAW**
881 **Deprecated.** Should be set to 1 if the route table
882 generator is sending "plain" messages (not using RMR to send
883 messages), 0 if the RTG is using RMR to send. The default is
884 1 as we don't expect the RTG to use RMR.
886 This variable is only recognised when using the NNG transport
887 library as it is not possible to support NNG "raw"
888 communications with other transport libraries. It is also
889 necessary to match the value of this variable with the
890 capabilities of the Route Manager; at some point in the
891 future RMR will assume that all Route Manager messages will
892 arrive via an RMR connection and will ignore this variable.
896 This is used to supply a static route table which can be used
897 for debugging, testing, or if no route table generator
898 process is being used to supply the route table. If not
899 defined, no static table is used and RMR will not report
900 *ready* until a table is received. The static route table may
901 contain both the route table (between newrt start and end
902 records), and the MEID map (between meid_map start and end
907 This is either the name or IP address which is placed into
908 outbound messages as the message source. This will used when
909 an RMR based application uses the rmr_rts_msg() function to
910 return a response to the sender. If not supplied RMR will use
911 the hostname which in some container environments might not
914 The value of this variable is also used for Route Manager
915 messages which are sent via an RMR connection.
919 Names the file where RMR should write the latest update it
920 receives from the source of route tables (generally Route
921 Manager). This is meant to assist with debugging and/or
922 troubleshooting when it is suspected that route information
923 isn't being sent and/or received correctly. If this variable
924 is not given, RMR will save the last update using the
925 ``RMR_SEED_RT`` variable value and adding a ``.stash`` suffix
926 to the filename so as not to overwrite the static table.
928 * - **RMR_VCTL_FILE**
930 This supplies the name of a verbosity control file. The core
931 RMR functions do not produce messages unless there is a
932 critical failure. However, the route table collection thread,
933 not a part of the main message processing component, can
934 write additional messages to standard error. If this variable
935 is set, RMR will extract the verbosity level for these
936 messages (0 is silent) from the first line of the file.
937 Changes to the file are detected and thus the level can be
938 changed dynamically, however RMR will only suss out this
939 variable during initialisation, so it is impossible to enable
940 verbosity after startup.
944 If set to 1, RMR will write some warnings which are
945 non-performance impacting. If the variable is not defined, or
946 set to 0, RMR will not write these additional warnings.
950 There are other, non-RMR, variables which may exist and are
951 used by RMR. These variable names are not under the control
952 of RMR, so they are subject to change without potentiallyb
953 being reflected in either RMR's code, or this document. The
954 following is a list of these environment variables.
962 * - **ALARM_MANAGER_SERVICE_NAME**
964 This is the DNS name, or IP address, of the process which is
965 listening for RMR alarm messages. If this variable is
966 missing, ``service-ricplt-alarmmanager-rmr`` is assumed.
968 * - **ALARM_MANAGER_SERVICE_PORT**
970 This is the port that the alarm manager is using to accept
971 RMR messages. If the environment variable is missing the
972 value ``4560`` is assumed.
980 As with nearly all UNIX libraries, errors, warnings and
981 informational messages are written in plain text to the
982 standard error device (stderr). All RMR messages are prefixed
983 with the current time (in milliseconds past the standard UNIX
984 epoch), the process ID, and a severity indicator. RMR
985 messages are written with one of three severity strings:
995 The event is of a critical nature and it is unlikely that RMR
996 will continue to operate correctly if at all. It is almost
997 certain that immediate action will be needed to resolve the
1002 The event is not expected and RMR is not able to handle it.
1003 There is a small chance that continued operation will be
1004 negatively impacted. Eventual action to diagnose and correct
1005 the issue will be necessary.
1009 The event was not expected by RMR, but can be worked round.
1010 Normal operation will continue, but it is recommended that
1011 the cause of the problem be investigated.
1017 Log message supression
1018 ----------------------
1020 For the most part, the *fast path* code in RMR does no
1021 logging; even when messages are squelched, there is a
1022 non-zero cosst to check for the setting each time a potential
1023 message is to be written. To that end, RMRM will log only
1024 severe errors once initialisation has completed. An exception
1025 to this policy exists in the route table collection thread.
1026 The thread of execution which collects route table updates
1027 does not need to be concerned with performance, and as such
1028 has the potential to log its actions in a very verbose
1029 manner. The environment variable `` RMR_VCTL_FILE `` can be
1030 used to define a file where the desired verbosity level (0 to
1031 4 where 0 is off) can be placed. If the environment variable
1032 is not set when the process starts, RMR will assume that the
1033 file ``/tmp/rmr.v`` will be used. Beginning with version
1034 4.6.0 this file does **not** need to exist when the process
1035 is started. To change the verbosity level, the desired value
1036 is written to the file on the first line.
1042 The route table colleciton thread is also responsible for
1043 watching for situations which need to be reported as alarms
1044 to the platform's alarm management service. When a state
1045 exists RMR will create and send an alarm (via RMR message) to
1046 the alarm service, and will send a *clear* message when the
1047 state no longer exists. Currently RMR will alarm only when
1048 the application is not removing messages from the receive
1049 ring quicklye enough causing RMR to drop messages as they are
1057 [1] It is entirely possible to design a routing table, and
1058 application group, such that the same message type is is
1059 left unchanged and the message is forwarded by an
1060 application after updating the payload. This type of
1061 behaviour is often referred to as service chaining, and can
1062 be done without any "knowledge" by an application with
1063 respect to where the message goes next. Service chaining is
1064 supported by RMR in as much as it allows the message to be
1065 resent, but the actual complexities of designing and
1066 implementing service chaining lie with the route table
1074 Appendix A -- Quick Reference
1075 =============================
1077 Please refer to the RMR manual pages on the Read the Docs
1080 https://docs.o-ran-sc.org/projects/o-ran-sc-ric-plt-lib-rmr/en/latest/index.html
1084 Appendix B -- Message Buffer Details
1085 ====================================
1087 The RMR message buffer is a C structure which is exposed in
1088 the ``rmr.h`` header file. It is used to manage a message
1089 received from a peer endpoint, or a message that is being
1090 sent to a peer. Fields include payload length, amount of
1091 payload actually used, status, and a reference to the
1092 payload. There are also fields which the application should
1093 ignore, and could be hidden in the header file, but we chose
1094 not to. These fields include a reference to the RMR header
1095 information, and to the underlying transport mechanism
1096 message struct which may or may not be the same as the RMR
1103 The following is the C structure. Readers are cautioned to
1104 examine the ``rmr.h`` header file directly; the information
1105 here may be out of date (old document in some cache), and
1106 thus it may be incorrect.
1113 int state; // state of processing
1114 int mtype; // message type
1115 int len; // length of data in the payload (send or received)
1116 unsigned char* payload; // transported data
1117 unsigned char* xaction; // pointer to fixed length transaction id bytes
1118 int sub_id; // subscription id
1119 int tp_state; // transport state (errno)
1121 // these things are off limits to the user application
1122 void* tp_buf; // underlying transport allocated pointer (e.g. nng message)
1123 void* header; // internal message header (whole buffer: header+payload)
1124 unsigned char* id; // if we need an ID in the message separate from the xaction id
1125 int flags; // various MFL_ (private) flags as needed
1126 int alloc_len; // the length of the allocated space (hdr+payload)
1127 void* ring; // ring this buffer should be queued back to
1128 int rts_fd; // SI fd for return to sender
1129 int cookie; // cookie to detect user misuse of free'd msg
1135 State vs Transport State
1136 ------------------------
1138 The state field reflects the state at the time the message
1139 buffer is returned to the calling application. For a send
1140 operation, if the state is not ``RMR_OK`` then the message
1141 buffer references the payload that could not be sent, and
1142 when the state is ``RMR_OK`` the buffer references a *fresh*
1143 payload that the application may fill in.
1145 When the state is not ``RMR_OK,`` C programmes may examine
1146 the global ``errno`` value which RMR will have left set, if
1147 it was set, by the underlying transport mechanism. In some
1148 cases, wrapper modules are not able to directly access the
1149 C-library ``errno`` value, and to assist with possible
1150 transport error details, the send and receive operations
1151 populate ``tp_state`` with the value of ``errno.``
1153 Regardless of whether the application makes use of the
1154 ``tp_state,`` or the ``errno`` value, it should be noted that
1155 the underlying transport mechanism may not actually update
1156 the errno value; in other words: it might not be accurate. In
1157 addition, RMR populates the ``tp_state`` value in the message
1158 buffer **only** when the state is not ``RMR_OK.``
1164 The transaction field was exposed in the first version of
1165 RMR, and in hindsight this shouldn't have been done. Rather
1166 than break any existing code the reference was left, but
1167 additional fields such as trace data, were not directly
1168 exposed to the application. The application developer is
1169 strongly encouraged to use the functions which get and set
1170 the transaction ID rather than using the pointer directly;
1171 any data overruns will not be detected if the reference is
1174 In contrast, the payload reference should be used directly by
1175 the application in the interest of speed and ease of
1176 programming. The same care to prevent writing more bytes to
1177 the payload buffer than it can hold must be taken by the
1178 application. By the nature of the allocation of the payload
1179 in transport space, RMR is unable to add guard bytes and/or
1180 test for data overrun.
1186 When RMR sends the application's message, the message buffer
1187 is **not** transmitted. The transport buffer (tp_buf) which
1188 contains the RMR header and application payload is the only
1189 set of bytes which are transmitted. While it may seem to the
1190 caller like the function ``rmr_send_msg()`` is returning a
1191 new message buffer, the same struct is reused and only a new
1192 transport buffer is allocated. The intent is to keep the
1193 alloc/free cycles to a minimum.
1197 Appendix C -- Glossary
1198 ======================
1200 Many terms in networking can be interpreted with multiple
1201 meanings, and several terms used in various RMR documentation
1202 are RMR specific. The following definitions are the meanings
1203 of terms used within RMR documentation and should help the
1204 reader to understand the intent of meaning.
1213 A programme which uses RMR to send and/or receive messages
1214 to/from another RMR based application.
1216 * - **Critical error**
1218 An error that RMR has encountered which will prevent further
1219 successful processing by RMR. Critical errors usually
1220 indicate that the application should abort.
1224 An RMR based application that is defined as being capable of
1225 receiving one or more types of messages (as defined by a
1228 * - **Environment variable**
1230 A key/value pair which is set externally to the application,
1231 but which is available to the application (and referenced
1232 libraries) through the ``getenv`` system call. Environment
1233 variables are the main method of communicating information
1234 such as port numbers to RMR.
1238 An abnormal condition that RMR has encountered, but will not
1239 affect the overall processing by RMR, but may impact certain
1240 aspects such as the ability to communicate with a specific
1241 endpoint. Errors generally indicate that something, usually
1242 external to RMR, must be addressed.
1246 The name of the host as returned by the ``gethostbyname``
1247 system call. In a containerised environment this might be the
1248 container or service name depending on how the container is
1249 started. From RMR's point of view, a host name can be used to
1250 resolve an *endpoint* definition in a *route* table.)
1254 Internet protocol. A low level transmission protocol which
1255 governs the transmission of datagrams across network
1258 * - **Listen socket**
1260 A *TCP* socket used to await incoming connection requests.
1261 Listen sockets are defined by an interface and port number
1262 combination where the port number is unique for the
1267 A series of bytes transmitted from the application to another
1268 RMR based application. A message is comprised of RMR specific
1269 data (a header), and application data (a payload).
1271 * - **Message buffer**
1273 A data structure used to describe a message which is to be
1274 sent or has been received. The message buffer includes the
1275 payload length, message type, message source, and other
1278 * - **Message type**
1280 A signed integer (0-32000) which identifies the type of
1281 message being transmitted, and is one of the two components
1282 of a *routing key.* See *Subscription ID.*
1286 The portion of a message which holds the user data to be
1287 transmitted to the remote *endpoint.* The payload contents
1288 are completely application defined.
1292 A set of information which defines the current state of the
1293 underlying transport connections that RMR is managing. The
1294 application will be give a context reference (pointer) that
1295 is supplied to most RMR functions as the first parameter.
1299 The method of selecting an *endpoint* from a list such that
1300 all *endpoints* are selected before starting at the head of
1305 A series of "rules" which define the possible *endpoints* for
1308 * - **Route table manager**
1310 An application responsible for building a *route table* and
1311 then distributing it to all applicable RMR based
1316 The process of selecting an *endpoint* which will be the
1317 recipient of a message.
1321 A combination of *message type* and *subscription ID* which
1322 RMR uses to select the destination *endpoint* when sending a
1327 The sender of a message.
1329 * - **Subscription ID**
1331 A signed integer value (0-32000) which identifies the
1332 subscription characteristic of a message. It is used in
1333 conjunction with the *message type* to determine the *routing
1338 The *endpoint* selected to receive a message.
1342 Transmission Control Protocol. A connection based internet
1343 protocol which provides for lossless packet transportation,
1348 Also called a *process thread, or pthread.* This is a
1349 lightweight process which executes in concurrently with the
1350 application and shares the same address space. RMR uses
1351 threads to manage asynchronous functions such as route table
1354 * - **Trace information**
1356 An optional portion of the message buffer that the
1357 application may populate with data that allows for tracing
1358 the progress of the transaction or application activity
1359 across components. RMR makes no use of this data.
1361 * - **Transaction ID**
1363 A fixed number of bytes in the *message* buffer) which the
1364 application may populate with information related to the
1365 transaction. RMR makes use of the transaction ID for matching
1366 response messages with the &c function is used to send a
1369 * - **Transient failure**
1371 An error state that is believed to be short lived and that
1372 the operation, if retried by the application, might be
1373 successful. C programmers will recognise this as
1378 A warning occurs when RMR has encountered something that it
1379 believes isn't correct, but has a defined work round.
1383 A direct connection managed by RMR between the user
1384 application and a remote, RMR based, application.
1390 Appendix D -- Code Examples
1391 ===========================
1393 The following snippet of code illustrate some of the basic
1394 operation of the RMR library. Please refer to the examples
1395 and test directories in the RMR repository for complete RMR
1402 The following code segment shows how a message buffer can be
1403 allocated, populated, and sent. The snippet also illustrates
1404 how the result from the ``rmr_send_msg()`` function is used
1405 to send the next message. It does not illustrate error and/or
1417 #include <sys/epoll.h>
1420 #include <rmr/rmr.h>
1422 int main( int argc, char** argv ) {
1423 void* mrc; // msg router context
1424 struct epoll_event events[1]; // list of events to give to epoll
1425 struct epoll_event epe; // event definition for event to listen to
1426 int ep_fd = -1; // epoll's file des (given to epoll_wait)
1427 int rcv_fd; // file des for epoll checks
1428 int nready; // number of events ready for receive
1429 rmr_mbuf_t* sbuf; // send buffer
1430 rmr_mbuf_t* rbuf; // received buffer
1433 char* listen_port = "43086";
1434 int delay = 1000000; // mu-sec delay between messages
1436 int stats_freq = 100;
1438 if( argc > 1 ) { // simplistic arg picking
1439 listen_port = argv[1];
1442 delay = atoi( argv[2] );
1445 mtype = atoi( argv[3] );
1448 fprintf( stderr, "<DEMO> listen port: %s; mtype: %d; delay: %d\\n",
1449 listen_port, mtype, delay );
1451 if( (mrc = rmr_init( listen_port, 1400, RMRFL_NONE )) == NULL ) {
1452 fprintf( stderr, "<DEMO> unable to initialise RMR\\n" );
1456 rcv_fd = rmr_get_rcvfd( mrc ); // set up epoll things, start by getting the FD from RMR
1458 fprintf( stderr, "<DEMO> unable to set up polling fd\\n" );
1461 if( (ep_fd = epoll_create1( 0 )) < 0 ) {
1462 fprintf( stderr, "[FAIL] unable to create epoll fd: %d\\n", errno );
1465 epe.events = EPOLLIN;
1466 epe.data.fd = rcv_fd;
1468 if( epoll_ctl( ep_fd, EPOLL_CTL_ADD, rcv_fd, &epe ) != 0 ) {
1469 fprintf( stderr, "[FAIL] epoll_ctl status not 0 : %s\\n", strerror( errno ) );
1473 sbuf = rmr_alloc_msg( mrc, 256 ); // alloc 1st send buf; subsequent bufs alloc on send
1474 rbuf = NULL; // don't need to alloc receive buffer
1476 while( ! rmr_ready( mrc ) ) { // must have route table
1477 sleep( 1 ); // wait til we get one
1479 fprintf( stderr, "<DEMO> rmr is ready\\n" );
1482 while( 1 ) { // send messages until the cows come home
1483 snprintf( sbuf->payload, 200,
1484 "count=%d received= %d ts=%lld %d stand up and cheer!", // create the payload
1485 count, rcvd_count, (long long) time( NULL ), rand() );
1487 sbuf->mtype = mtype; // fill in the message bits
1488 sbuf->len = strlen( sbuf->payload ) + 1; // send full ascii-z string
1490 sbuf = rmr_send_msg( mrc, sbuf ); // send & get next buf to fill in
1491 while( sbuf->state == RMR_ERR_RETRY ) { // soft failure (device busy?) retry
1492 sbuf = rmr_send_msg( mrc, sbuf ); // w/ simple spin that doesn't give up
1496 // check to see if anything was received and pull all messages in
1497 while( (nready = epoll_wait( ep_fd, events, 1, 0 )) > 0 ) { // 0 is non-blocking
1498 if( events[0].data.fd == rcv_fd ) { // waiting on 1 thing, so [0] is ok
1500 rbuf = rmr_rcv_msg( mrc, rbuf ); // receive and ignore; just count
1507 if( (count % stats_freq) == 0 ) { // occasional stats out to tty
1508 fprintf( stderr, "<DEMO> sent %d received %d\\n", count, rcvd_count );
1521 The receiver code is even simpler than the sender code as it
1522 does not need to wait for a route table to arrive (only
1523 senders need to do that), nor does it need to allocate an
1524 initial buffer. The example assumes that the sender is
1525 transmitting a zero terminated string as the payload.
1537 #include <rmr/rmr.h>
1540 int main( int argc, char** argv ) {
1541 void* mrc; // msg router context
1542 long long total = 0;
1543 rmr_mbuf_t* msg = NULL; // message received
1544 int stat_freq = 10; // write stats after reciving this many messages
1546 char* listen_port = "4560"; // default to what has become the standard RMR port
1547 long long count = 0;
1549 long long empty = 0;
1552 listen_port = argv[1];
1555 stat_freq = atoi( argv[2] );
1557 fprintf( stderr, "<DEMO> listening on port: %s\\n", listen_port );
1558 fprintf( stderr, "<DEMO> stats will be reported every %d messages\\n", stat_freq );
1560 mrc = rmr_init( listen_port, RMR_MAX_RCV_BYTES, RMRFL_NONE );
1562 fprintf( stderr, "<DEMO> ABORT: unable to initialise RMr\\n" );
1566 while( ! rmr_ready( mrc ) ) { // wait for RMR to get a route table
1567 fprintf( stderr, "<DEMO> waiting for ready\\n" );
1570 fprintf( stderr, "<DEMO> rmr now shows ready\\n" );
1572 while( 1 ) { // receive until killed
1573 msg = rmr_rcv_msg( mrc, msg ); // block until one arrives
1576 if( msg->state == RMR_OK ) {
1577 count++; // nothing fancy, just count
1585 if( (count % stat_freq) == 0 ) {
1586 fprintf( stderr, "<DEMO> total received: %lld; errors: %lld; empty: %lld\\n",
1587 count, bad, empty );
1595 Receive and Send Sample
1596 -----------------------
1598 The following code snippet receives messages and responds to
1599 the sender if the message type is odd. The code illustrates
1600 how the received message may be used to return a message to
1601 the source. Variable type definitions are omitted for clarity
1602 and should be obvious.
1604 It should also be noted that things like the message type
1605 which id returned to the sender (99) is a random value that
1606 these applications would have agreed on in advance and is
1607 **not** an RMR definition.
1612 mrc = rmr_init( listen_port, MAX_BUF_SZ, RMRFL_NOFLAGS );
1613 rmr_set_stimeout( mrc, 1 ); // allow RMR to retry failed sends for ~1ms
1615 while( ! rmr_ready( mrc ) ) { // we send, therefore we need a route table
1619 mbuf = NULL; // ensure our buffer pointer is nil for 1st call
1622 mbuf = rmr_rcv_msg( mrc, mbuf ); // wait for message
1624 if( mbuf == NULL || mbuf->state != RMR_OK ) {
1628 if( mbuf->mtype % 2 ) { // respond to odd message types
1629 plen = rmr_payload_size( mbuf ); // max size
1631 // reset necessary fields in msg
1632 mbuf->mtype = 99; // response type
1633 mbuf->sub_id = RMR_VOID_SUBID; // we turn subid off
1634 mbuf->len = snprintf( mbuf->payload, plen, "pong: %s", get_info() );
1636 mbuf = rmr_rts_msg( mrc, mbuf ); // return to sender
1637 if( mbuf == NULL || mbuf->state != RMR_OK ) {
1638 fprintf( stderr, "return to sender failed\\n" );
1643 fprintf( stderr, "abort: receive failure\\n" );
Code Review / ric-plt / lib / rmr.git / blob