2 .. Copyright (c) 2019 AT&T Intellectual Property.
3 .. Copyright (c) 2019 Nokia.
5 .. Licensed under the Creative Commons Attribution 4.0 International
6 .. Public License (the "License"); you may not use this file except
7 .. in compliance with the License. You may obtain a copy of the License at
9 .. https://creativecommons.org/licenses/by/4.0/
11 .. Unless required by applicable law or agreed to in writing, documentation
12 .. distributed under the License is distributed on an "AS IS" BASIS,
13 .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15 .. See the License for the specific language governing permissions and
16 .. limitations under the License.
28 Subscription Manager is a basic platform service in RIC. It is responsible for managing E2 subscriptions from xApps to the
29 E2 Node (eNodeB or gNodeB).
31 xApp can make subscriptions to E2 Node through Subscription Manager. xApp can subscribe REPORT, INSERT, CONTROL and POLICY type services
32 from E2 Node. For some of those subscription types E2 Node sends E2 Indication messages back to xApp. For those messages there need to be
33 created route between E2 Termination and xApp. Subscription Manager manages E2 subscriptions and message request routing of the subscribed
34 messages from Routing Manager. If one xApp has already made a subscription and then another xApp initiates identical subscription, Subscription
35 Manager does not forward the subscription to E2 Node but merges the subscriptions internally and request Routing Manager to create route also
36 for the second xApp and sends response to xApp. xApp can indicate in the request should Subscription Manager send request for Routing Manager
37 to create route for the E2 subscription in the request.
39 Interface between xApp and Subscription Manager is HTTP based REST interface. Interface codes are generated with help of Swagger from a
40 yaml definition file. REST interface is used also between Subscription Manager and Routing Manager. Subscription Manager and E2 Termination
41 interface is based on RMR messages. xApp should use also Swagger generated code when it implements subscription REST interface towards Subscription
44 .. image:: images/PlaceInRICSoftwareArchitecture.png
46 :alt: Place in RIC's software architecture picture
49 One xApp generated REST subscription request message can contain multiple E2 subscriptions. For every E2 subscription in the message there must be
50 also xApp generated xApp instance id. In E2 interface there can be only one ongoing E2 subscription or subscription delete procedure towards
51 E2 Node at any time. That is because Subscription Manager is able to merge new E2 subscriptions only which those it has already received
52 successful response from E2 Node. E2 subscriptions and delete subscriptions may be therefore queued for a moment in Subscription Manager.
54 Subscription Manager may need to do reties towards E2 Node during subscribe or subscription delete procedure. Retries will increase completion
55 time of the procedure. This needs to be considered in xApp's implementation. Subscription Manager sends REST notification to xApp for every
56 completed E2 subscription procedure. That is regardless was the E2 subscription successful or not. Notification is not sent for E2 subscription delete
57 procedures. Subscription Manager allocates globally unique REST request id for each new REST subscription request. That is returned to xApp in
58 response. When xApp wants to delete REST subscription, xApp need to use the same id in deletion request.
60 Subscription Manager allocates also unique id for every E2 subscriptions (E2 instance id). The id called 'InstanceId' in E2 specification.
61 In successful case the REST notification contains the id generated for the REST request, xApp instance id and E2 instance id. From xApp point
62 of view xApp instance id identifies response REST notification for the E2 subscription in the REST request. REST notification contains also Subscription
63 Manager generated E2 instance id. xApp can use that to map received E2 Indication message to E2 subscription. If E2 subscription procedure is unsuccessful
64 then E2 instance id is 0 and the notification contains non-zero error cause string and source of that error information.
66 xApp need to be able preserve Subscription Manager allocated REST request id over xApp restart. The id is needed for deletion of the REST
67 subscription and if there is need to resend the same REST request.
69 Three different type subscriptions are supported. REPORT, POLICY and INSERT. REPORT and INSERT works the same way from subscription point of view.
70 REPORT and INSERT type REST subscription request can contain multiple E2 subscriptions. POLICY subscription can also contain multiple E2 subscriptions but
71 using it in that way may not be feasible. REPORT, POLICY and INSERT type subscriptions in the same REST request are not supported supported in Subscription Manager.
73 REPORT and INSERT type subscription can contain multiple E2 subscriptions. If xApp needs to resend the same REST request, the request must contain Subscription
74 Manager allocated id for the REST request, which was returned to xApp when the request was sent first time. The request must also contain the same content as
75 first time. Reason for xApp to resend the same request could be timeout or some failure which is not permanent like E2 interface connection break or xApp restart.
76 In retry case Subscription Manager retries the E2 subscriptions towards to E2 Node in the request message which it does not have record. For subscriptions
77 which record is found Subscription Manager returns successful REST notification to xApp without forwarding request to E2 Node. One REST Subscription request can
78 contain E2 subscriptions only for one E2 Node.
80 Subscription Manager indicates to xApp in response notification if there has happened any error or timeout. xApp can send retry based on that information but
81 resending should happen after Subscription Manager has processed the previous request completely otherwise the request is rejected. If timeout happens response
82 notification contains information where it happened. Timeout can happen in E2, Routing Manager and SDL interface.
84 If there is need to change REPORT or INSERT type subscription then previously made subscription need to be deleted first. If there are any REPORT or INSERT
85 type E2 subscription which need to change frequently, it is not good idea to bundle them with other REPORT or INSERT type E2 subscriptions in the same REST
88 POLICY type subscription can contain multiple E2 subscriptions but it may not be feasible as POLICY subscription may change. Idea in POLICY subscription is that
89 xApp can send changed contend to E2 Node without making new subscription but just send update. Message contend can be changed frequently. In such case it may not
90 be good idea to bundle the POLICY type E2 subscription with any other POLICY type E2 subscriptions in the same REST subscription request.
92 xApp must be able to preserve is the Subscription Manager allocated REST requests ids over restart or send the same request as was sent restart. If xApp is able
93 to send exactly the same requests then Subscription manager can identify the request and send same responses back to xApp. This way xApp can restore the identifies
94 related to existing subscriptions. Another alternative is store needed information in database. Subscription manager calculates md5 sum over the REST Subscription
95 request message. That is how it can detect identical request coming from a specific xApp.
97 xApp can set in the request how many times Subscription Manager will retry subscription in E2 interface if there is timeout. xApp can also set time Subscription
98 Manager Will wait response from E2 Node. xApp may need to update POLICY type subscription frequently. In such case it is not necessary feasible that
99 Subscription Manager retries outdated subscription. By default Subscription Manager retries twice and response waiting time is two seconds.
101 In error case REST notification contains source of error. Source of error can be Subscription Manager, Routing Manager, E2 Node, ASN1 encoding/decoding and DBAAS/SDL.
102 The error information received from these sources is directly copied in the response.
109 Subscribed messages from E2 Node are transported to RIC inside RIC Indication message. RIC Indication message is transported to xApp
110 inside RMR message in Payload field of the RMR message. RMR message is routed to xApp based on SubId field (E2 instance id) in
113 Subscription Manager allocates unique E2 instance id for every E2 subscription during subscription procedure. Subscription Manager
114 puts allocated E2 instance id to InstanceId field in the ASN.1 packed RIC Subscription Request message which is sent to E2 Node. That
115 E2 instance id is then used for the E2 subscription in RIC and E2 Node as long the E2 subscription lives. xApp gets the
116 allocated E2 instance id in REST notification message when E2 subscription procedure is completed.
118 Subscribed messages are routed to xApps based on InstanceId in E2 Indication message. InstanceId is placed in the SubId field of the RMR
119 message header when E2 Termination sends the subscribed message to xApp.
121 RIC Subscription Request and RIC Subscription delete Request messages are pre configured to be routed to E2 Termination and responses
122 to those messages back to Subscription Manager.
124 Subscription Manager allocates RIC Requestor Id for E2 interface communication. Currently the id value is always 123. E2 Node gets the Request
125 of the xApp who makes the first subscription. E2 Node uses Subscription Manager allocated RIC Requestor ID in all RIC Indication messages it sends
126 to RIC for the subscription. In merge case subscription in E2 Node is created only for the first requestor.
128 More information about Routing manager, how routes are created and related signaling can be found from here (Note that there is still
129 RMR based interface between xApp and Subscription Manager in the pictures on those pages):
131 `<https://wiki.o-ran-sc.org/display/RICP/Routing+Manager+Architecture>`_
133 `<https://wiki.o-ran-sc.org/display/RICP/Routing+Manager+and+Subscription+Manager>`_
136 * Subscription Request message
138 .. image:: images/REST_Subscription_Request.png
140 :alt: Subscription Request message
143 * Subscription RESTRequest Response message
145 .. image:: images/REST_Subscription_Response.png
147 :alt: REST Subscription Request Response message
150 * Subscription Request Notification message
152 .. image:: images/REST_Subscription_Notification.png
154 :alt: REST Subscription Notification message
156 * Routing Manager REST interface messages
158 .. image:: images/Routing_Manager_REST_interface_messages.png
160 :alt: Routing Manager REST interface messages
163 * Subscription procedure
167 xApp sends REST Subscription Request message to Subscription Manager. The request can contain multiple E2 subscriptions. It contains also
168 xApp generated xApp instance id for every E2 subscription. Subscription Manager checks does the message contain Subscription Manager allocated
169 REST request id for the request. When xApp sends the request first time there is no REST request id and Subscription Manager allocates it.
171 Then Subscription Manager makes simple validation for data in the request and copies data to Golang data types. When all data is copied successfully
172 Subscription Manager sends successful respond to the REST request. Response contains Subscription Manager allocated REST request id.
173 Then Subscription Manager sends route create request to Routing Manager over REST interface. When route is created successfully, Subscription Manager
174 ASN.1 encodes the E2 messages and forwards those to E2 Termination. When RIC Subscription Response arrives from E2 Termination
175 Subscription Manager forwards REST notification to xApp. The notification contains REST request id, xApp instance id and E2 instance id.
177 Subscription Manager supervises route creation and RIC Subscription Request with a timer.
179 RIC Indication messages which are used to transport subscribed messages from E2 Node are routed from E2 Termination to xApps directly using
180 the routes created during Subscription procedure.
182 Subscription Manager supports REPORT, POLICY and INSERT type subscriptions (E2 RICActionTypes). CONTROL is not supported. POLICY type
183 subscription can be updated. In update case signaling sequence is the same as above, except route is not created to Routing manager.
184 xApp uses initially allocated REST request id, xApp instance id in update case. Route in POLICY type subscription case is needed
185 only that Error Indication could be to xApp, but it is not used currently. RIC Subscription Request message contains list of ActionsToBeSetup
186 information elements. The list cannot have REPORT, POLICY or INSERT action types at the same time. Subscription Manager checks actions types
187 in the message. If different action types is found the REST request is not accepted.
190 .. image:: images/Successful_Subscription.png
192 :alt: Successful subscription picture
197 Failure can happen already before REST request reaches Subscription Manager. Swagger make value checks for the message passed to it.
198 If values are does not accepted then send function returns "unknown error".
200 If failure happens when Subscription Manager validates the REST request then error is returned instantly and processing of request is
201 stopped. xApp receives bad request (HTTP response code 400) response.
203 If failure response is received from E2 Node then REST notification is forwarded to xApp with appropriate error cause. The notification
204 contains REST request id, xApp instance id and zero E2 instance id.
206 .. image:: images/Subscription_Failure.png
208 :alt: Subscription failure picture
210 * Timeout in Subscription Manager
212 In case of timeout in Subscription Manager, Subscription Manager may resend the RIC Subscription Request to E2 Node. By default Subscription
213 Manager retries twice. If there is no response after retries, Subscription Manager sends unsuccessful REST notification to xApp. The notification
214 contains REST request id, xApp instance id and zero E2 instance id.
218 xApp can resend the same REST Subscription Request if request timeouts.
220 xApp may resend the same request if it does not receive expected notification in expected time. If xApp resends the same request while Subscription
221 Manager is still processing previous request then Subscription Manager responds accepts the request and continues processing previous request.
223 .. image:: images/Subscription_Timeout.png
225 :alt: Subscription timeout picture
227 * Subscription delete procedure
231 xApp sends REST Subscription Delete Request message to Subscription Manager. xApp must use the same REST request id which it received in REST Subscription
232 Response. REST delete request will delete all successfully subscribed E2 subscriptions which was subscribed earlier when the REST request id was created.
233 When Subscription Manager receives REST Subscription Delete Request it check has it such REST subscription. If it has then Subscription Manager sends successful
234 response to xApp and starts sending E2 delete requests to E2 Termination one by one. When RIC Subscription Delete Response arrives from E2 Termination to
235 Subscription Manager, Subscription Manager request route deletion from Routing Manager. xApp does not get any notification about deleted E2 subscriptions.
237 Subscription Manager supervises RIC Subscription Deletion Request and route delete with a timer.
239 .. image:: images/Successful_Subscription_Delete.png
241 :alt: Successful subscription delete picture
245 Delete procedure cannot fail from xApp point of view. Subscription Manager always responds with successful REST Subscription Response to xApp.
246 E2 Node could respond with delete failure in case the subscription which Subscription Manager wants to delete does not exist. In this case delete procedure
249 .. image:: images/Subscription_Delete_Failure.png
251 :alt: Subscription delete failure picture
253 * Timeout in Subscription Manager
255 In case of timeout in Subscription Manager, Subscription Manager may resend the RIC Subscription Delete Request to E2 Node. By default Subscription Manager
256 retries twice. If there is no response after retry, Subscription Manager stops trying.
260 xApp can resend the same REST Subscription Delete Request if request timeouts.
262 .. image:: images/Subscription_Delete_Timeout.png
264 :alt: Subscription delete timeout picture
266 * Unknown REST request id
268 If Subscription Manager receives RIC Subscription Delete Request for a REST request id which does not exist, Subscription Manager sends
269 successful REST response to xApp.
271 * Subscription merge procedure
275 Merge is possible only for REPORT type subscription. It is possible only when Action Type and Event Trigger Definition of subscriptions are equal.
277 xApp sends REST Subscription Request message to Subscription Manager. The request can contain multiple E2 subscriptions as in normal Subscription
278 procedure but some of the E2 subscriptions in the list are already subscribed from E2 Node. For those which are not yet subscribed Subscription Manager
279 applies normal Subscription procedure. E2 subscriptions in the list which are already subscribed are just assigned to existing subscriptions and Subscription
280 Manager just sends route create to Routing Manager and then forwards successful REST notification to xApp for the E2 subscriptions. The notification
281 contains REST request id, xApp instance id and E2 instance id.
283 One thing to note! REST Subscription request and returned REST notification goes through different TCP ports. For that reason there is no guarantee that
284 response for REST Subscription request arrives to xApp before first REST notification. That is possible mostly in merge case where subscription already exist
285 in Subscription Manager records. Successful REST notification is returned to xApp without making subscription from E2 Node which would cause some delay before
286 REST notification can be sent.
288 Route create is supervised with a timer.
290 ``Only REPORT type subscriptions can be be merged.``
292 .. image:: images/Successful_Subscription_Merge.png
294 :alt: Successful subscription merge picture
298 Failure can happen already before REST request reaches Subscription Manager. Swagger make value checks for the message passed to it.
299 If values are does not accept then send function returns "unknown error".
301 If failure happens when Subscription Manager validates the REST request then error is returned instantly and processing of request is
302 stopped. xApp receives bad request (HTTP response code 400) response.
304 If error happens during route create then Subscription Manager forwards REST notification toxApp with appropriate error cause. The notification contains
305 also REST request id, xApp instance id and zero E2 instance id.
307 * Timeout in Subscription Manager
309 Timeout can come only in route create during merge operation. If error happens during route create then Subscription Manager forwards REST
310 notification toxApp with appropriate error cause. The notification contains also REST request id, xApp instance id and zero E2 instance id.
314 xApp can resend the same REST Subscription Request if request timeouts.
316 * Subscription delete merge procedure
320 xApp sends REST Subscription Delete Request message to Subscription Manager. If delete concerns merged subscription, Subscription Manager
321 responds with REST Subscription Delete Response to xApp and then sends route delete request to Routing manager.
323 Subscription Manager supervises route delete with a timer.
325 .. image:: images/Successful_Subscription_Delete_Merge.png
327 :alt: Successful subscription delete merge picture
331 Delete procedure cannot fail from xApp point of view. Subscription Manager always responds with successful REST Subscription Delete Response to xApp.
333 * Timeout in Subscription Manager
335 Timeout can only happen in route delete to Routing manager. Subscription Manager always responds with successful REST Subscription Delete Response to xApp.
339 xApp can resend the same REST Delete Request if request timeouts.
343 When xApp is restarted for any reason it may resend REST subscription requests for subscriptions which have already been subscribed. If REPORT or INSERT type
344 subscription already exists and RMR endpoint of requesting xApp is attached to subscription then successful response is sent to xApp directly without
345 updating Routing Manager and E2 Node. If POLICY type subscription already exists, request is forwarded to E2 Node and successful response is sent to xApp.
346 E2 Node is expected to accept duplicate POLICY type requests. In restart IP address of the xApp may change but domain service address name does not.
347 RMR message routing uses domain service address name.
349 * Subscription Manager restart
351 Subscription Manager stores REST request ids, E2 subscriptions and their mapping to REST request ids in db (SDL). In start up Subscription Manager restores REST request
352 ids, E2 subscriptions and their mapping from db. For E2 subscriptions which were not successfully completed, Subscription Manager sends delete request to E2 Node and
353 removes routes created for those. In restart case xApp may need to resend the same REST request to get all E2 subscriptions completed.
355 Restoring subscriptions from db can be disable via submgr-config.yaml file by setting "readSubsFromDb": "false".
359 Subscription Manager adds following statistic counters:
361 Subscription create counters:
362 - SubReqFromXapp: The total number of SubscriptionRequest messages received from xApp
363 - SubRespToXapp: The total number of SubscriptionResponse messages sent to xApp
364 - SubFailToXapp: The total number of SubscriptionFailure messages sent to xApp
365 - SubReqToE2: The total number of SubscriptionRequest messages sent to E2Term
366 - SubReReqToE2: The total number of SubscriptionRequest messages resent to E2Term
367 - SubRespFromE2: The total number of SubscriptionResponse messages from E2Term
368 - SubFailFromE2: The total number of SubscriptionFailure messages from E2Term
369 - SubReqTimerExpiry: The total number of SubscriptionRequest timer expires
370 - RouteCreateFail: The total number of subscription route create failure
371 - RouteCreateUpdateFail: The total number of subscription route create update failure
372 - MergedSubscriptions: The total number of merged Subscriptions
374 Subscription delete counters:
375 - SubDelReqFromXapp: The total number of SubscriptionDeleteResponse messages received from xApp
376 - SubDelRespToXapp: The total number of SubscriptionDeleteResponse messages sent to xApp
377 - SubDelReqToE2: The total number of SubscriptionDeleteRequest messages sent to E2Term
378 - SubDelReReqToE2: The total number of SubscriptionDeleteRequest messages resent to E2Term
379 - SubDelRespFromE2: The total number of SubscriptionDeleteResponse messages from E2Term
380 - SubDelFailFromE2: The total number of SubscriptionDeleteFailure messages from E2Term
381 - SubDelReqTimerExpiry: The total number of SubscriptionDeleteRequest timer expires
382 - RouteDeleteFail: The total number of subscription route delete failure
383 - RouteDeleteUpdateFail: The total number of subscription route delete update failure
384 - UnmergedSubscriptions: The total number of unmerged Subscriptions
386 SDL failure counters:
387 - SDLWriteFailure: The total number of SDL write failures
388 - SDLReadFailure: The total number of SDL read failures
389 - SDLRemoveFailure: The total number of SDL read failures
391 Configurable parameters
392 -----------------------
393 Subscription Manager has following configurable parameters.
394 - Retry timeout for RIC Subscription Request message
395 - e2tSubReqTimeout_ms: 2000 is the default value
397 - Retry timeout for RIC Subscription Delete Request message
398 - e2tSubDelReqTime_ms: 2000 is the default value
400 - Waiting time for RIC Subscription Response and RIC Subscription Delete Response messages
401 - e2tRecvMsgTimeout_ms: 2000 is the default value
403 - Try count for RIC Subscription Request message
404 - e2tMaxSubReqTryCount: 2 is the default value
406 - Try count for RIC Subscription Delete Request message
407 - e2tMaxSubDelReqTryCount: 2 is the default value
409 - Are subscriptions read from database in Subscription Manager startup
410 - readSubsFromDb: "true" is the default value
412 The parameters can be changed on the fly via Kubernetes Configmap. Default parameters values are defined in Helm chart
414 Use following command to open Subscription Manager's Configmap in Nano editor. First change parameter and then store the
415 change by pressing first Ctrl + o. Close editor by pressing the Ctrl + x. The change is visible in Subscription Manager's
416 log after some 20 - 30 seconds.
420 KUBE_EDITOR="nano" kubectl edit cm configmap-ricplt-submgr-submgrcfg -n ricplt
422 REST interface for debugging and testing
423 ----------------------------------------
424 Give following commands to get Subscription Manager pod's IP address
428 kubectl get pods -A | grep submgr
430 ricplt submgr-75bccb84b6-n9vnt 1/1 Running 0 81m
432 Syntax: kubectl exec -t -n ricplt <add-submgr-pod-name> -- cat /etc/hosts | grep submgr | awk '{print $1}'
434 Example: kubectl exec -t -n ricplt submgr-75bccb84b6-n9vnt -- cat /etc/hosts | grep submgr | awk '{print $1}'
442 Example: curl -s GET "http://10.244.0.181:8080/ric/v1/metrics"
444 Get REST subscriptions
448 Example: curl -X GET "http://10.244.0.181:8080/ric/v1/restsubscriptions"
454 Example: curl -X GET "http://10.244.0.181:8088/ric/v1/subscriptions"
456 Delete single E2 subscription from db
460 Syntax: curl -X POST "http://10.244.0.181:8080/ric/v1/test/deletesubid={SubscriptionId}"
462 Example: curl -X POST "http://10.244.0.181:8080/ric/v1/test/deletesubid=1"
464 Remove all subscriptions from db
468 Example: curl -X POST "http://10.244.0.181:8080/ric/v1/test/emptydb"
470 Make Subscription Manager restart
474 Example: curl -X POST "http://10.244.0.181:8080/ric/v1/test/restart"
476 Use this command to get Subscription Manager's log writings
480 Example: kubectl logs -n ricplt submgr-75bccb84b6-n9vnt
482 Logger level in configmap.yaml file in Helm chart is by default 2. It means that only info logs are printed.
483 To see debug log writings it has to be changed to 4.
490 RAN services explained
491 ----------------------
492 RIC hosted xApps may use the following RAN services from a RAN node:
494 * REPORT: RIC requests that RAN sends a REPORT message to RIC and continues further call processing in RAN after each occurrence of a defined SUBSCRIPTION
495 * INSERT: RIC requests that RAN sends an INSERT message to RIC and suspends further call processing in RAN after each occurrence of a defined SUBSCRIPTION
496 * CONTROL: RIC sends a Control message to RAN to initiate or resume call processing in RAN
497 * POLICY: RIC requests that RAN executes a specific POLICY during call processing in RAN after each occurrence of a defined SUBSCRIPTION
499 Supported E2 procedures and RAN services
500 ----------------------------------------
501 * RIC Subscription procedure with following RIC action types:
507 * RIC Subscription Delete procedure
509 * Merge and delete of equal REPORT type subscriptions.
511 Recommendations for xApps
512 -------------------------
514 * Recommended retry delay in xApp
516 Subscription Manager makes two retries for E2 subscriptions and E2 subscription deletions. xApp should not retry before it has received REST notification for
517 all E2 subscriptions sent in REST subscription request. Maximum time to complete all E2 subscriptions in Subscription Manager can be calculated like this:
518 t >= 3 * 2s * count_of_subscriptions in the REST request. Length of supervising timers in Subscription Manager for the requests it sends to E2 Node is by
519 default 2 seconds. There can be only one ongoing E2 subscription request towards per E2 Node other requests are queued in Subscription Manager.