2 .. Copyright (c) 2019 AT&T Intellectual Property.
3 .. Copyright (c) 2019 Nokia.
5 .. Licensed under the Creative Commons Attribution 4.0 International
6 .. Public License (the "License"); you may not use this file except
7 .. in compliance with the License. You may obtain a copy of the License at
9 .. https://creativecommons.org/licenses/by/4.0/
11 .. Unless required by applicable law or agreed to in writing, documentation
12 .. distributed under the License is distributed on an "AS IS" BASIS,
13 .. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15 .. See the License for the specific language governing permissions and
16 .. limitations under the License.
31 RIC alarm system consists of three components: Alarm Manager, Application Library and Command Line Interface
33 The Alarm Manager is responsible for managing alarm situations in RIC cluster and interfacing with Northbound applications
34 such as Prometheus Alert Manager to post the alarms as alerts. Alert Manager takes care of de-duplicating, silencing and
35 inhibition (suppressing) of alerts, and routing them to the VES-Agent, which, in turn, takes care of converting alerts to
36 faults and sending them to ONAP as VES events.
38 The Alarm Library provides a simple interface for RIC applications (both platform application and xApps) to raise and clear
39 alarms. The Alarm Library interacts with the Alarm Manager via RMR interface.
41 .. image:: images/RIC_Alarm_System.png
43 :alt: Place in RIC's software architecture picture
48 The Alarm Manager listens alarms coming via RMR and REST interfaces. An application can raise or clear alarms via either
49 of interfaces. Alarm Manager listens also commands coming from CLI (Command Line Interface). In addition Alarm Manager supports few
50 other commands that can be given through the interfaces. Such as list active alarms, list alarm history, add new alarms
51 definition, delete existing alarm definition, re-raise alarms and clear all alarms. Those are not typically used by applications while
52 running. Alarm Manager itself re-raises alarms periodically to keep alarms in active state. The other commands are can be used through
53 CLI interface by operator or are used when applications is starting up or restarting.
55 Maximum amount of active alarms and size of alarm history are configurable. By default, the values are Maximum number of active
56 alarms = 5000, Maximum number of alarm history = 20,000.
58 Alarm definitions can be updated dynamically via REST interface. Default definitions are read from JSON configuration file when FM
64 The Alarm Library provides simple interface for RIC applications (both platform application and xApps) to raise and clear
65 alarms. A new alarm instance is created with InitAlarm()-function. ManagedObject (mo) and Application (ap) identities are
66 given as parameters for Alarm Context/Object
68 The Alarm object contains following parameters:
70 SpecificProblem: problem that is the cause of the alarm \(*
72 PerceivedSeverity: The severity of the alarm, see below for possible values
74 ManagedObjectId: The name of the managed object that is the cause of the fault \(*
76 ApplicationId: The name of the process raised the alarm \(*
78 AdditionalInfo: Additional information given by the application
80 IdentifyingInfo: Identifying additional information, which is part of alarm identity \(*
82 Items marked with \*, i.e., ManagedObjectId (mo), SpecificProblem (sp), ApplicationId (ap) and IdentifyingInfo (IdentifyingInfo) make
83 up the identity of the alarm. All parameters must be according to the alarm definition, i.e. all mandatory parameters should be present,
84 and parameters should have correct value type or be from some predefined range. Addressing the same alarm instance in a clear() or reraise()
85 call is done by making sure that all four values are the same is in the original raise() / reraise() call.
87 Alarm Manager does not allow raising "same alarm" more than once without that the alarm is cleared first. Alarm Manager compares
88 ManagedObjectId (mo), SpecificProblem (sp), ApplicationId (ap) and IdentifyingInfo (IdentifyingInfo) parameters to check possible
89 duplicate. If the values are the same then alarm is suppressed. If application raises the "same alarm" but PerceivedSeverity of the alarm
90 is changed then Alarm Manager deletes the old alarm and makes new alarm according to new information.
95 Raise: Raises the alarm instance given as a parameter
97 Clear: Clears the alarm instance given as a parameter, if it the alarm active
99 Reraise: Attempts to re-raise the alarm instance given as a parameter
101 ClearAll: Clears all alarms matching moId and appId given as parameters (not supported yet)
104 Command line interface
105 ----------------------
107 Through CLI operator can do the following operations:
109 - Check active alarms
110 - Check alarm history
113 - Configure maximum active alarms and maximum alarms in alarm history
114 - Add new alarm definitions that can be raised
115 - Delete existing alarm definition that can be raised
117 CLI commands need to be given inside Alarm Manger pod. To get there first print name of the Alarm Manger pod.
121 kubectl get pods -A | grep alarmmanager
123 Output should be look someting like this:
127 ricplt deployment-ricplt-alarmmanager-6cc8764749-gnwjh 1/1 running 0 15d
129 Then give this command to enter inside the pod. Replace the pod name with the actual name from the printout.
133 kubectl exec -it deployment-ricplt-alarmmanager-6cc8764749-gnwjh bash
135 CLI commands can have some of the following parameters
139 --moid ManagedObjectId, example string: RIC
140 --apid ApplicationId string, example string: UEEC
141 --sp SpecificProblem, example value: 8007
142 --severity Severity of the alarm, possible values: UNSPECIFIED, CRITICAL, MAJOR, MINOR, WARNING, CLEARED or DEFAULT
143 --iinfo Identifying info, a user specified string, example string: INFO-1
144 --mal Maximum number of active alarms, example value 1000
145 --mah Maximum number of alarms in alarm history, example value: 2000
146 --aid Alarm id, example value: 8007
147 --atx Alarm text string, example string: E2 CONNECTIVITY LOST TO E-NODEB
148 --ety Event type string, example string: Communication error
149 --oin Operation instructions string, example string: Not defined
150 --rad Raise alarm delay in seconds. Default value = 0
151 --cad Clear alarm delay in seconds. Default value = 0
152 --prf Performance profile id, possible values: 1 = peak performance test or 2 = endurance test
153 --nal Number of alarms, example value: 50
154 --aps Alarms per second, example value: 1
155 --tim Total time of test in minutes, example value: 1
156 --host Alarm Manager host address. Default value = localhost
157 --port Alarm Manager port. Default value = 8080
158 --if Used Alarm Manager command interface, http or rmr: default value = http
159 --active Active alerts in Prometheus Alert Manager. Default value = true
160 --inhibited Inhibited alerts in Prometheus Alert Manager. Default value = true
161 --silenced Silenced alerts in Prometheus Alert Manager. Default value = true
162 --unprocessed Unprocessed alerts in Prometheus Alert Manager. Default value = true
163 --host Prometheus Alert Manager host address
164 --port Prometheus Alert Manager port. Default value = 9093
167 ``Note that there are two minus signs before parameter name!``
169 If parameter contains any white spaces then it must be enclosed in quotation marks like: "INFO 1"
171 CLI command examples:
173 Following command are given at top level directory!
179 Syntax: cli/alarm-cli active [--host] [--port]
181 Example: cli/alarm-cli active
183 Example: cli/alarm-cli active --host localhost --port 8080
189 Syntax: cli/alarm-cli active [--host] [--port]
191 Example: cli/alarm-cli history
193 Example: cli/alarm-cli history --host localhost --port 8080
199 Syntax: cli/alarm-cli raise --moid --apid --sp --severity --iinfo [--host] [--port] [--if]
201 Example: cli/alarm-cli raise --moid RIC --apid UEEC --sp 8007 --severity CRITICAL --iinfo INFO-1
203 Following is meant only for testing and verification purpose!
205 Example: cli/alarm-cli raise --moid RIC --apid UEEC --sp 8007 --severity CRITICAL --iinfo INFO-1 --host localhost --port 8080 --if rmr
211 Syntax: cli/alarm-cli clear --moid --apid --sp --severity --iinfo [--host] [--port] [--if]
213 Example: cli/alarm-cli clear --moid RIC --apid UEEC --sp 8007 --iinfo INFO-1
215 Example: cli/alarm-cli clear --moid RIC --apid UEEC --sp 8007 --iinfo INFO-1 --host localhost --port 8080 --if rmr
217 Configure maximum active alarms and maximum alarms in alarm history:
221 Syntax: cli/alarm-cli configure --mal --mah [--host] [--port]
223 Example: cli/alarm-cli configure --mal 1000 --mah 5000
225 Example: cli/alarm-cli configure --mal 1000 --mah 5000 --host localhost --port 8080
227 Add new alarm definition:
231 Syntax: cli/alarm-cli define --aid 8007 --atx "E2 CONNECTIVITY LOST TO E-NODEB" --ety "Communication error" --oin "Not defined" [--rad] [--cad] [--host] [--port]
233 Example: cli/alarm-cli define --aid 8007 --atx "E2 CONNECTIVITY LOST TO E-NODEB" --ety "Communication error" --oin "Not defined" --rad 0 --cad 0
235 Example: cli/alarm-cli define --aid 8007 --atx "E2 CONNECTIVITY LOST TO E-NODEB" --ety "Communication error" --oin "Not defined" --rad 0 --cad 0 --host localhost --port 8080
237 Delete existing alarm definition:
241 Syntax: cli/alarm-cli undefine --aid [--host] [--port]
243 Example: cli/alarm-cli undefine --aid 8007
245 Example: cli/alarm-cli undefine --aid 8007 --host localhost --port 8080
247 Conduct performance test:
249 Note that this is meant only for testing and verification purpose!
251 Before any performance test command can be issued, an environment variable needs to be set. The variable holds information where
252 test alarm object file is stored.
256 PERF_OBJ_FILE=cli/perf-alarm-object.json
258 Syntax: cli/alarm-cli perf --prf --nal --aps --tim [--host] [--port] [--if]
260 Peak performance test example: cli/alarm-cli perf --prf 1 --nal 50 --aps 1 --tim 1 --if rmr
262 Peak performance test example: cli/alarm-cli perf --prf 1 --nal 50 --aps 1 --tim 1 --if http
264 Peak performance test example: cli/alarm-cli perf --prf 1 --nal 50 --aps 1 --tim 1 --host localhost --port 8080 --if rmr
266 Endurance test example: cli/alarm-cli perf --prf 2 --nal 50 --aps 1 --tim 1 --if rmr
268 Endurance test example: cli/alarm-cli perf --prf 2 --nal 50 --aps 1 --tim 1 --if http
270 Endurance test example: cli/alarm-cli perf --prf 2 --nal 50 --aps 1 --tim 1 --host localhost --port 8080 --if rmr
272 Get alerts from Prometheus Alert Manager:
276 Syntax: cli/alarm-cli alerts --active --inhibited --silenced --unprocessed --host [--port]
278 Example: cli/alarm-cli alerts --active true --inhibited true --silenced true --unprocessed true --host 10.102.36.121 --port 9093
281 REST interface usage guide
282 --------------------------
284 REST interface offers all the same services plus some more that are available via CLI. The CLI also uses the REST interface to implement the services it offers.
286 Below are examples for REST interface. Curl tool is used to send REST commands.
290 Example: curl -X GET "http://localhost:8080/ric/v1/alarms/active" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
294 Example: curl -X GET "http://localhost:8080/ric/v1/alarms/history" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
298 Example: curl -X POST "http://localhost:8080/ric/v1/alarms" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"managedObjectId\": \"RIC\", \"applicationId\": \"UEEC\", \"specificProblem\": 8007, \"perceivedSeverity\": \"CRITICAL\", \"additionalInfo\": \"-\", \"identifyingInfo\": \"INFO-1\", \"AlarmAction\": \"RAISE\", \"AlarmTime\": 0}"
302 Example: curl -X DELETE "http://localhost:8080/ric/v1/alarms" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"managedObjectId\": \"RIC\", \"applicationId\": \"UEEC\", \"specificProblem\": 8007, \"perceivedSeverity\": \"\", \"additionalInfo\": \"-\", \"identifyingInfo\": \"INFO-1\", \"AlarmAction\": \"CLEAR\", \"AlarmTime\": 0}"
304 Get configuration of maximum active alarms and maximum alarms in alarm history:
306 Example: curl -X GET "http://localhost:8080/ric/v1/alarms/config" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
308 Configure maximum active alarms and maximum alarms in alarm history:
310 Example: curl -X POST "http://localhost:8080/ric/v1/alarms/config" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"maxactivealarms\": 1000, \"maxalarmhistory\": 5000}"
312 Get all alarm definitions:
314 Example: curl -X GET "http://localhost:8080/ric/v1/alarms/define" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
316 Get an alarm definition:
318 Syntax: curl -X GET "http://localhost:8080/ric/v1/alarms/define/{alarmId}" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
320 Example: curl -X GET "http://localhost:8080/ric/v1/alarms/define/8007" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
322 Add one new alarm definition:
324 Example: curl -X POST "http://localhost:8080/ric/v1/alarms/define" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"alarmdefinitions\": [{\"alarmId\": 8007, \"alarmText\": \"E2 CONNECTIVITY LOST TO E-NODEB\", \"eventtype\": \"Communication error\", \"operationinstructions\": \"Not defined\, \"raiseDelay\": 1, \"clearDelay\": 1"}]}"
326 Add two new alarm definitions:
328 Example: curl -X POST "http://localhost:8080/ric/v1/alarms/define" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"alarmdefinitions\": [{\"alarmId\": 8007, \"alarmText\": \"E2 CONNECTIVITY LOST TO E-NODEB\", \"eventtype\": \"Communication error\", \"operationinstructions\": \"Not defined\, \"raiseDelay\": 0, \"clearDelay\": 0"},{\"alarmId\": 8008, \"alarmText\": \"ACTIVE ALARM EXCEED MAX THRESHOLD\", \"eventtype\": \"storage warning\", \"operationinstructions\": \"Clear alarms or raise threshold\", \"raiseDelay\": 0, \"clearDelay\": 0}]}"
330 Delete one existing alarm definition:
332 Syntax: curl -X DELETE "http://localhost:8080/ric/v1/alarms/define/{alarmId}" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
334 Example: curl -X DELETE "http://localhost:8080/ric/v1/alarms/define/8007" -H "accept: application/json" -H "Content-Type: application/json" -d "{}"
337 RMR interface usage guide
338 -------------------------
339 Through RMR interface application can only raise and clear alarms. RMR message payload is similar JSON message as in above REST interface use cases.
341 Supported events via RMR interface
346 - ClearAll alarms (not supported yet)
349 Example on how to use the API from Golang code
350 ----------------------------------------------
351 Alarm library functions can be used directly from Golang code. Rising and clearing alarms goes via RMR interface from alarm library to Alarm Manager.
359 alarm "gerrit.o-ran-sc.org/r/ric-plt/alarm-go.git/alarm"
363 // Initialize the alarm component
364 alarmer, err := alarm.InitAlarm("my-pod", "my-app")
366 // Create a new Alarm object (SP=8004, etc)
367 alarm := alarmer.NewAlarm(8004, alarm.SeverityMajor, "NetworkDown", "eth0")
369 // Raise an alarm (SP=8004, etc)
370 err := alarmer.Raise(alarm)
372 // Clear an alarm (SP=8004)
373 err := alarmer.Clear(alarm)
375 // Re-raise an alarm (SP=8004)
376 err := alarmer.Reraise(alarm)
378 // Clear all alarms raised by the application - (not supported yet)
379 err := alarmer.ClearAll()
388 INFO[2020-06-08T07:50:10Z]
391 "commonEventHeader": {
393 "eventId": "fault0000000001",
394 "eventName": "Fault_ricp_E2 CONNECTIVITY LOST TO G-NODEB",
395 "lastEpochMicrosec": 1591602610944553,
396 "nfNamingCode": "ricp",
397 "priority": "Medium",
398 "reportingEntityId": "035EEB88-7BA2-4C23-A349-3B6696F0E2C4",
399 "reportingEntityName": "Vespa",
402 "startEpochMicrosec": 1591602610944553,
407 "alarmCondition": "E2 CONNECTIVITY LOST TO G-NODEB",
408 "eventSeverity": "MAJOR",
409 "eventSourceType": "virtualMachine",
410 "faultFieldsVersion": 2,
411 "specificProblem": "eth12",
416 INFO[2020-06-08T07:50:10Z] Schema validation succeeded