X-Git-Url: https://gerrit.o-ran-sc.org/r/gitweb?a=blobdiff_plain;f=test%2Fauto-test%2FREADME.md;h=cd3c77874a2836c57869e9f1fbb30fd8b17c2f83;hb=007b64509101d8e3ef881955adee2ad15d062213;hp=9aac651e6a95598376c02a1456f08489a1702449;hpb=7a7a13a47eeebf9a61fa84d14af59b1cbe1598bc;p=nonrtric.git diff --git a/test/auto-test/README.md b/test/auto-test/README.md index 9aac651e..cd3c7787 100644 --- a/test/auto-test/README.md +++ b/test/auto-test/README.md @@ -1,57 +1,135 @@ +# Overview -## Automated test Description -This auto-test repo stores test script for automated test cases for Policy Agent. -Each of the testcase script will bring up a containerized test enviroment for Policy Agent, -CBS, consul, and simulator(TBD). +The bash scripts in this dir are intended for function test of the Non-RT RIC in different configurations, using simulators when needed for the external interfaces. +A few of the bash scripts are so called 'suites', These suite scripts calls a sequence of the other bash scripts. -### Overview +## Automated test scripts -Right now, test cases are written in bash scripts. \ -Each test case script(ex. `FTC1.sh)` will call functions defined in `../common`. \ -The environment vriables are set in`test_env.sh`. \ -The automated test support both local build Policy Agent image testing and remote image stored in Nexus. -``` -# Local image -export POLICY_AGENT_LOCAL_IMAGE=o-ran-sc/nonrtric-policy-agent -# Remote image -export POLICY_AGENT_REMOTE_IMAGE=nexus3.o-ran-sc.org:10004/o-ran-sc/nonrtric-policy-agent -``` -### Test Cases Description(more TBD) -`FTC1.sh`: Test policy-agent can refresh configurations from consul - -### Logs -All log files are stored at `logs/`. \ -The logs include the application.log and the container log from Policy Agent, the container logs from each simulator and the -test case log (same as the screen output). \ -In the test cases the logs are stored with a prefix so the logs can be stored at different steps during the test. -All test cases contains an entry to save all logs with prefix 'END' at the end of each test case. - -### Manual -Test case command: -``` -./.sh local | remote +There are two types of scripts, filenames in the format FTCXXX.sh test one or more components of the Non-RT RIC. Filenames in the format SuiteZZZZ.sh tests a number of FTCXXX.sh script as one suite. (XXX is an integer selected from the categories described further below). +FTC is short for Function Test Case. In addition, there are also other test scripts with other naming format used for demo setup etc (e.g PM_DEMO.sh). -Discription: -local: test image: POLICY_AGENT_LOCAL_IMAGE=o-ran-sc/nonrtric-policy-agent -remote: test image: nexus3.o-ran-sc.org:10004/o-ran-sc/nonrtric-policy-agent -``` +The requirements, in terms of the execution enviroment, to run a script or a suite is to have docker, docker-compose and python3 installed (the scripts warns if not installed). As an option, the scripts can also be executed in a Minikube or Kubernetes installation. The additional requirement is to have a clean minikube/kubernetes installation, perferably with the kube dashboard installed. +The scripts have been tested to work on both MacOS and Ubuntu using docker. They should work also in git-bash on windows (for docker) but only partly verified. Running using minikube has only been verified on Ubuntu and running on kubernetes has only been verified on MacOS. + +## Configuration + +The test scripts uses configuration from a single file, found in `../common/test_env.sh`, which contains all needed configuration in terms of image names, image tags, ports, file paths, passwords etc. This file can be modified if needed. See the README.md in `../common/` for all details of the config file. + +## How to run + +A test script, for example FTC1, is executed from the cmd line using the script filename and one or more parameters: + + >```./FTC1.sh remote docker --env-file ../common/test_env-oran-cherry.sh``` + +Note that not is running on a released verion, the parameter "release" shall be included to run the released images. + +See the README.md in `../common/` for all details about available parameters and their meaning. + +Each test script prints out the overall result of the tests in the end of the execution. + +The test scripts produce quite a number of logs; all container logs, a log of all http/htps calls from the test scripts including the payload, some configuration created during test and also a test case log (same as what is printed on the screen during execution). All these logs are stored in `logs/FTCXXX/`. So each test script is using its own log directory. + +To test all components on a very basic level, run the demo test script(s) for the desired release. +Note that oran tests only include components from oran. +Note that onap test uses components from onap combined with released oran components available at that onap release (e.g. Honolulu contains onap images from honolulu and oran images from cherry) + + +ORAN CHERRY +=========== +>```./PM_EI_DEMO.sh remote-remove docker release --env-file ../common/test_env-oran-cherry.sh``` + +>```./PM_EI_DEMO.sh remote-remove kube release --env-file ../common/test_env-oran-cherry.sh``` + +ORAN D-RELEASE +========= +>```./PM_EI_DEMO.sh remote-remove docker release --env-file ../common/test_env-oran-d-release.sh --use-release-image SDNC``` + +>```./PM_EI_DEMO.sh remote-remove kube release --env-file ../common/test_env-oran-d-release.sh --use-release-image SDNC``` + +ORAN E-RELEASE +========= +>```./PM_EI_DEMO.sh remote-remove docker --env-file ../common/test_env-oran-e-release.sh``` + +>```./PM_EI_DEMO.sh remote-remove kube --env-file ../common/test_env-oran-e-release.sh``` + +ONAP GUILIN +=========== +>```./PM_DEMO.sh remote-remove docker release --env-file ../common/test_env-onap-guilin.sh``` + +>```./PM_DEMO.sh remote-remove kube release --env-file ../common/test_env-onap-guilin.sh``` + +Note that ICS was not available before oran cherry so a test script without ICS is used. + +ONAP HONOLULU +============= +>```./PM_EI_DEMO.sh remote-remove docker release --env-file ../common/test_env-onap-honolulu.sh``` + +>```./PM_EI_DEMO.sh remote-remove kube release --env-file ../common/test_env-onap-honolulu.sh``` -### Test case file -A test case file contains a number of steps to verify a certain functionality. -A description of the test case should be given to the ``TC_ONELINE_DESCR`` var. The description will be printed in -the test result. +ONAP ISTANBUL +============= +>```./PM_EI_DEMO.sh remote-remove docker release --env-file ../common/test_env-onap-istanbul.sh``` -The empty template for a test case files looks like this: +>```./PM_EI_DEMO.sh remote-remove kube release --env-file ../common/test_env-onap-istanbul.sh``` -(Only the parts noted with < and > shall be changed.) +Note: When istanbul is released, add the 'release' arg to run released images. + +## Test case categories + +The test script are number using these basic categories where 0-999 are releated to the policy managment and 1000-1999 are related to information management. 2000-2999 are for southbound http proxy. There are also demo test cases that test more or less all components. These test scripts does not use the numbering scheme below. + +The numbering in each series corresponds to the following groupings +1-99 - Basic sanity tests + +100-199 - API tests + +300-399 - Config changes and sync + +800-899 - Stability and capacity test + +900-999 - Misc test + +11XX - ICS API Tests + +18XX - ICS Stability and capacity test + +2000 - Southbound http proxy tests + +30XX - rApp tests + +Suites + +To get an overview of the available test scripts, use the following command to print the test script description: +'grep ONELINE *.sh' in the dir of the test scripts. + +## Test case file - template + +A test script contains a number of steps to verify a certain functionality. +The empty template for a test case file looks like this. +Only the parts noted with < and > shall be changed. +It is strongly suggested to look at the existing test scripts, it is probably easier to copy an existing test script instead of creating one from scratch. The README.md in `../common/` describes the functions available in the test script in detail. ----------------------------------------------------------- + ``` -#!/usr/bin/env bash +#!/bin/bash TC_ONELINE_DESCR="" -. ../common/testcase_common.sh $1 +DOCKER_INCLUDED_IMAGES= + +KUBE_INCLUDED_IMAGES= +KUBE_PRESTARTED_IMAGES= + +SUPPORTED_PROFILES= + +SUPPORTED_RUNMODES= + +CONDITIONALLY_IGNORED_IMAGES= + +. ../common/testcase_common.sh $@ + +setup_testenvironment #### TEST BEGIN #### @@ -64,7 +142,20 @@ TC_ONELINE_DESCR="" store_logs END ``` + ----------------------------------------------------------- -The ../common/testcase_common.sh contains all functions needed for the test case file. See the README.md file in -the ../common dir for a description of all available functions. \ No newline at end of file +## License + +Copyright (C) 2020 Nordix Foundation. All rights reserved. +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.