What are different characteristics of resilience testing?

What does resilience mean? It’s important to know the meaning of resilience first because so many people confuse themselves with recovery, reliability and resilience. They think it’s all the same. But it is not so.

- Resilience means to recover from a change.
- It’s slightly different from recovery and reliability.
- Every software application or system has to have some degree of resilience in it in order to be more secure and recoverable and reliable.
- Resilience is a non functional requirement of a software system or application.
- Resilience testing falls under the category of non functional testing.

It is very common for the interchanging use of many non functional tests because of the overlapping in the scope between many non functional aspects or requirements.
One thing to be noted is that software performance is a broad and vast term and includes many specific requirements like scalability, reliability, compatibility, security and resilience.

Non functional testing contains the following testing techniques:
- Compliance testing
- Baseline testing
- Documentation testing
- Compatibility testing
- Load testing
- Localization testing
- Endurance testing
- Internationalization testing
- Recovery testing
- Performance testing
- Security testing
- Volume testing
- Usability testing
- Stress testing
- Scalability testing
- Resilience testing

Software system or application developers with disaster recovery plans or techniques are said to be actively and effectively engaged in reducing the risk of the software system or application crash, failure or data loss. But, the irony is that these disaster recovery plans become complacent.

This happens so because many of the software developers or testers have a false sense of security based on the existence of their disaster recovery plans. To ensure the safety of the software system or application the software developers need to test their data recovering strategies. Some software developers or testers feel this doesn’t applies to all programs because they conducted resilience testing when the software system or applications were put in place.

But one should always keep this in mind that the testing environment, the testing strategies and the range of cost effective solutions and tools available are always changing. It is required to keep pace with all these changes.

- The resilience testing strategies need to be tested and reviewed frequently in order to update for these changes.
- Some software developers and testers fear about the time and cost of test cases that give a better grade of tests and hence they are not able to put their good intention in to the practice and hence there remains a lack of resiliency in the software system or application.
- This does not necessarily means that each and every available test case should be implemented for testing the software system or application.

- There should be test plan for carrying out the resilience testing.
- A structured methodology always ensures that the amount of time consumed is minimum and the effectiveness of the testing is maximum.
- Resilience testing is some what similar to stability testing, fail over testing or recovery testing.
- Resilience testing is aimed at determining the behavior of the software system or application in the case of unreliable events, catastrophic problems and system failures, crashes and data losses.
- Resiliency is one of the core attributes of a good and reliable software system or application.
- Any software or hardware malfunctioning or failures are likely to have a considerable impact on the software system or application.

A software system needs to resilient against the following:
- Changes in requirements and specifications of the system.
- Hardware and software faults.
- Changes in data sources.

Resilience needs to be incorporated in the following stages of software development:
- Software design
- Hardware specification
- Configuration
- Documentation
- Testing

What are different characteristics of compatibility testing?

First let us clear up with the concept of compatibility.

Compatibility of a software system or an application or any hardware system or components can be defined as the ability of that software system or hardware component to work efficiently with all the versions either newer or older of all CPU architecture designs and operating systems.

- Compatibility is one of the most important properties of any software system or application and hardware system.
- Not every where, every one uses the same CPU architecture designs and operating systems.
- Therefore, it becomes necessary to make software and hardware compatible will all sorts of systems available.
- Otherwise, the software or the hardware will remain confined to only one CPU architecture and operating system.
- There will be no benefit to the software and hardware developers who developed that software or hardware.

In other words, the software or the hardware product or artifact get promotion and hence it would not be widely accepted.
Keeping in view all these issues, care is taken to provide maximum compatibility to the software or the hardware system. Before its release to the open public, the software or the hardware product needs to undergo testing for determining its compatibility.
Such testing is called compatibility testing.

- Compatibility is categorized under non functional testings.
- Compatibility testing can be defined as the testing that is conducted on the software application or the hardware component to determine the concerned product’s compatibility with the computing technological environment.

A proper computing technological environment contains all of the below mentioned aspects:

- Bandwidth handling capacity: the environment should be able to handle bandwidth of the networking software and hardware.
- It should have a computing capacity of hardware platform like HP 9000 and IBM 360 etc.
- It should be compatible with all kinds of peripherals. In other words, it should have compatibility of peripherals. Peripherals include DVD drive, printers, monitors, speakers and so on.
- It should be compatible with all operating systems including UNIX, MVS, windows and so on.
- It should support all types of data bases like oracle, DB2, Sybase etc.
- It should be well compatible with other software systems like messaging tools, networking systems and web servers.
- It should be browser compatible. It should support all the available browsers till date like Netscape, internet explorer, Firefox, Google chrome, safari etc.

The above aspects together make up a proper and efficient computing environment to carry out compatibility testing for the software and hardware products.

Compatibility testing comprises many other small testings like peripheral compatibility testing, browser compatibility testing etc. browser compatibility testing is also known as user experience testing.
It involves the checking of web and network applications on all the available different browsers.

It is done to ensure the following:
- The application under testing should respond exactly in the same way with all the different browsers. It should exhibit same features and functionalities when run under different browsers.
- The visual experience for the users must be the same irrespective through which browser they are using the web application.
- The applications should be backwards compatible i.e., it should work with the older versions of the browser as well.
- The applications should be carrier compatible i.e., data transformation should be same no matter which carrier is being used. Some famous carriers include orange, sprint, Verizon, Airtel, O2 etc.
- Apart from software compatibility, the application should be hardware compatible.
- The applications should be compiler compatible i.e., there should be no difference in compilation by different compilers. All the compilers should compile the source code correctly.
- The applications should be able to run on emulators.

What are different characteristics of non-functional testing?

Non functional testing can be defined as the testing of the software system or the application for its all non functional requirements. It happens many times that the names of the non functional requirements get overlapped during the non functional testing.

Therefore, the name so the non functional requirements are used interchangeably many times. Non functional testing includes various testing aspects. Few have been given below:
- Compatibility testing
- Baseline testing
- Compliance testing
- Endurance testing
- Documentation testing
- Internationalization testing
- Localization testing
- Load testing
- Recovery testing
- Performance testing
- Security testing
- Resilience testing
- Scalability testing
- Usability testing
- Volume testing
- Stress testing

In non functional testing, typically the software system or the application is tested against the specifications and conditions listed by the client or the customer.
Non functional testing is entirely based upon the test cases and specifications and the requirements stated by the customer or the client.

Apart from the above listed non functional requirements aspects there are several other aspects covered by the non functional testing. They have been listed below:
- Ergonomics testing
- Migration testing
- Data conversion testing
- Penetration testing
- Installation testing
- Operational testing
- Readiness testing
- Application security testing
- Network security testing
- System security testing

Non functional testing is carried out only after the functional testing has been successfully completed on the software system. Earlier, non functional testing was not considered important. But, eventually software engineers started realizing its importance and they began to concentrate on the extra external feature of the whole software and the hardware system like reliability, interface testing and configuration testing. So, the non functional testing came to the scene and has got a great importance in the field of software development.

Comparison between functional and non-functional testing
- Functional testing gives coverage and concentrates on the relevant functionality or part of the software system or the application whereas non functional testing focuses primarily on the irrelevant components or functionality of the whole software system. The best example is given by graphical user interface or GUI which is also known as look and feel testing.

Generally non functional requirements include performance, volume, stress, load, and security, recovery etc and so on.

- Manual testing as an aspect of non functional testing relates to the testing of functionality which are simply not mentioned in the functional requirements documentation. This is how manual testing and non functional testing is related to each other.

- Functional testing relates to the business documents, requirements documents, test plans and exact mapping of the variables. Non functional testing is in great contrast with the functionality testing.

- Functional testing deals with the stability of the software system or the application. For example, it checks how the software system or the application responds to the irrelevant data input for some function. Non functional testing is a kind of ad hoc testing. But it also checks the stability.

Non functional testing tests the requirements of a software system that do not relate to the functionalities of the software system or application. These non functional requirements include fail over and recovery, social-ability, performance, stress, security and so on.

Non functional testing is to test the features and specifications and the stability of the software system which don’t correspond directly to the business related functionalities i.e., functional requirements, but are equally necessary.
In non functional testing you can look and feel the objects. For example, the installation testing.

What is the difference between functional and non-functional testing ?

Functional testing is a kind of testing whose cases of tests are based on the specific conditions of the component of the software that is under test. Inputs are fed to the function and the resulting output is examined. In this type of testing, the internal structure of the software source code or the program is not known. Functional testing can be considered as a kind of black box testing. Functional testing checks a program against specifications of the program and design of the documents. This kind of testing involves basically 5 steps.
The first being the identification of the functions expected to be performed by the software.
The second step involves the creation of data for input according to the specifications of the functions.
In the third step the output after processing of data is determined according to the specifications of the functions.
The fourth step deals with the execution of the cases of the tests.
The actual output is compared with the expected output in the last and final fifth step of the functional testing.

Functional testing forms a crucial part in software processes. What all does functional testing do ?
Functional testing is used to check performance.
Functional testing tests the graphical user interface (GUI) of the program.
Functional testing requires checking the whole program from one end to the other end.
This helps to ensure performance and quality.
It helps in developing programs faster.
Errors also result during functional testing which should be corrected with supporting unit tests.
Later they should be checked once more and verified with a functional test.
Functionality tests have got a great benefit. Many correct tests means the program is correct and working.
Functional testing increases the probability of automatically checking the program.
Functional testing is used to validate the whole input output conversion process.

To make functional testing more effective a process called “continuous integration relentless testing” should be implied. Functional testing is done according to business requirements as stated by client’s specifications. The various aspects of functional testing are listed below:
1. Smoke testing
2. Sanity testing
3. Unit testing
4. Top down testing
5. Bottom up testing
6. Interface testing
7. Usability testing
8. Regression testing
9. Alpha and beta testing
10. System testing
11. Pre user acceptance testing
12. White box testing
13. Black box testing
14. User acceptance testing
15. Globalization testing
16. Localization testing
In contrast to functional testing, the non- functional testing of a software application tests it for non-functional features and requirements. Sometimes there is an overlap of scope of various non functional tests amidst many non- functional requirements. Therefore, in such cases the names of the non functional tests are interchanged to avoid confusion. There are various aspects of non functional testing which have been listed below:
1. Documentation testing
2. Compliance testing
3. Baseline testing
4. Compatibility testing
5. Recovery testing
6. Localization testing
7. Performance testing
8. Endurance testing
9. Load testing
10. Internalization testing
11. Volume testing
12. Stress testing
13. Usability testing
14. Resilience testing
15. Scalability testing and
16. Security testing
In addition to the above mentioned aspects, the non- functionality testing also covers the following additional features:
1. Load testing
2. ergonomics testing
3. migration testing
4. penetration testing
5. data conversion testing
6. operational testing
7. network security testing
8. system security testing
9. installation testing
There’s a basic difference in both types of testing which is that the functional testing tells one what software does whereas non functional testing shows one how well the software executes.