What is the possible test approach for simulation system ?

A simulation system's primary responsibility is to replicate the behavior of the real system as accurately as possible. Therefore, a good place to start creating a test plan would be to understand the behavior of the real system.

- Subjective Testing:
It mainly depends on an expert's opinion. An expert is a person who is proficient and experienced in the system under test. Conducting the test involves test runs of the simulation by the expert and then the expert evaluates and validates the results based on some criteria. Advantage of this approach is that it can test those conditions which cannot be tested objectively. Disadvantage is that the evaluation of the system is based on the expert's opinion which may differ from expert to expert.
- Objective Testing:
It is mainly used in the systems where the data can be recorded while the simulation is running. This testing technique relies on the application of statistical and automated methods to the data collected.
Statistical methods are used to provide an insight into the accuracy of the simulation. These methods include hypothesis testing, data plots, principle component analysis and cluster analysis.
Automated testing requires a knowledge base of valid outcomes for various runs of simulation. The knowledge base is created by domain experts of the simulation system being tested. The data collected in various test runs is compared against this knowledge base to automatically validate the system under test. An advantage of this kind of testing is that the system can continually be regression tested as it is being developed.

Automation in Software Testing

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test. There can be two types of testing manual and automated. The automation of software testing is becoming more and more popular but at the same time manual testing cannot be ignored. In this article we shall be mostly focusing on the merits and demerits of automated software testing. As chopping and adding of requirements is nothing new in today’s software business and the testing window is getting smaller there is a realization of a greater need for test automation.
The sad part about automation testing is the expectations that the managers have from these testing models. It’s a general belief that automation can help find more bugs which is not true. The efficiency of the test scripts is solely dependent on the efficiency on the test cases that make the test scripts. People expect that since they have introduced automated testing, they can do away with or at least reduce the manual testing which is a “BIG MISTAKE”. As already mentioned it is the test cases that define the efficiency of the automated test scripts and these test cases are written by these manual testers hence if you think of putting away with them the customers might also think of putting away your product.
Continuing regarding the unrealistic expectations that people have with automation testing, part of the blame for these expectations goes to the fact that when vendors give demos of these products they only tell you what you want to hear and not the reality about the difficulties faced when you try to use the tool for your application. Proper planning is essential when you go for the selection of any automated tool. It is very important that the people who are actually going to use the tool get a hands-on experience of the tool. This might sometimes be impractical because of the tight work schedules and time bound deadlines but if applied it’s a very useful technique for the selection.
Many test automation tools provide record and playback features that allow users to interactively record user actions and replay them back any number of times, comparing actual results to those expected. This approach can be applied to any application that has a graphical user interface. However if the developers continuously keep changing the GUI even when it’s not called for it indicates towards lack of process in place and hence the automation test will have to be configured again and again.
People have to be educated about the advantages and limitations of automation testing techniques. It is important that pitfalls of automation testing are properly evaluated to avoid inconvenience at a later stage. It should be taken care of that the selected scripts are compatible to your applications. There is no doubt that automation testing is an asset in the armory of the testing team but without proper knowledge and understanding it can even turn out to be a negative catalyst.

Some reasons why test automation projects fail, and some precautions to take to reduce these chances

One of the most important reasons why test automation projects fail is because not enough planning is give for such projects. In so many organizations, people talk about automation, decide that bringing in software automated testing will solve things for them. One of the prime reasons is that there is a drive towards bringing in software testing automation as a solution, looking at the benefits, but not keeping in mind that an organization needs to also plan carefully for software automation.
So, for example, when a company brings in software automation, it would need to do the following steps, else, it will face problems during the implementation, and could actually reach a situation where the organization considers that the project is a failure.
- Plan the required resources that you need for implementation of software automation, which can be different from the profile needed for testing purposes
- In the initial stages, the amount of effort needed for software testing automation would be more, since the company needs to keep at the regular manual testing and also start building up the effort needed to do the automation of its test cases and test plans
- In many cases, there is a piecemeal implementation, in terms of doing conversion of cases one by one to become automated, but a comprehensive framework is not employed for this purpose. Without this level of framework implementation, as the amount of automation increases, the maintenance of such test cases becomes more complicated and intensive
- In many cases, the organizations have not really thought through the needs for implementation in terms of creating a new sub-structure within the testing team for automation, and this creates strains since the black box testing and automation teams have different needs
- For doing automation, there is a need to even modify existing testing processes (including the plan for creating test plans and test cases), so that creating automation becomes an intrinsic part of the testing process
- Set expectations of the executives with respect to the time frames and effort needed for automation; I know a case where the senior manager wanted the implementation of an automation project without increases in resources and without existing testing getting impacted. This is an impossible task.

Test Automation Framework: What is Data Driven testing (definition and more) ..

In the previous posts, we have been talking about different testing automation frameworks. In the current post, we talk about another test automation framework, called 'Data Driven Testing', including definition and some details.
In this framework, the variables are used for testing of both the output verification values and input values. These values are read from data files (different kind of data objects such as datapools, ODBC sources, csv files, Excel files, DAO objects, ADO objects, and such), and are then loaded into variables (and these variables could be used in scripts that are either manually written or recorded). The test script in turn is supposed to take care of all the process of moving through the application, opening and reading of the data files, and logging of test results.
This may sound similar to the table driven testing (and it is similar in the sense that the test case is contained not in the test script, but in a data file), with the script just being used for moving through the workflow (navigating through the application). The difference is that in this case the data is stored in data objects, not in tables (and the navigation is not stored in the data).
What are some of the advantages of using this automation framework ? There is a reduction in the number of scripts you need for your overall test cases, and if you have a need to accommodate bugs in your workflow, then this framework is the one to use. It is also very handy in terms of effort required for maintenance.

Test Automation Framework: What is The Test Library Architecture Framework (including definition)

In the previous 2 posts on the subject of Test Automation Frameworks (Keyword Driven / Table Driven, Test Script Modularity), we have covered 2 of the models used for test automation frameworks. In this post, we will cover another test automation framework called 'Test Library Architecture'.
The Test Library Architecture is similar to the Test Script Modularity framework using the same level of abstraction; the difference being that the application being tested is broken down into functions and procedures (or you could break it down into objects and methods rather than scripts). The tester will need to create libraries (in the form of SQABasic libraries, APIs, DLLs, and such) that are supposed to be representing the modules, and functions of the application; once these are created, they are called directly from the test case script.
One of the advantages of this framework is that it provides a high degree of modularization, and increases the ease of maintenance of the automation test case. You can simply modify the script if a control is changed by modifying the library file.

Test Automation Frameworks: What is Keyword-driven/table-driven testing (including definition)

For a few of the last posts, we have been looking at more details about test automation, including the benefits of automation, some scenarios in which we should not use automation, and the scenarios in which we should use automation. In addition, we started discussions around the use of test automation frameworks and how they prove to be more beneficial than just creation test cases as you do recording of your test scenarios. In this post, let us consider one of the test automation frameworks, based on "Keyword-driven/table-driven testing".
'Keyword driven testing' and 'table driven testing' seem like 2 different sets of words, but they actually are used to refer to the same method. They denote an application independent framework, which requires the development of data tables and keywords. These data tables and keywords are independent of the test automation took being used, and also independent of the test scripts used to drive the application that is being tested. These keyword driven tests look very similar to the manual test cases. When keyword driven tests are being used, a table is used to document the functionality being used, and this functionality is also mapped through step by step instructions for each test. The entire testing process is driven based on data.

Benefits of keyword driven testing:
- If the tester needs to be quickly on the job, extensive training on the tool can be done later, but at the instant of testing, the tester needs to know the keywords, and the format of the test plan.
- The Scripting language can be written by somebody who has expertise on the scripting language, and this activity can happen earlier to the test plan. The tester does not have to be bothered about the scripting process.
- A spreadsheet format can be used for writing the detail test plan.

Some problems with this technique:
- If there are a large number of keywords, then the tester needs to learn all these, and this initial effort can take time. Once done, then this is no longer a constraint.
- You need people skilled in using the Scripting language of the tool being used.

Test automation framework: What is Test script modularity (definition and some details)

In the previous post (Test Automation Frameworks), I started out by doing a short definition of Test Automations Frameworks, some benefits, and also listed out the 5 different types of Test Automation Frameworks that are currently there (and based on the number of tools available, the combination can be much more since there are many high end and complex tools available for test automation). In this post, I will talk more about one of these, which is called "Test script modularity".
A simple definition for the Test Script Modularity Framework: This framework is the most basic of the various test automation frameworks, with the ideology being towards the creation of a number of small and independent test scripts that in turn represent the modules, sections and functions of the application that is under testing. Once these scripts are created, they are then added in a hierarchical fashion to create larger tests, with the aim of creating test cases.
What is the basic principle behind this ? It is a basic principle of design to create a layer (or to be more technical, an abstraction layer) for a component. This layer in turn ensures that the component is available to the rest of the application in a way that even when the component is modified, the rest of the application is not affected. This concept is one of the key concepts of the test script modularity framework.
One of the key advantages of this framework is that it results in a high degree of modularization, and ensures that the test suite is easily maintainable. By encapsulating all components, when a component changes, you don't have to change the other components or the test cases that call this component.

What is a test automation framework and why you should go in for one ..

In the previous post (problems with manual automation testing), I had talked about why it makes sense to go in for using a strategy for test automation, and to use test automation frameworks. Working without a strategy results in issues dealing with maintenance, loss of efficiency, and ever expanding list of files.
So what is a test automation framework ? Well, if you were to take a basic description, then it is something as simple as a set of concepts, practices, tools that provide support for automated software testing; with one of the main benefits of adopting such practices resulting in lower costs of maintenance. Using a test automation framework means that the need to update a test case can be done with minimal effort, with just the actual test script needing to be updated, and everything else remaining the same. It also helps in documenting the overall set of automated test scripts, thus ensuring that even with team turnover, the process still keeps on working with minimal disturbance.
Let us take some of the test automation frameworks that are used, and then define each of these.
- Test script modularity:
- Keyword-driven/table-driven testing
- Data-driven testing
- Test library architecture
- Hybrid test automation

We will move with these definitions in the next post, since each of them needs to be covered in some detail ..

Process for creating an automation test framework and how to go about automation testing

Every automation tool will give you the ability to record a series of actions so that these can then be played back and you can create a script out of these. However, this is a very basic level of test automation, which does not provide you any flexibility. This process has its uses when you are creating a simple testing strategy whereby you have to take a few scenarios where you do not expect variations or changes, and you can record these sequences and then use them again and again. However, when you need to make changes or modify values, then the cost of doing so starts increasing tremendously. This is a time when you should evaluating the use of creating testing frameworks for you automated testing. Just the process of starting to design a test automation framework will ensure that you are starting to work through your requirements methodically, and will prevent your team from ending up in a mess (a typical example of a mess is when you have a huge number of unconnected automation scrips with their own needs for maintenance, with poor documentation leading to a disaster when people change).
How do you go about creating test automation frameworks ? Before we even go down this route, we should consider some of the benefits that you would get if you were to have an automation framework / or in the examples below, more of an automation strategy:
- You start to separate your data from tests, something that makes it easier to manipulate different types of data (very useful when you want to test the same test with a wide range of data)
- You can look towards reusing functions (building reusable functions can eventually help in saving a lot of time and make the building of these functions more efficient)
- You prevent a situation where you end up with a huge amount of test cases with high maintenance requirements; maintaining these scripts become easier (very useful when you have teams with high personnel changes or attrition rates)
- You will also start evaluating as to which stage is it practical to start building an automation test plan - in some cases, you may delay till the major elements of UI changes are done and over
- Refinements to the manual test cases are avoided later, instead the necessary adaptation of test cases to automation needs is done when they are getting generated; this saves a lot of effort later