Software Testing Automation refers to the use of test support tools to automate software testing tasks or parts of it. These can be test case design, test execution and results confirmation, test progress management, report creation and so on.

Testing is also essential for quality assurance in system development projects. As a rule, the testing happens once during the testing phase and there is no need to repeat it.

However, it becomes difficult to repeat testing if you fix something on a system that has entered the maintenance/operation phase. For example, a bug fix or a specification change. Another case might be when you have to develop/renovate a back-end system for providing services for BtoC in a fast release cycle. Of course, it is necessary to verify that the existing functions do not have problems caused by the modification. Also, it is equally important to verify that the developed/reworked contents work correctly.

In other words, the industry is in a situation or shorter release cycles along with reduced time for testing. On the other hand, the need to repeat the same test has been significantly increased. Therefore, software testing automation is popular lately.

Why bother with software testing automation?

1. Short delivery time for projects

“Delay in system development = loss of business opportunities” 
The project for system development has been increasing in short delivery type, such as 3-6 months, and a slight delay will lead to major damage.

2. Small budget

Each new model comes with more features and more test items, but the budget for that is not added as easily. Not only new functions but even the existing functions have to be tested in order to confirm the integrity. Also, the resources might be scarce.

3. Changes in development methods

Now that the business environment for software development is drastically changing. Presently, there are also cases where the specifications are not decided. In order to suit the development method, there are new SDLC models too.

4. Quality

News coverage on software quality, such as system failures, continues unabated. Awareness of quality is growing.

As such, it is required to develop large and complex systems in a short period of time, at a low cost, and with good quality. Automation tools are drawing attention as one of the means to solve such requests.

Advantages of introducing software testing automation

Broadly speaking, there are three advantages. They are as follows:

1. Improve test efficiency and reduce man-hours
2. Eliminate human error in test work
3. Realize test tasks that could not be achieved manually

Advantage 1: Improve test efficiency and reduce man-hours

1. More time for developers to spend on development

The ability to run automatic software tests can significantly reduce the amount of manual testing. Since developers (engineers) will be able to spend time on the development work itself, it will be possible to “use the time to improve the quality” as well as “increase the time for the development effort”.

2. It can be repeatedly executed even if the testing takes a lot of time

Complex scenarios and tests that take time to prepare can be repeated many times by automating tests.

3 Run tests in real time

By combining a continuous integration (CI) environment with test automation, it is possible to detect that the source code of a system or program has been tampered with and automatically execute tests at that time.

Advantage 2: Eliminate human error in test work

There is a limit to human concentration. Losing focus can lead to omissions and oversights in testing. Test automation is the best way to avoid these mistakes.

With the introduction of test automation tools and test automation with RPA tools, human errors can be eliminated by the machine performing testing operations accurately and repeatedly.

Advantage 3: Realize test tasks that could not be achieved manually

For example, in the case of “trying tens of thousands of simultaneous accesses”, it is virtually impossible to carry them out manually. Or, it is too inefficient to test after 48 hours actually to check “the behavior when clicking a link after 48 hours after receiving mail”.

Load tests and irregular tests are essential for testing back-end systems for BtoC services that are likely to be widely used by users, so test automation is effective when performing such tests as easily as possible. It is to do.

To execute these tests, not only test code and test tool automation but also test test test with RPA tool (link to RPA article) that can record and reproduce user’s operation can be considered effective method.

History of Software Testing Automation

1960s

The method of test automation is considered early in 1962.

• Software crisis
  • At the end of the 1960s, the talks at the 1st NATO Software Engineering Conference called for some kind of standardization systems. This was to improve computer performance as well as to accelerate software development.
  • Software development had become increasingly complex since this time. The methodologies for software development such as structured programming and automatic testing were debated.

1970s-80s

• Software Testing Techniques [1976]
  • The complexity of software kept increasing. As a result, methods for testing and improving the quality of software started to emerge.
• Object-oriented programming
  • There was a need to cope with the complexity of software. So the object-oriented way of thinking about programming began to prevail.
• The Mythical Man-Month: Essays on Software Engineering
  • A famous book on software development. It basically says that there is no universal way to do things in software development. “Brook’s Law” is quite famous too.

“adding human resources to a late software project makes it later”

-Fred Brooks (The Mythical Man-Month, 1975)

1990s

• Extreme Programming (known as XP) [1999] The method itself is from around 1996
  • A book of development methods written by Kent Beck, famous as a developer of Smalltalk. It lists the main ideas of modern automated testing.
• Scrum
  • One of the methods of Agile development. It is quite popular in companies lately.
  • A method of creating a product in which a Scrum master, a product owner, and a team cooperate to create things in ‘sprints’ in a short period of time.
  • The Scrum masters will work together to assist in the implementation of Scrum, product owners in the business requirements and responsibilities for products, and teams in responding to the business requirements from product owners.
  • Although there is no detailed description of test automation in the development method advocated at the same time as XP, test automation is often introduced basically as a method for developing in a short period of time.

2000s

• Lean startup
  • Strictly speaking, it is not a software development method, but a methodology of entrepreneurship, but a development method often used in startup companies. One of the agile development methods. A method of creating a minimum practical product MVP (Minimum Viable Product), repeating hypotheses and verifications, and improving the product. There are many areas where repetitive tasks and methodologies are common to XP and Agile methods.
• DevOps
  • A combination of development and operations in developing language that combines development and operations personnel. A development method in which operation, infrastructure, and development work seamlessly together. The introduction of test automation is proposed along with infrastructure automation.

Types of Software testing that can benefit from automation

The following may be used as phases of testing. It is also common for many companies to classify these as types of testing.

Unit testing

• Testing to confirm that the functionality of each module is satisfied.
• Commonly known as UT (Unit Testing).
• It is basically done as a white box testing.
• Test to make sure your code works the same as the developer’s intentions
  • Note: It does not secure your business requirements
• Create tests for each method.

Advantages

• By writing test code, developers can further understand the functional scope of each module etc.
• Fine-grained, you can analyze in detail where the bug occurred.
• You can create test code that expresses the function of a method, and developers can read the test code and read the intent of the method later.

Disadvantages

• Because of the required completeness, the logic content increases the cost of writing test code
• Tightly coupled code makes it difficult to write test code and loosely coupled tests are likely to require code that is easy to write

Integration Testing

• Connecting and testing of each module.
• Commonly known as IT (Integration testing), also known as combined testing.
• Related to functional testing.
• In some cases, tests for external connections such as external APIs are performed in stages.
• Basically, it is a type of black box testing. However, it might even be tested as a white box.

Merits

• Since we can link and test everything, we can confirm the range of what works and what doesn’t to a large extent.
  • As we can confirm in an environment that is similar to production, we can confirm external factors such as browser environment dependency.
• Demerit
  • Because the scope of confirmation is large, any complex code might be difficult to maintain.
  • As it is black box testing, it might be difficult to understand where the bug occurs.

Comprehensive testing

• Test that combines all conditions such as external API and browser.
• Prepare and test an environment similar to the production environment. Here we check if the program works according to the requirements.
• UAT (User Acceptance Testing)
• It is a type of black box testing.

Functional and Non-Functional Testing

There are roughly two types of testing scenarios based on viewpoint: functional requirements and non-functional requirements.

Functional Testing

• The part that meets client requirements.
• Check if the code you have created meets the functionality your client requires. Also, check that it works correctly.
• The part that can be secured by the test code is basically this part.
• We also perform tests on the parts linking each module (external API or DB part).
• Testing including multiple modules together. This includes whether the program meets the requirements.

Advantages

• You can find bugs caused by the binding condition of each module that you do not know by itself.
• API etc. can be checked up to the level of requirements, not the logic of a single program.

Disadvantages

• Because the test granularity is large, it is very difficult to identify the bug occurrence point.
• Since the test content also requires business logic, requirements, and external communication IFs, it is necessary to check simple code content + understanding of requirements, and the scope that must be understood is expanded.

Non-functional testing and requirements

Parts other than specifications or functions.

• Information security
• performance
  • Server performance
• Network design
• How to recover from a failure
  

Testing processes

• Testing analysis
• Test design

Test cases

You might want to write test cases for long-standing projects. Especially those projects that need maintenance. Another case where test cases are useful is when there is a project with a lot of changes from a lot of people.

Why write test cases

• To lower the cost of testing.
• By making it easy to test code, you will be aware of loosely coupled programs.
• The test code itself is a specification for the programmer.
• By thinking about the test code, you will deepen your understanding of the requirements.
• The importance of test code is generally this: Unit Testing → Functional Testing → Integration Testing.

Effect of test cases

• Reduce stress on programmers.
• Programming can be a stressful task.
  • You have to understand the specifications.
  • Write in consideration of various things such as abnormal system as well as a normal system.
  • You have to think about how to express in the program in the first place.
  • If the specification changes, you have to redo it all over again.
  • You also have to think about performance and readability.
  • Good specifications make it easy to accommodate changes.
• The programmer has to write a program with various considerations
• It’s very difficult to write all the code in one go. There is also a possibility of specification changes.
• We do not know where a program breaks without test cases.
• Write test cases.
  • → It makes it easy to see where errors can happen during refactoring.
  • → Reduce stress on programmers.

About the cost of test cases

• The cost of writing test cases should be as fast as accustomed to half of the implementation, and as new to an unaccustomed person as the implementation or 1.5 to 2 times the implementation.
• The test cases also have a maintenance cost that changes according to the code of the main unit.
• Cost of introducing test case environment, learning how to write test cases themselves.

Summary

At the site of recent system development, projects with short delivery times are increasing. In order to make a project on a short lead time a success, quick feedback to developers (engineers) is essential.

Software testing automation has become an important keyword in order to make testing as fast and accurate as possible.