In the realm of software development, test automation has become an essential component of the development lifecycle. It promises speed, efficiency, and consistency, making it an attractive option for teams aiming to deliver high-quality products faster.
But as a test automation engineer, have you ever been asked by the higher management or stakeholders, “Why aren’t all the tests automated?” or “Why are the automation scripts unstable?”. Have you ever felt that no matter how valid your explanations are, they might not fully satisfy them? And have you ever questioned whether every test should (not CAN) indeed be automated?
The notion that every test should be automated is a common misconception. While automation offers many benefits, it’s crucial to recognize that not all tests are suitable for automation. And even though all tests can be automated, that doesn’t really mean they all should be automated. Let’s explore why it’s essential to be selective about what tests to automate and highlight scenarios where manual testing remains irreplaceable.
Limits of Test Automation
- Initial Setup and Maintenance Costs
Automating tests is not a one-time investment; it requires significant initial setup and ongoing maintenance. Complex test scenarios can be time-consuming to script, and maintaining these scripts in response to application changes can be labor-intensive. In some cases, the time and resources spent on automating a test may outweigh the benefits, especially for tests that are infrequently executed or likely to change frequently. - Human Insight and Exploratory Testing
Automated tests are excellent for validating predefined scenarios and catching regressions. However, they lack the human intuition and creativity required for exploratory testing. Exploratory testing allows testers to interact with the application in real-time, uncovering issues that automated scripts might miss. This type of testing is invaluable for identifying usability issues, understanding the user experience, and discovering edge cases. - Tests with High Variability
Certain tests, such as those involving UI elements, can be highly variable and sensitive to minor changes. Automated UI tests can become brittle and prone to false positives, leading to frustration and wasted effort. In such cases, manual testing may be more efficient and effective, allowing testers to adapt to changes on the fly and provide accurate feedback. - Short-Lived Projects or Features
For projects or features with a short lifespan, investing in automation might not be justifiable. If a feature is expected to be deprecated or significantly altered soon after release, the time spent on creating and maintaining automated tests may be better utilized elsewhere. In these scenarios, manual testing can provide the necessary coverage without the overhead of automation.
Balancing Automation and Manual Testing
To achieve an optimal testing strategy, it’s essential to strike a balance between automation and manual testing. Here are some guidelines (in my perspective) to determine which tests to automate and which to leave for manual execution.
- Automate Repetitive and High-Risk Tests
Focus on automating repetitive tests that need to be run frequently, such as regression tests and smoke tests. These tests benefit the most from automation due to their repetitive nature and the need for consistent execution. - Prioritize Stable and Critical Paths
Automate tests for stable and critical parts of the application. These areas are less likely to change frequently and have a significant impact on the application’s functionality. - Leverage Manual Testing for Exploratory and Usability Testing
Use manual testing for exploratory testing, usability testing, and scenarios requiring human judgment. These areas benefit from the creativity and intuition of skilled testers. - Assess ROI for Each Test
Evaluate the return on investment (ROI) for automating each test. Consider factors such as the frequency of test execution, stability of the test environment, and the complexity of test scripting and maintenance.
While looking for some way to determine which tests should be automated and at what priority; I decided to create my own technique to do so.
Automation Priority – 1 (Must be automated)
Automating tests for frequently used and critical application features is essential for ensuring the reliability and performance of the software. These features often form the core functionality of the application and directly impact user satisfaction and business operations. Here are examples of such features.
- User Authentication (Login and Registration)
Context: The login and registration processes are fundamental to user access and security in most applications.
Reason for Automation: Ensuring that users can reliably log in and register is crucial. Automated tests can verify different scenarios, including successful logins, failed logins, password resets, and account creation with valid and invalid data. - Shopping Cart and Checkout Process
Context: For e-commerce applications, the shopping cart and checkout process are vital for completing transactions.
Reason for Automation: Any issues in the checkout process can lead to lost sales and frustrated customers. Automating tests for adding items to the cart, applying discounts, calculating totals, and completing payments ensures that these critical paths function correctly. - Search Functionality
Context: Search features are widely used in applications to help users find content, products, or information.
Reason for Automation: Automated tests can validate that search results are accurate and relevant, handle different search queries, and check for performance under various conditions. This ensures users can efficiently find what they need. - Data Input and Form Submission
Context: Forms are used for various purposes, such as data entry, feedback, and contact forms.
Reason for Automation: Ensuring that forms can handle various types of input, validate data correctly, and submit without errors is crucial. Automating these tests helps maintain data integrity and user experience. - Payment Gateway Integration
Context: Applications that involve financial transactions need reliable payment processing.
Reason for Automation: Automated tests can verify that payments are processed correctly, including handling of different payment methods, successful and failed transactions, and security checks. This is essential for maintaining user trust and revenue flow. - User Profile Management
Context: Users frequently update their profiles, including personal information, settings, and preferences.
Reason for Automation: Automated tests ensure that profile updates are saved correctly, data is validated, and changes are reflected immediately. This helps maintain user satisfaction and data consistency. - APIs and Data Integration
Context: Many applications rely on APIs for data exchange and integration with other services.
Reason for Automation: Automated tests for APIs ensure that data is correctly sent and received, endpoints are responsive, and integrations function seamlessly. This is crucial for maintaining application functionality and performance. - Load and Performance Testing
Context: Applications need to perform well under various load conditions to provide a good user experience.
Reason for Automation: Automated load and performance tests can simulate different user loads, identify bottlenecks, and ensure the application can handle peak traffic without degradation. This is essential for scalability and reliability. - Security Testing
Context: Security is paramount for protecting user data and application integrity.
Reason for Automation: Automated security tests, including vulnerability scanning, SQL injection tests, and authentication checks, help identify and mitigate potential security risks. This is critical for compliance and user trust. - Continuous Integration and Deployment (CI/CD)
Context: CI/CD pipelines are essential for delivering updates and new features quickly and reliably.
Reason for Automation: Automated tests within CI/CD pipelines ensure that new code changes do not introduce regressions or break existing functionality. This allows for faster, more reliable releases and higher software quality.
Automation Priority – 2 (Must be automated)
Automating tests for application features that are least used but critical is essential to ensure the stability and reliability of the application. These features, while not frequently accessed, often have a significant impact on the application’s functionality and user experience. Here are some examples.
- Password Reset Functionality
Context: Password reset is crucial for user access and security.
Reason for Automation: If this feature fails, users might be unable to access their accounts, leading to frustration and potential loss of trust. Ensure various scenarios such as expired tokens, invalid tokens, and security questions (if applicable) works perfectly. - User Role and Permission Management
Context: Proper management of user roles and permissions is essential for maintaining security and ensuring users have the appropriate level of access.
Reason for Automation: Verify that users with different roles have correct permissions, including creating, editing, deleting, and viewing resources. Ensure that unauthorized actions are correctly restricted and that permissions changes are properly reflected across the application. - Payment Processing
Context: Payment processing is critical for e-commerce and subscription-based applications.
Reason for Automation: Failures in this area can directly impact revenue and user satisfaction. Tests various payment scenarios, including successful transactions, declined payments, refunds, and handling of different payment methods (credit cards, PayPal, etc.). Verify integration with third-party payment gateways and ensure proper error handling. - Backup and Restore Functionality
Context: Backup and restore functionalities are vital for data integrity and disaster recovery.
Reason for Automation: Verify that backups are created correctly and in case of data loss or corruption data can be restored accurately. Test various backup scenarios, including full and incremental backups, and ensure that the restore process maintains data integrity. - Security Features (e.g., Two-Factor Authentication)
Context: Security features like two-factor authentication (2FA) add an extra layer of protection to user accounts. Failures in these features can compromise account security.
Reason for Automation: Ensure that 2FA works as expected, including the generation and verification of authentication codes, handling of lost devices, and backup authentication methods. Test both successful and failed authentication attempts. - Audit Logging
Context: Audit logging is essential for tracking user activities and changes within the application, which is crucial for compliance and troubleshooting.
Reason for Automation: Verify that audit logs capture all necessary actions, including user logins, data changes, and administrative actions. Ensure that logs are accurate, complete, and securely stored. - Localization and Internationalization
Context: For applications with a global user base, proper localization and internationalization ensure that users can interact with the application in their preferred language and format.
Reason for Automation: Verify that all UI elements, dates, times, and currencies are correctly localized. Test multiple languages and regions, ensuring that translations are accurate and formatting is consistent.
Automation Priority – 3 (Can be ignored from automation; Automate if ROI is justified)
It’s generally recommended to automate tests for features that are most frequently used, as these are often critical to the application’s core functionality and user experience. However, there may be some exceptions where automated testing may not be necessary or practical despite high usage. These kind of scenarios should be ignored from automation and can be automated only if ROI is justified. Here are examples of such features.
- Basic Content Verification (Static Texts and Images)
Context: Applications often display static content such as company information, contact details, or promotional banners.
Reason for not Automating: Static content rarely changes and doesn’t involve complex interactions. Manual checks can ensure that the content is correct and properly displayed, making automation unnecessary for these elements. - Simple Form Validations
Context: Forms that collect basic information (e.g., contact forms or newsletter sign-ups) often have straightforward validation rules.
Reason for not Automating: While these forms are used frequently, their validation logic is usually simple and can be quickly verified manually. The effort to automate these tests may not be justified given their simplicity. - Visual Design and Layout Checks
Context: Ensuring the correct layout and visual appearance of the application across different devices and screen sizes is crucial for a good user experience.
Reason for not Automating: Visual design checks often require human judgment to assess aesthetics and user experience. Automated visual regression tests can help, but manual reviews are often more effective for nuanced design elements. - Text Input Fields
Context: Applications with multiple text input fields, such as search bars or comment sections, are heavily used.
Reason for not Automating: While these fields are frequently used, they typically involve straightforward interactions that can be easily checked manually. Automated tests might not add significant value compared to the manual effort required. - User Navigation and Basic Link Checks
Context: Ensuring that all links and navigation paths within the application work correctly is essential for user experience.
Reason for not Automating: Manual testers can quickly check navigation paths and links. While automated tools can crawl links, human testers are better suited to assess the context and relevance of navigation flows. - Non-Critical Notifications and Alerts
Context: Applications often display notifications or alerts for non-critical information, such as reminders or tips.
Reason for not Automating: These notifications are not crucial to the application’s core functionality. Manual testing can ensure that they appear as expected without the need for extensive automation. - Minor UI Elements (Icons, Tooltips)
Context: Icons, tooltips, and other minor UI elements are part of the user interface but often do not impact core functionality.
Reason for not Automating: Manual checks can efficiently verify the presence and correct functioning of these elements. The effort to automate these tests might outweigh the benefits, especially if the UI elements are stable and rarely change.
Automation Priority – 4 (Should not be automated)
Some application features are both rarely used and not crucial enough to warrant the time and resources needed for automation. These kind of scenarios should not be automated. Here are examples of such features.
- User Profile Customization
Context: Many applications allow users to customize their profiles with features like setting a profile picture, changing themes, or updating bio information.
Reason for not Automating: These features are often not mission-critical and change infrequently. The potential impact of defects is usually minor, and manual testing can cover these without significant overhead. - Help and FAQ Pages
Context: Help and FAQ sections are essential for user support but are rarely changed once created.
Reason for not Automating: These pages typically involve static content that doesn’t require frequent validation. Manual checks can ensure that the content is accurate and displayed correctly, without the need for automation. - Social Media Sharing Options
Context: Applications often provide buttons for sharing content on social media platforms.
Reason for not Automating: While useful, social media sharing is not critical to the application’s core functionality. Changes to these features are infrequent, and manual testing can verify the functionality when necessary. - Rarely Used Settings and Preferences
Context: Applications may offer extensive settings for personalization, such as font size adjustments, color schemes, or notification preferences.
Reason for not Automating: These settings are typically set once by users and rarely changed. Their impact on the application’s core functionality is minimal, making manual testing sufficient. - Legacy Browser Support
Context: Some applications may need to support older browsers for a small subset of users.
Reason for not Automating: The usage of legacy browsers is declining, and the cost of automating tests for these browsers may not be justified. Manual testing can ensure basic compatibility without extensive automation effort. - Accessibility Features for Non-Core Audiences
Context: Features designed to enhance accessibility, such as screen reader compatibility or keyboard navigation, might be used by a small segment of the user base.
Reason for not Automating: While accessibility is important, specific features catering to niche user groups can often be validated manually to ensure they meet standards without comprehensive automation. - Seasonal or Promotional Features
Context: Temporary features related to holidays, promotions, or special events.
Reason for not Automating: These features are short-lived and often not central to the application’s primary functions. Manual testing ensures they work correctly during their active period without the need for long-term automation maintenance.
To summarize, we should and must automate the critical features irrespective of their frequency of use. We may or may not automate less critical but most used features. We should absolutely avoid automating less critical and least used features.
The examples given for each category is not applicable for all application types. Few of the examples may belong to other categories based on the domain and context of the application under test. But the intention of this blog is to categorize each test scenarios in to 4 quadrants mentioned above, and then make an informed decision to automate the tests which should be automated instead of can be automated.