• Skip to main content

AutomationSTAR

A 2-Day Test Automation Event

  • Programme
    • Programme Team
    • 2023 Programme
  • Why Attend
    • Get Approval
    • Testimonials
    • 2022 Photos
    • Academic Discount
    • Conference Location
    • Huddle Area
  • Exhibit
    • Partner Opportunities
    • Download EXPO Brochure
  • About Us
    • FAQ
    • Blog
    • Code of Conduct
    • Contact Us
  • Tickets

test automation

May 25 2023

Discovering Test Automation Patterns

Patterns will help you solve test automation issues. This blog is an introduction to a Wiki that collates system level test automation patterns to help with (common) automation problems. It is open to all in the test community to avail of the knowledge or to contribute more patterns.

Have you heard about Test Automation Patterns?

We are talking more than patterns for unit tests. Seretta Gamba and Dot Graham created the Test Automation Patterns Wiki that collates system level test automation patterns.  There are patterns for management, process, design and execution, and the issues that can be solved applying them. Right now, there are some 70-80 patterns and about 50-60 issues. They included a diagnostic functionality that lets you easily find the issue or issues that are bothering you and then the issue suggests what patterns to apply. The test automation patterns follow a rule of putting information in only one place. You’ll also notice a convention in the wiki: the patterns are written in capital letters and the issues in italic capitals to tell them apart. This way you can use their names almost as words in a kind of meta-language, a pattern language!

Test Automation Patterns - Dot Graham and Seretta Gamba

Test automation patterns are different from the unit test patterns because:

They are not prescriptive.

Unit-test or software design patterns take a development problem and give you exactly the code to solve it. You can implement it just as is and at the most you have to translate it into your own development language.

The system-level patterns are more generalised/vague.

Think for instance of management issues: companies can be so different in structure, size, hierarchy, etc that it would be impossible to give a single solution for every situation. That’s why many of the patterns just suggest what kind of solutions one should try out.

How does the Test Automation Patterns Wiki work?

Below is an example of you might use the wiki to help you improve or revive test automation. First, go to www.TestAutomationPatterns.org. Try following this example. When you want to improve or revive test automation, the first question will show as:

What below describes the most pressing problem you have to tackle at the moment?

  1. Lack of support (from management, testers, developers etc.).
  2. Lack of resources (staff, software, hardware, time etc.).
  3. Lack of direction (what to automate, which automation architecture to implement etc.).
  4. Lack of specific knowledge (how to test the Software Under Test (SUT), use a tool, write maintainable automation etc.).
  5. Management expectations for automation not met (Return on Investment (ROI), behind schedule, etc.).
  6. Expectations for automated test execution not met (scripts unreliable or too slow, tests cannot run unattended, etc.).
  7. Maintenance expectations not met (undocumented data or scripts, no revision control, etc.).

When several of the options may be an issue, the best strategy is to start with the topic that hurts the most. In this example, we will opt for number 2 because to be able to work efficiently you need time and resources to develop the framework. Once you click on your chosen link, a second question will appear. Second question (for “Lack of resources”)

Please select what you think is the main reason for the lack of resources:

For the following items look up the issue AD-HOC AUTOMATION:

  1. Expenses for test automation resources have not been budgeted.
  2. There are not enough machines on which to run automation.
  3. The databases for automation must be shared with development or testing.
  4. There are not enough tool licences.

More reasons for lack of resources:

  1. No time has been planned for automation or it is not sufficient. Look up the issue SCHEDULE SLIP for help in this case.
  2. Training for automation has not been planned for. See the issue LIMITED EXPERIENCE for good tips.
  3. Nobody has been assigned to do automation, it gets done “on the side” when one has time to spare. The issue INADEQUATE TEAM should give you useful suggestions on how to solve this problem.

If you feel the choices offered do not fully represent the issue, you can initially select one to see the next suggested steps, and if it doesn’t fit, you can go back a step. For this example, selecting the issue SCHEDULE SLIP suggests the following.

SCHEDULE SLIP (Management Issue) Examples:

  1. Test automation is done only in people’s spare time.
  2. Team members are working on concurrent tasks which take priority over automation tasks.
  3. Schedules for what automation can be done were too optimistic.
  4. Necessary software or hardware is not available on time or has to be shared with other projects.
  5. The planned schedule was not realistic.

Alternatively selecting number 1, ‘Lack of support: from management, testers, developers, etc.’ will bring up another question. Second question (for “Lack of support”)

What kind of support are you missing?

If you are lacking support in a specific way, one of the following may give you ideas:

  1. Managers say that they support you, but back off when you need their support. Probably managers don’t see the value of test automation and thus give it a lower priority than for instance going to market sooner.
  2. Testers don’t help the automation team.
  3. Developers don’t help the automation team.
  4. Specialists don’t help the automation team with special automation problems (Databases, networks etc.).
  5. Nobody helps new automators.
  6. Management expected a “silver bullet” or magic: managers think that after they bought an expensive tool, they don’t need to invest in anything else. See the issue UNREALISTIC EXPECTATIONS.

If you are having general problems with lack of support in many areas, the issue INADEQUATE SUPPORT may help. If having difficulty choosing, always select the answer that is most pressing now. This may be option number 2, ‘Testers don’t help the automation team’ which will bring up a Third question (for “Testers don’t help the automation team”)

Please select what you think is the main reason for the lack of support from testers:

  1. Testers think that the Software Under Test (SUT) is so complex that it’s impossible to automate, so why try.
  2. Testers don’t have time because they have an enormous load of manual tests to execute.
  3. Testers think that the SUT is still too unstable to automate and so don’t want to waste their time. Take a look at the issue TOO EARLY AUTOMATION.
  4. Testers don’t understand that automation can also ease their work with manual tests. The issue INADEQUATE COMMUNICATION will show you what patterns can help you in this case.
  5. Testers have been burned before with test automation and don’t want to repeat the experience. Look up issue UNMOTIVATED TEAM for help here.
  6. Testers do see the value of automation, but don’t want to have anything to do with it. Your issue is probably NON-TECHNICAL TESTERS.
  7. Supporting automation is not in the test plan and so testers won’t do it. Check the issue AD-HOC AUTOMATION for suggestions.

Should you think option 6 is the most suitable, simply click on NON-TECHNICAL TESTERS.

NON-TECHNICAL TESTERS (Process Issue) Examples

  1. Testers are interested in testing and not all testers want to learn the scripting languages of different automation tools. On the other hand, automators aren’t necessarily well acquainted with the application, so there are often communication problems.
  2. Testers can prepare test cases from the requirements and can therefore start even before the application has been developed. Automators must usually wait for at least a rudimentary GUI or API.

Resolving Patterns

Most recommended:

  • DOMAIN-DRIVEN TESTING: Apply this pattern to get rid of this issue for sure. It helps you find the best architecture when the testers cannot also be automators.
  • OBJECT MAP: This pattern is useful even if you don’t implement DOMAIN-DRIVEN TESTING because it forces the development of more readable scripts.

Other useful patterns:

  • KEYWORD-DRIVEN TESTING: This pattern is widely used already, so it will be not only easy to apply for your testers, but you will also find it easier to find automators able to implement it.
  • SHARE INFORMATION: If you have issues like Example 1. this is the pattern for you!
  • TEST AUTOMATION FRAMEWORK: If you plan to implement DOMAIN-DRIVEN TESTING you will need this pattern too. Even if you don’t, this pattern can make it easier for testers to use and help implement the automation.

What is the difference between the most recommended and the other useful patterns?

You should always look first at the most recommended patterns, but if for some reason you cannot apply them, then you should at least apply one or more of the useful ones. Here the most recommended pattern is DOMAIN-DRIVEN TESTING.

DOMAIN-DRIVEN TESTING (Design Pattern)

Description – Testers develop a simple domain-specific language to write their automated test cases with. Practically this means that actions particular to the domain are described by appropriate commands, each with a number of required parameters. As an example, let’s imagine that we want to insert a new customer into our system. The domain-command will look something like this:

New_Customer (FirstName, LastName, HouseNo, Street, ZipCode, City, State)

Now testers only have to call New_Customer and provide the relevant data for a customer to be inserted. Once the language has been specified, testers can start writing test cases even before the System under Test (SUT) has actually been implemented.

Implementation – To implement a domain-specific language, scripts or libraries must be written for all the desired domain-commands. This is usually done with a TEST AUTOMATION FRAMEWORK that supports ABSTRACTION LEVELS.

There are both advantages and disadvantages to this solution. The greatest advantage is that testers who are not very adept with the tools can write and maintain automated test cases. The downside is that you need developers or test automation engineers to implement the commands so that testers are completely dependent on their “good will”.  Another negative point is that the domain libraries may be implemented in the script language of the tool, so that to change the tool may mean to have to start again from scratch (TOOL DEPENDENCY). This can be mitigated to some extent using ABSTRACTION LEVELS.

KEYWORD-DRIVEN TESTING is a good choice for implementing a domain-specific language: Keyword = Domain-Command. Potential problems – It does take time and effort to develop a good domain-driven automated testing infrastructure.

The above suggests, the pattern you need is a TEST AUTOMATION FRAMEWORK!

TEST AUTOMATION FRAMEWORK (Design Pattern)

Description – Using or building a test automation framework helps solve a number of technical problems in test automation. A framework is an implementation of at least part of a testware architecture.

Implementation – Test automation frameworks are included in many of the newer vendor tools. If your tools don’t provide a support framework, you may have to implement one yourself. Actually, it is often better to design your own TESTWARE ARCHITECTURE, rather than adopt the tool’s way of organising things – this will tie you to that particular tool, and you may want your automated tests to be run one day using a different tool or on a different device or platform. If you design your own framework, you can keep the tool-specific things to a minimum, so when (not if) you need to change tools, or when the tool itself changes, you minimise the amount of work you need to do to get your tests up and running again.

The whole team, developers, testers, and automators, should come up with the requirements for the test automation framework, and choose by consensus. If you are comparing two frameworks (or tools) use SIDE-BY-SIDE to find the best fit for your situation.

A test automation framework should offer at least some of the following features:

  • Support ABSTRACTION LEVELS.
  • Support use of DEFAULT DATA.
  • Support writing tests.
  • Compile usage information.
  • Manage running the tests, including when tests don’t complete normally.
  • Report test results.

You will have to have MANAGEMENT SUPPORT to get the resources you will need, especially developer time if you have to implement the framework in-house. This pattern tells you what a framework should do, but you may have plenty of ideas yourself. If you’d like suggestions about how to do it, you can delve into the detail of the other patterns, where you will find more advice, particularly things like ABSTRACTION LEVELS and TESTWARE ARCHITECTURE.

However, just having a good technical framework isn’t always going to work, especially if the people seem to have no time for progressing the automation! Therefore this pattern also suggests getting MANAGEMENT SUPPORT.

MANAGEMENT SUPPORT (Management Pattern)

Description – Many issues can only be solved with good management support. When you are starting (or re-starting) test automation, you need to show managers that the investment in automation (not just in the tools) has a good potential to give real and lasting benefits to the organisation.

Implementation – Some suggestions when starting (or re-starting) test automation:

  • Make sure to SET CLEAR GOALS. Either review existing goals for automation or meet with managers to ensure that their expectations are realistic and adequately resourced and funded.
  • Build a convincing TEST AUTOMATION BUSINESS CASE. Test automation can be quite expensive and requires, especially at the beginning, a lot of effort.
  • A good way to convince management is to DO A PILOT. In this way they can actually “touch” the advantages of test automation and it will be much easier to win them over.
  • Another advantage is that it is much easier to SELL THE BENEFITS of a limited pilot than of a full test automation project. After your pilot has been successful, you will have a much better starting position to obtain support for what you actually intend to implement.

If you don’t know what their goals are for automation, you can try to find out about that but challenging a customer’s automation goals probably isn’t the best way to get the help you need! Essentially, that is going in and telling them they’ve done it all wrong which isn’t the best way to build new relationships. Therefore, building a business case isn’t relevant here, but DO A PILOT would be a good choice.

DO A PILOT (Management Pattern)

Context – This pattern is useful when you start an automation project from scratch, but it can also be very useful when trying to find the reasons your automation effort is not as successful as you expected.

Description – You start a pilot project to explore how to best automate tests on your application. The advantage of such a pilot is that it is time boxed and limited in scope, so that you can concentrate in finding out what the problems are and how to solve them. In a pilot project nobody will expect that you automate a lot of tests, but that you find out what are the best tools for your application, the best design strategy and so on.

You can also deal with problems that occur and will affect everyone doing automation and solve them in a standard way before rolling out automation practices more widely. You will gain confidence in your approach to automation. Alternatively, you may discover that something doesn’t work as well as you thought, so you find a better way – this is good to do as early as possible! Tom Gilb says: “If you are going to have a disaster, have it on a small scale”!

Implementation – Here some suggestions and additional patterns to help:

  • First of all, SET CLEAR GOALS: with the pilot project you should achieve one or more of the following goals:
    • Prove that automation works on your application.
    • Chose a test automation architecture.
    • Select one or more tools.
    • Define a set of standards.
    • Show that test automation delivers a good return on investment.
    • Show what test automation can deliver and what it cannot deliver.
    • Get experience with the application and the tools.
  • Try out different tools in order to select the RIGHT TOOLS that fit best for your SUT, but if possible PREFER FAMILIAR SOLUTIONS because you will be able to benefit from available know-how from the very beginning.
  • Do not be afraid to MIX APPROACHES.
  • AUTOMATION ROLES: see that you get the people with the necessary skills right from the beginning.
  • TAKE SMALL STEPS, for instance start by automating a STEEL THREAD: in this way you can get a good feeling about what kind of problems you will be facing, for instance check if you have TESTABLE SOFTWARE.
  • Take time for debriefing when you are through and don’t forget to LEARN FROM MISTAKES.
  • In order to get fast feedback, adopt SHORT ITERATIONS.

What kind of areas are explored in a pilot? This is the ideal opportunity to try out different ways of doing things, to determine what works best for you. These three areas are very important:

  • Building new automated tests. Try different ways to build tests, using different scripting techniques (DATA-DRIVEN TESTING or KEYWORD-DRIVEN TESTING). Experiment with different ways of organising the tests, i.e. different types of TESTWARE ARCHITECTURE. Find out how to most efficiently interface from your structure and architecture to the tool you are using. Take 10 or 20 stable tests and automate them in different ways, keeping track of the effort needed.
  • Maintenance of automated tests. When the application changes, the automated tests will be affected. How easy will it be to cope with those changes? If your automation is not well structured, with a good TESTWARE ARCHITECTURE, then even minor changes in the application can result in a disproportionate amount of maintenance to the automated tests – this is what often “kills” an automation effort! It is important in the pilot to experiment with different ways to build the tests in order to minimise later maintenance. Putting into practice GOOD PROGRAMMING PRACTICES and a GOOD DEVELOPMENT PROCESS are key to success. In the pilot, use different versions of the application – build the tests for one version, and then run them on a different version, and measure how much effort it takes to update the tests. Plan your automation to cope the best with application changes that are most likely to occur.
  • Failure analysis. When tests fail, they need to be analysed, and this requires human effort. In the pilot, experiment with how the failure information will be made available for the people who need to figure out what happened. What you want to have are EASY TO DEBUG FAILURES. A very important area to address here is how the automation will cope with common problems that may affect many tests. This would be a good time to put in place standard error-handling that every test can call on.

Potential problems – Trying to do too much: Don’t bite off more than you can chew – if you have too many goals you will have problems achieving them all. Worthless experiment: Do the pilot on something that is worth automating, but not on the critical path. Under-resourcing the pilot: Make sure that the people involved in the pilot are available when needed – managers need to understand that this is “real work”!

If you’d like to learn more about how to use the Test Automation Patterns Wiki, check it out directly or look up A Journey Through Test Automation Patterns book by Seretta Gamba & Dot Graham on Amazon. The book follows the story of Liz, a test automator, who thought she had just got the best job in the world. Through different experiences of her team, they learn to solve their problems (most of them) through using the test automation patterns. As you get to know the people on the team, you see how the patterns have helped to improve their automation, using very realistic examples.

Written by Suzanne Meade · Categorized: test automation

Apr 27 2023

The 2023 Lineup is LIVE

It’s here! The AutomationSTAR 2023 programme is LIVE. Are you looking for a blast of test automation inspiration?

We’ve got 2 whole days of headline keynotes, training tutorials, technical track talks, and plenty of opportunity to meet new people in the awesome test automation community.

The 2023 Programme Team, Chris Loder (Canada), John Kent (UK), and Sophie Küster (Germany) have put together a phenomenal line-up.  Get in on the action at Europe’s fastest growing test automation conference.

Check out the talks and secure your place today!

See Programme

3 Keynote Talks

Our 4 keynote speakers are delivering an inspiring dose of knowledge and insights.  Marie Cruz shares her experiences on getting started in tech, and what it was like for her as a member of a minority group in the industry. Ryan Volker and Bastian Schurmann will give you new ideas to continue advancing your test automation framework, and overcome unanticipated headwind; while Janna delves into the vital role of test automation in CI/CD pipelines, and show you techniques for balancing fast feedback. Notebooks at the ready!

4 Half Day Tutorials

2 days of intensive training from leading experts: learn how to use Playwright to automate end-to-end tests, see the importance of using test automation patterns, get the keys to a successful test automation framework, learn how investing in your automation can pay off massively, and lots more.

17 Track Talks & 2 Workshops

The AutomationSTAR track talks and workshops are filled with actionable learnings to advance your test automation projects. What does the Apollo 13 spacecraft disaster have in common with various automation problems in DevOps, CI/CD, and agile? What are the shortfalls faced by freeform UI apps when comes to Visual Testing? What’s the best open source test frameworks for automating web UI tests? What if we applied the Shift-Left approach to performance tests? Get answers to these questions and LOTS more.

We hope you like the programme, and see you in Berlin!

See Programme

A huge welcome to our first AutomationSTAR 2023 sponsors – join them in Berlin, Nov 20-21.

Our AutomationSTAR EXPO packages are bursting with opportunities for you to speak directly to the test automation community, and help solve their problems with your tools and services. Download our EXPO brochure now, or chat to Clare from our EXPO team.

Written by Suzanne Meade · Categorized: AutomationSTAR, test automation

Oct 03 2022

Email deliverability testing with SpamAssassin

Learn about SpamAssassin’s email filtering system to keep inboxes clear of unwanted emails & assign scores to emails. Thanks to Mailosaur for providing us with this blog post.

SpamAssassin (SA) is a well-established email filtering system designed to live up to its name. It uses comprehensive spam-fighting methods to keep inboxes clear of unwanted email. The filter assigns scores to emails to separate the genuine from the unwanted.

In this article, we’ll look at how a SpamAssassin score is calculated and what you can do it improve it. Inspecting and improving your SpamAssassin results will help you write better emails that your recipients will be happy to receive.

What is SpamAssassin?

SpamAssassin (officially, ‘Apache SpamAssassin’) is an open-source project developed and operated by the Apache Software Foundation. It was initially released in 2001 with the aim of providing a robust and customisable filter for detecting ‘email spam’, the Monty Python-inspired term for the practice of sending out unsolicited emails en masse.

The filter employs a range of different tests. These include scanning an email’s body and header, and checking a sender’s IP against several different block and allow lists. Users can add and adapt rules, or simply resort to SpamAssassin’s spam classifier and train it with their own data. Many email providers rely on SpamAssassin scores to classify incoming email as spam or the opposite, ‘ham’.

The SpamAssassin score

Each SpamAssassin rule is associated with a value that can be either negative or positive. The SpamAssassin filter runs its tests on each incoming email and adds up the values for the rules that are triggered. It then returns an aggregated SpamAssassin score. Perhaps counterintuitively, a higher score signifies a higher probability that an email is spam. Therefore, when passing your transactional emails through the SpamAssassin filter, you should aim for a lower rather than a higher score.

If an email passes a certain threshold, it’s regarded as spam. By default that threshold is set to 5.0 in the SpamAssassin configuration, though it can be adjusted by the user. It’s common to tweak the threshold to reach a good balance between low numbers of false positives (genuine email wrongly classified as spam) and false negatives (spam email that tricks the filter into thinking it’s genuine). That’s why your email might make it into the inbox of one recipient, but land in the spam folder of another—they might be using an email service with more restrictive anti-spam settings. Therefore, when you test your emails against the SpamAssassin filter, simply being under the 5.0 threshold may not be enough. Rather, you should aim for the lowest possible score.

Rule-based spam filters

So what does a SpamAssassin rule look like? Let’s look at an example that uses the KAM rules, a widely used custom rule set:

You can probably figure out the type of scam targeted here, but let’s go through it line-by-line. The first line is simply a comment for the developer that isn’t very descriptive. The next two lines are more interesting: tagged as ‘body’, they consist of two Perl regular expressions (‘regexes’). Regexes are powerful expressions for pattern-matching in text and are frequently part of the SpamAssassin filter.

Together, the regexes describe a pattern often used in spammy email in which a sender describes a job offer with unrealistically great conditions. Interestingly, the rule only gets triggered if the text includes the claim that the recipient can make ‘twice as much’ (not more, not less) money than with their current employer. This is ensured by a ‘meta’ rule (a complex combination of expressions) which requires that the sum of two individual patterns is at least two.

When this meta rule is triggered, a spam score increases by 4.3 points. Remember when we said that a score of 5.0 often suffices for a spam classification? So if your email uses the above pattern, it gets dangerously close to the spam folder.

While the McGrail Foundation which developed the KAM rules is closely associated with Apache’s SpamAssassin, the ruleset is not strictly speaking part of SA’s core rule collection. Have a look at all of SpamAssassin’s default rules here. To add your own rules, you’ll need to incorporate them into your local configuration and set the parameter ‘allow_local_rules’ to 1.

Probabilistic spam filters

Hard-coded rules are fine for catching spam, but an even smarter solution is to combine these rules with a score from a probabilistic spam classifier. Enter SpamAssassin’s Bayesian classifier, which is a machine learning model often employed in spam filtering. As is common with machine-learning techniques, the more emails the classifier sees, the better it gets at categorizing them.

SpamAssassin users thus have a way of fine-tuning the filter to their requirements, by feeding their own emails designated ‘spam’ or ‘ham’ to the classifier. By enabling the ‘bayes_auto_learn’ parameter one may even use the system without any previously labeled data. In such cases, incoming emails which the filter tagged as either spam or ham with a very high probability are fed back to the classifier as training data.

How to check your SpamAssassin score

Let’s finally look at SpamAssassin in action! After setting up a new server on our Mailosaur account, we’ll send it this deliberately suspicious email:

It does not look particularly trustworthy. We can already tell that we’ll get a pretty high SpamAssassin score for this email, but can you guess which parts will most trigger the filter?

Let’s open our Mailosaur dashboard to investigate:

While there are a few slightly positive elements in our email (notably the authentication signatures in the header), we made just about every mistake possible in the email’s body. The two most negative items, however, were in addressing our recipient a ‘dear friend’ and mentioning a ‘100% guarantee’.

Interestingly, despite its mediocre SpamAssassin score, Gmail accepted the above email into an inbox folder. That’s likely due to the fact that this email service prioritises spam-catching rules differently from SpamAssassin’s default settings.

How to improve your SpamAssassin score

Improving our SpamAssassin score is not too difficult. Let’s simply avoid the spammy phrases that trigger the addition of high values to our score. So as a little challenge, let’s try to do the best we can and achieve a negative SpamAssassin score:

Not only does our email sound much less frantic than the previous one (lowercasing may have helped), but it’s also personalised and friendly, managing to get our message across just as well. What does our SpamAssassin score say now?

What a lovely sight! SpamAssassin likes our email just as much as we do—and we’ve achieved our goal of generating a negative SpamAssassin score.

Content is not everything

While examples that look at the body of an email are fun and intuitively understandable, a lot of SpamAssassin checks are performed behind the scenes, and many of these require more than a nicely written email to be fooled. For instance, if your domain lands on a block list, there’s not much you can do other than use a different domain or a different email provider.

Likewise, message authentication protocols such as DKIM and SPF have been designed specifically to address the problem of email spoofing, where the sender pretends to be somebody else. While you can still manipulate the content of your email headers, you cannot simply fake a DKIM protocol. What makes SpamAssassin so robust as a spam filter is that it combines all those metrics (and more) to form a score.
So what can you do besides writing polished emails and using well-formed HTML? Here are a few tips:

• Use the email standards we mentioned: DKIM and SPF. They prove that you wrote the email and that you’re authorised to send emails from your domain.

• Avoid sending email in bulk from a brand new ip address: that’s considered suspicious behaviour. Instead, if sending emails to multiple recipients, make sure to distribute them over several days, slowly increasing the daily load.

• Work on your sender reputation. Make sure to send personalised emails that your recipients actually want to read. The more your emails get opened and read, and the less they’re marked as spam, the better your reputation will be.

Use a dedicated email testing solution

Checking your SpamAssassin score is one tiny piece of the much larger puzzle that is email deliverability and testing. If you’re serious about communicating with your customers, consider using a dedicated email testing platform like Mailosaur, which allows you to do everything above, and much more, as part of your automated testing suite.

Mailosaur is an exhibitor at AutomationSTAR 2022. Join the software testing community in Munich, 17-18 October. Get your ticket now.

This blog is translated from English to German by Google.

Written by Fiona Nic Dhonnacha · Categorized: test automation

Jul 22 2022

Why attend AutomationSTAR

Why is AutomationSTAR an unmissable experience? This is where the leaders in software testing are coming together.

Accelerate your testing projects, make connections, & join a global community. This is a space to deep dive into all things automation in testing, and share different perspectives, discuss the problems and issues you face, and find solutions together. Check out everything that’s on offer, and join us in Munich, 17-18 Oct.

Valuable learnings

Whether you are new to coding, building a test automation project from scratch, or interested in codeless tools, AutomationSTAR is where you’ll find the answers. Our superstar speakers are bringing you answers, to help you accelerate your delivery, get future ready, and achieve your goals. Talks will cover different levels of automation in testing, including methodologies such as DevOps, new technologies like AI, and lots more

Meaningful connections

It’s a diverse community bursting with creativity – with plenty of chances to jump into different conversations and make meaningful connections. With coffee breaks, networking lunches, and lots of bonus sessions, AutomationSTAR is where creativity flourishes. You’ll chat with software testers from every industry, and have lots of lightbulb moments as you spark new ideas.

Expert speakers

25 speakers will bring you in-depth knowledge on everything automation – get razor sharp insights directly from testing leaders. Learn about the role of automation engineers, how to achieve reliable test automation, and answer questions on why we test and why we make the choices we do. Kickstart your cypress test automation, create an automation framework, learn the importance of soft skills, and LOTS more.

Solve testing challenges

We know that you’re looking for tangible results – that’s why our testing experts will be answering your tough questions, whether that’s during an inspiring track talk session, or an in-depth tutorial. The AutomationSTAR speakers will share their own experiences and learnings, to get you where you want to be in your test automation projects. You’ll also be surrounded by like-minded peers who are having the same issues you are, and finding solutions together.

Teamwork

We are all about the community, and we love welcoming teams to the conference. QA team training is crucial for implementing successful automation projects. Create a whole team approach at AutomationSTAR, ensuring everyone is well versed in their roles, and possess the technical skills they need to support the tests, scale productivity – and deliver an exceptional user experience. Bonus: groups save on tickets, so get the office together and join us in Munich.

Our 2-for-1 offer is currently running until July 31st – this is your chance to save 50% on tickets! Book now.

get tickets

This blog is translated from English to German by Google.

Written by Fiona Nic Dhonnacha · Categorized: test automation · Tagged: 2022

Jul 14 2022

Playwright vs Selenium: what are the main differences and which is better?

Wondering how to choose between Playwright vs Selenium for your test automation? Read on to see a comparison. Thanks to Danny Shain at Applitools for providing this blog.

When it comes to web test automation, Selenium has been the dominant industry tool for several years. However, there are many other automated testing tools on the market. Playwright is a newer tool that has been gaining popularity. How do their features compare, and which one should you choose?

What is Selenium?

Selenium is a long-running open source tool for browser automation. It was originally conceived in 2004 by Jason Huggins, and has been actively developed ever since. Selenium is a widely-used tool with a huge community of users, and the Selenium WebDriver interface even became an official W3C Recommendation in 2018.

The framework is capable of automating and controlling web browsers and interacting with UI elements, and it’s the most popular framework in the industry today. There are several tools in the Selenium suite, including:

• Selenium WebDriver: WebDriver provides a flexible collection of open source APIs that can be used to easily test web applications.

• Selenium IDE: This record-and-playback tool enables rapid test development for both engineers and non-technical users.

• Selenium Grid: The Grid lets you distribute and run tests in parallel on multiple machines.

The impact of Selenium goes even beyond the core framework, as a number of other popular tools, such as Appium and WebDriverIO, have been built directly on top of Selenium’s API.

Selenium is under active development and recently unveiled a major version update to Selenium 4. It supports just about all major browsers and popular programming languages. Thanks to a wide footprint of use and extensive community support, the Selenium open source project continues to be a formidable presence in the browser automation space.

What is Playwright?

Playwright is a relatively new open source tool for browser automation, with its first version released by Microsoft in 2020. It was built by the team behind Puppeteer, which is a headless testing framework for Chrome/Chromium. Playwright goes beyond Puppeteer and provides support for multiple browsers, among other changes.

Playwright is designed for end-to-end automated testing of web apps. It’s cross-platform, cross-browser and cross-language, and includes helpful features like auto-waiting. It is specifically engineered for the modern web and generally runs very quickly, even for complex testing projects.

While far newer than Selenium, Playwright is picking up steam quickly and has a growing following. Due in part to its young age, it supports fewer browsers/languages than Selenium, but by the same token it also includes newer features and capabilities that are more aligned with the modern web. It is actively developed by Microsoft.

Selenium vs Playwright

Selenium and Playwright are both capable web automation tools, and each has its own strengths and weaknesses. Depending on your needs, either one could serve you best. Do you need a wider array of browser/language support? How much does a long track record of support and active development matter to you? Is test execution speed paramount?

Each tool is open source, cross-language and developer friendly. Both support CI/CD (via Jenkins, Azure Pipelines, etc.), and advanced features like screenshot testing and automated visual testing. However, there are some key architectural and historical differences between the two that explain some of their biggest differences.

Selenium Architecture and History

• Architecture: Selenium uses the WebDriver API to interact between web browsers and browser drivers. It operates by translating test cases into JSON and sending them to the browsers, which then execute the commands and send an HTTP response back.

• History: Selenium has been in continuous operation and development for 18+ years. As a longstanding open source project, it offers broad support for browsers/languages, a wide range of community resources and an ecosystem of support.

Playwright Architecture and History

• Architecture: Playwright uses a WebSocket connection rather than the WebDriver API and HTTP. This stays open for the duration of the test, so everything is sent on one connection. This is one reason why Playwright’s execution speeds tend to be faster.

• History: Playwright is fairly new to the automation scene. It is faster than Selenium and has capabilities that Selenium lacks, but it does not yet have as broad a range of support for browsers/languages or community support. It is open source and backed by Microsoft.

Comparing Playwright vs Selenium Features

It’s important to consider your own needs and pain points when choosing your next test automation framework. The table below will help you compare Playwright vs Selenium.

CriteriaPlaywrightSelenium
Browser SupportChromium, Firefox, and WebKit (note: Playwright tests browser projects, not stock browsers)Chrome, Safari, Firefox, Opera, Edge, and IE
Language SupportJava, Python, .NET C#, TypeScript and JavaScript.Java, Python, C#, Ruby, Perl, PHP, and JavaScript
Test Runner Frameworks SupportJest/Jasmine, AVA, Mocha, and VitestJest/Jasmine, Mocha, WebDriver IO, Protractor, TestNG, JUnit, and NUnit
Operating System SupportWindows, Mac OS and LinuxWindows, Mac OS, Linux and Solaris
ArchitectureHeadless browser with event-driven architecture4-layer architecture (Selenium Client Library, JSON Wire Protocol, Browser Drivers and Browsers)
Integration with CIYesYes
PrerequisitesNodeJSSelenium Bindings (for your language), Browser Drivers and Selenium Standalone Server
Real Device SupportNative mobile emulation (and experimental real Android support)Real device clouds and remote servers
Community SupportSmaller but growing set of community resourcesLarge, established collection of documentation and support options
Open SourceFree and open source, backed by MicrosoftFree and open source, backed by large community


Should You Use Selenium or Playwright for Test Automation?

Is Selenium better than Playwright? Or is Playwright better than Selenium? Selenium and Playwright both have a number of things going for them – there’s no easy answer here. When choosing between Selenium vs Playwright, it’s important to understand your own requirements and research your options before deciding on a winner.

Selenium vs Playwright: Let the Code Speak

A helpful way to go beyond lists of features and try to get a feel for the practical advantages of each tool is to go straight to the code and compare real-world examples side by side. At Applitools, our goal is to make test automation easier for you – so that’s what we did!

In the video below, you can see a head to head comparison of Playwright vs Selenium. Angie Jones and Andrew Knight take you through ten rounds of a straight-to-the-code battle, with the live audience deciding the winning framework for each round. Check it out for a unique look at the differences between Playwright and Selenium.

Selenium Vs Applitools webinar

If you like these code battles and want more, we’ve also pitted Playwright vs Cypress and Selenium vs Cypress – check out all our versus battles here.

In fact, our original Playwright vs Cypress battle (recap here) was so popular that we’ve even scheduled our first rematch. Who will win this time? Register for the Playwright vs Cypress Rematch now to join in and vote for the winner yourself!

Applitools is an exhibitor at AutomationSTAR 2022. Join the software testing community in Munich, 17-18 October. Get your ticket now.

Author

Danny Shain Content Marketing Manager Applitools

Danny Shain – Content Marketing Manager at Applitools

Writer at heart passionate about meaningful effective communication. Extensive experience working with a range of material, from fiction to academic to promotional. Technology lover and e-commerce veteran.

This blog has been translated from German to English by Google.

Written by Lauren Payne · Categorized: Selenium, test automation · Tagged: 2022

  • Go to page 1
  • Go to page 2
  • Go to Next Page »

Copyright © 2023 · Impressum · Privacy · T&C