test automation

Learn about SpamAssassin’s email filtering system to keep inboxes clear of unwanted emails & assign scores to emails. Thanks to Mailosaur for providing us with this blog post.

SpamAssassin (SA) is a well-established email filtering system designed to live up to its name. It uses comprehensive spam-fighting methods to keep inboxes clear of unwanted email. The filter assigns scores to emails to separate the genuine from the unwanted.

In this article, we’ll look at how a SpamAssassin score is calculated and what you can do it improve it. Inspecting and improving your SpamAssassin results will help you write better emails that your recipients will be happy to receive.

What is SpamAssassin?

SpamAssassin (officially, ‘Apache SpamAssassin’) is an open-source project developed and operated by the Apache Software Foundation. It was initially released in 2001 with the aim of providing a robust and customisable filter for detecting ‘email spam’, the Monty Python-inspired term for the practice of sending out unsolicited emails en masse.

The filter employs a range of different tests. These include scanning an email’s body and header, and checking a sender’s IP against several different block and allow lists. Users can add and adapt rules, or simply resort to SpamAssassin’s spam classifier and train it with their own data. Many email providers rely on SpamAssassin scores to classify incoming email as spam or the opposite, ‘ham’.

The SpamAssassin score

Each SpamAssassin rule is associated with a value that can be either negative or positive. The SpamAssassin filter runs its tests on each incoming email and adds up the values for the rules that are triggered. It then returns an aggregated SpamAssassin score. Perhaps counterintuitively, a higher score signifies a higher probability that an email is spam. Therefore, when passing your transactional emails through the SpamAssassin filter, you should aim for a lower rather than a higher score.

If an email passes a certain threshold, it’s regarded as spam. By default that threshold is set to 5.0 in the SpamAssassin configuration, though it can be adjusted by the user. It’s common to tweak the threshold to reach a good balance between low numbers of false positives (genuine email wrongly classified as spam) and false negatives (spam email that tricks the filter into thinking it’s genuine). That’s why your email might make it into the inbox of one recipient, but land in the spam folder of another—they might be using an email service with more restrictive anti-spam settings. Therefore, when you test your emails against the SpamAssassin filter, simply being under the 5.0 threshold may not be enough. Rather, you should aim for the lowest possible score.

Rule-based spam filters

So what does a SpamAssassin rule look like? Let’s look at an example that uses the KAM rules, a widely used custom rule set:

You can probably figure out the type of scam targeted here, but let’s go through it line-by-line. The first line is simply a comment for the developer that isn’t very descriptive. The next two lines are more interesting: tagged as ‘body’, they consist of two Perl regular expressions (‘regexes’). Regexes are powerful expressions for pattern-matching in text and are frequently part of the SpamAssassin filter.

Together, the regexes describe a pattern often used in spammy email in which a sender describes a job offer with unrealistically great conditions. Interestingly, the rule only gets triggered if the text includes the claim that the recipient can make ‘twice as much’ (not more, not less) money than with their current employer. This is ensured by a ‘meta’ rule (a complex combination of expressions) which requires that the sum of two individual patterns is at least two.

When this meta rule is triggered, a spam score increases by 4.3 points. Remember when we said that a score of 5.0 often suffices for a spam classification? So if your email uses the above pattern, it gets dangerously close to the spam folder.

While the McGrail Foundation which developed the KAM rules is closely associated with Apache’s SpamAssassin, the ruleset is not strictly speaking part of SA’s core rule collection. Have a look at all of SpamAssassin’s default rules here. To add your own rules, you’ll need to incorporate them into your local configuration and set the parameter ‘allow_local_rules’ to 1.

Probabilistic spam filters

Hard-coded rules are fine for catching spam, but an even smarter solution is to combine these rules with a score from a probabilistic spam classifier. Enter SpamAssassin’s Bayesian classifier, which is a machine learning model often employed in spam filtering. As is common with machine-learning techniques, the more emails the classifier sees, the better it gets at categorizing them.

SpamAssassin users thus have a way of fine-tuning the filter to their requirements, by feeding their own emails designated ‘spam’ or ‘ham’ to the classifier. By enabling the ‘bayes_auto_learn’ parameter one may even use the system without any previously labeled data. In such cases, incoming emails which the filter tagged as either spam or ham with a very high probability are fed back to the classifier as training data.

How to check your SpamAssassin score

Let’s finally look at SpamAssassin in action! After setting up a new server on our Mailosaur account, we’ll send it this deliberately suspicious email:

It does not look particularly trustworthy. We can already tell that we’ll get a pretty high SpamAssassin score for this email, but can you guess which parts will most trigger the filter?

Let’s open our Mailosaur dashboard to investigate:

While there are a few slightly positive elements in our email (notably the authentication signatures in the header), we made just about every mistake possible in the email’s body. The two most negative items, however, were in addressing our recipient a ‘dear friend’ and mentioning a ‘100% guarantee’.

Interestingly, despite its mediocre SpamAssassin score, Gmail accepted the above email into an inbox folder. That’s likely due to the fact that this email service prioritises spam-catching rules differently from SpamAssassin’s default settings.

How to improve your SpamAssassin score

Improving our SpamAssassin score is not too difficult. Let’s simply avoid the spammy phrases that trigger the addition of high values to our score. So as a little challenge, let’s try to do the best we can and achieve a negative SpamAssassin score:

Not only does our email sound much less frantic than the previous one (lowercasing may have helped), but it’s also personalised and friendly, managing to get our message across just as well. What does our SpamAssassin score say now?

What a lovely sight! SpamAssassin likes our email just as much as we do—and we’ve achieved our goal of generating a negative SpamAssassin score.

Content is not everything

While examples that look at the body of an email are fun and intuitively understandable, a lot of SpamAssassin checks are performed behind the scenes, and many of these require more than a nicely written email to be fooled. For instance, if your domain lands on a block list, there’s not much you can do other than use a different domain or a different email provider.

Likewise, message authentication protocols such as DKIM and SPF have been designed specifically to address the problem of email spoofing, where the sender pretends to be somebody else. While you can still manipulate the content of your email headers, you cannot simply fake a DKIM protocol. What makes SpamAssassin so robust as a spam filter is that it combines all those metrics (and more) to form a score.
So what can you do besides writing polished emails and using well-formed HTML? Here are a few tips:

• Use the email standards we mentioned: DKIM and SPF. They prove that you wrote the email and that you’re authorised to send emails from your domain.

• Avoid sending email in bulk from a brand new ip address: that’s considered suspicious behaviour. Instead, if sending emails to multiple recipients, make sure to distribute them over several days, slowly increasing the daily load.

• Work on your sender reputation. Make sure to send personalised emails that your recipients actually want to read. The more your emails get opened and read, and the less they’re marked as spam, the better your reputation will be.

Use a dedicated email testing solution

Checking your SpamAssassin score is one tiny piece of the much larger puzzle that is email deliverability and testing. If you’re serious about communicating with your customers, consider using a dedicated email testing platform like Mailosaur, which allows you to do everything above, and much more, as part of your automated testing suite.

Mailosaur is an exhibitor at AutomationSTAR 2022. Join the software testing community in Munich, 17-18 October. Get your ticket now.

This blog is translated from English to German by Google.

This article will dive deeper into how to handle popups with selenium webdriver, and how to automate them. This is a blog post provided by Mark Collin on EuroSTAR Huddle, our online community for free software testing resources.

Demystifying popups with Selenium

After you have been working with Selenium for a while you will eventually be given a task to automate something that has some sort of popup functionality. Popups can be a sticky subject in Selenium because the definition of what a popup is can be unclear and different drivers exhibit different default behaviour. In this article I will try and provide some clear definitions of different popups, show you how to handle Popup in Selenium Webdriver and show you how to automate them, and a few gotchas as well.

First of all we need to define what we mean by a popup; it’s a nebulous term that people use to describe multiple different types of functionality that is out in the wild. I like to put popups into three main categories:

OS level popups

The first popup listed is an OS level dialogue; these are the things you will see when the browser hands over control to the operating system to do a task for it (although browsers like chrome are blurring the lines here a little bit by making it look like the browser is doing it). You will commonly see these if you do something that the browser cannot do, at this point the browser will hand over to the OS to find another program to perform the required task. Some examples of these things are:

• Printing a page you are viewing in the browser.
• Sending a link as an email.
• Downloading a file.

The best option to deal with these OS level popups, is to never do anything that will trigger them in your test. Selenium tutorial is only designed to interact with websites rendered in
the browser, controlling the OS is totally out of scope. If you did want to try and interact with these OS level popups you would need to write some code that can control the
OS itself (e.g. something using the Java Robot classes). There are tools out there that will do this, but it’s a complex bit of work and you really don’t need to test OS level functionality to check that your website works.

HTML popups (Lightboxes)

A HTML popup can look like a JavaScript popup, or a OS level popup, but in reality it’s just more HTML layered over the top of existing HTML on the website. Due to the fact that these popups are HTML website designers can use CSS to make these things look like something they are not, which can cause confusion. You don’t need to do anything special to work with these; after all they are just HTML and CSS. You just interact with them in same way that you would interact with any other element on screen and then do what you want.

JavaScript popups

These are blocking popups; they will pause JavaScript execution until they have been interacted with. Historically, these were the default type of Selenium popup, although nowadays they are not quite as common because website designers tend to prefer using non-blocking HTML popups instead. These are things that Selenium can interact with, but you will need to know what actions are going to trigger these popups in advance.

We should now have a clear idea in our mind of the various types of popup. Next, let’s have a look at some code.

Working with alerts

First of all, we are going to look at how to automate things alerts. To start with, we will need some HTML to automate. Save the following html as ‘alert.html’:

private String alertFileLocation = “file://<PATH_TO>/alert.html”;

<!doctype html>
    <html lang="en">
    <head>
        <title>Alerts in Selenium</title>
        <link rel="stylesheet" href="http://code.jquery.com/ui/1.12.1/themes/base/jquery-ui.css">
        <link rel="stylesheet" href="http://jqueryui.com/resources/demos/style.css">
        <script src="https://code.jquery.com/jquery-1.12.4.js"></script>
        <script src="https://code.jquery.com/ui/1.12.1/jquery-ui.js"></script>
        <script>
            function triggerAlert() {
                $('#alert').dialog();
            }
        </script>
    </head>
    <body style="background-color: #6cb5ff">
    <div id="buttons">
        <button id="js_alert" onclick="alert('This is a JavaScript Alert');">Click me for a basic JavaScript alert</button>
        <button id="html_alert" onclick="triggerAlert()">Click me for a basic jQuery alert</button>
    </div>
    <div id="alert" title="Dialog" style="display: none">
        <p>This is an alert box triggered by JavaScript, but rendered in rendered in HTML</p>
    </div>
    </body>
    </html>

This is a simple page that has a couple of buttons. One will trigger a HTML alert, and one will trigger a JavaScript alert. Load it up in the browser, and try it out to see how it works.

We are now going to write some code to automate this page. First of all, we need to define the location where you saved the above HTML. Just replace in the below code with the path to the file you saved above.

private String alertFileLocation = “file://<PATH_TO>/alert.html”;

Now on to our first basic piece of automation:

@Test
        public void triggerBothPopups() throws Exception {
            driver.get(alertFileLocation);

            WebElement generateAlertOne = driver.findElement(By.id("js_alert"));
            WebElement generateAlertTwo = driver.findElement(By.id("html_alert"));

            generateAlertOne.click();
            generateAlertTwo.click();
        }

This code will load up the web page: click on the button to trigger the JavaScript alert, and then click on the button to trigger the HTML alert. If you now run this test it should fail, and you should see an error that looks like this:

org.openqa.selenium.UnhandledAlertException: unexpected alert open: {Alert text : This is a JavaScript Alert}

However you may not have seen an error…

This is because different drivers are implemented differently; if you are running Firefox v53.0.3 and GeckoDriver v0.16.1 it will have ignored the fact that there is a popup there and tried to carry on regardless. We will come back and look at this later on. For now, let’s make sure we are using chromedriver.

Now that we are using chromedriver, let’s try and fix our automation code. Selenium provides an API that enables you to interact with alerts, so let’s use it.

   @Test
        public void triggerBothPopups() throws Exception {
            driver.get(alertFileLocation);
            WebElement generateAlertOne = driver.findElement(By.id("js_alert"));
            WebElement generateAlertTwo = driver.findElement(By.id("html_alert"));

            generateAlertOne.click();
            driver.switchTo().alert().accept();
            generateAlertTwo.click();
            driver.switchTo().alert().accept();
        }

As you can see, we are now telling Selenium to switch to the alert once it has popped up. Once we have switched to it, we are telling Selenium to accept it. Let’s run the test again and see what happens now.

You should now see the following error:

org.openqa.selenium.NoAlertPresentException: no alert open

This is because our first alert is a JavaScript popup, but our second one is a HTML popup (or a Lightbox). The functionality to switch to an alert and accept it will only work with JavaScript popups, not HTML popups. To interact with the HTML popup we will need to look at the markup, and deal with it like we would any other HTML element.

Here is our final bit of test code.

 @Test
        public void triggerBothPopups() throws Exception {
            driver.get(alertFileLocation);
            WebElement generateAlertOne = driver.findElement(By.id("js_alert"));
            WebElement generateAlertTwo = driver.findElement(By.id("html_alert"));

            generateAlertOne.click();
            driver.switchTo().alert().accept();

            generateAlertTwo.click();
            WebElement closeHtmlAlertButton = driver.findElement(By.cssSelector(".ui-button"));
            closeHtmlAlertButton.click();
        }

We are now treating each type of alert correctly, and we have one happy piece of automation.

Why the test didn’t fail in Firefox

Continuing with our Selenium tutorial, let’s go back to the problem with Firefox that we mentioned when we wrote our initial test. Firefox was not throwing an exception when an unexpected alert was present, so we weren’t seeing an error. If something like this happens it can be very confusing. The main reason our test was not failing was because we were not performing any checks after
interacting with the elements. Let’s take the final test that we wrote and delete the code that switches to the alert, and see what happens when we run it in Firefox v53.0.3 and GeckoDriver v0.16.1.

@Test
            public void triggerBothPopups() throws Exception {
            driver.get(alertFileLocation);
            WebElement generateAlertOne = driver.findElement(By.id("js_alert"));
            WebElement generateAlertTwo = driver.findElement(By.id("html_alert"));

            generateAlertOne.click();

            generateAlertTwo.click();
            WebElement closeHtmlAlertButton = driver.findElement(By.cssSelector(".ui-button"));
            closeHtmlAlertButton.click();
        }

Now this time the test will fail, but you will see the following error:

org.openqa.selenium.NoSuchElementException: Unable to locate element: .ui-button

It has failed because we can’t find the button that is used to close the HTML popup; the implication here is that clicking on the second generate alert button did not work. Now we know that this is not the case, this test is failing because we have removed the code that switches to the alert and accepts it. Can we make this clearer by writing a screenshot of what the test is seeing to file?

 @Test
        public void triggerBothPopups() throws Exception {
            driver.get(alertFileLocation);
            WebElement generateAlertOne = driver.findElement(By.id("js_alert"));
            WebElement generateAlertTwo = driver.findElement(By.id("html_alert"));

            generateAlertOne.click();

            generateAlertTwo.click();

            File screenshot = new File("screenshot.png");
            FileOutputStream screenshotStream = new FileOutputStream(screenshot);
            screenshotStream.write(((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES));
            screenshotStream.close();
            System.out.println("Screenshot saved at: " + screenshot.getAbsolutePath());

            WebElement closeHtmlAlertButton = driver.findElement(By.cssSelector(".ui-button"));
            closeHtmlAlertButton.click();
        }

Let’s run the test again; this time a screenshot will be generated and the file location will be shown in the console log. This should show us that JavaScript alert that is causing all of the problems.

If you now open up the screenshot and look at it, it’s probably not going to show you what you expected. The screenshot shows the rendered website with no JavaScript or HTML alerts. What’s going on?

JavaScript popups are not part of the website that is being rendered, they are instead rendered by the browser. As a result they don’t show up in screenshots taken by selenium. To all intents and purposes, it looks like we have found a bug with the website implementation. It looks like clicking on the second button does not trigger an HTML alert to be shown. However, if you go back and test manually you will see that it works.

We know what the problem is, but imagine how confusing this would be to somebody who didn’t understand that we had a mixture of different types of popups. You would probably raise a bug stating that the second button didn’t work and pass it back to a dev who would not be able to reproduce the problem (because the only problem that exists is a problem with the test code).

Working with other types of JavaScript popup

So far we have looked at only one type of popup, the alert. There are two other types of JavaScript popups that you may come across, the prompt popup and the confirmation popup. Let’s create the last bit of HTML we are going to automate, save the following as ‘popup.html’:

    <!doctype html>
    <html lang="en">
    <head>
        <title>Prompts in Selenium</title>
    </head>
    <body style="background-color: #6cb5ff">
    <div id="buttons">
        <button id="js_confirm" onclick="confirm('This is a JavaScript confirmation box, you can select OK or Cancel.');">Click me for a basic JavaScript confirmation box</button>
        <button id="js_prompt" onclick="prompt('This is a JavaScript Prompt, it wants to get some information.');">Click me for a basic JavaScript prompt</button>
    </div>
    </body>
    </html>

As before, we need to define the location where you saved the above HTML. Just replace in the below code with the path to the file you saved above.

private String popupFileLocation = “file:///popup.html”;

First of all, we will look at how we can interact with a confirmation popup. Confirmation popups will ask you a question that you can either agree to, or cancel out of. They will always have two options, ‘OK’ or ‘Cancel’. Popups like this are usually used to block code execution until you agree to something.

Let us assume that we are going to agree with the confirmation popup first of all.

            @Test
            public void triggerJavaScriptConfirmation() throws Exception {
                driver.get(popupFileLocation);

                WebElement generateConfirmationPopup = driver.findElement(By.id("js_confirm"));
                generateConfirmationPopup.click();

                driver.switchTo().alert().accept();
            }

The code is very simple, in fact it’s exactly the same as the code we used to automate the alert popup. If we don’t want to agree with the confirmation popup the code is just as simple:

            @Test
            public void triggerJavaScriptConfirmation() throws Exception {
                driver.get(popupFileLocation);

                WebElement generateConfirmationPopup = driver.findElement(By.id("js_confirm"));
                generateConfirmationPopup.click();

                driver.switchTo().alert().dismiss();
            }

The last type of popup we are going to look at is the JavaScript prompt. These are blocking popups that are used to get some information from you. When you use these types of popup you are probably going to be asking the user a question that they need to respond to, so we will have a look at how we can check that the right question is being asked as well as entering the required information. Here is our code:

            @Test
            public void triggerJavaScriptPrompt() throws Exception {
                driver.get(popupFileLocation);

                WebElement generatePromptPopup = driver.findElement(By.id("js_prompt"));
                generatePromptPopup.click();

                Alert prompt = driver.switchTo().alert();

                assert(prompt.getText().equals("This is a JavaScript Prompt, it wants to get some information."));

                prompt.dismiss();
            }

This time, after triggering our JavaScript prompt, we have decided to define an Alert object. We are going to be interacting with the alert more than once so it makes sense to do so for code clarity and cleanliness. We have then used the getText() command to get the text that is being displayed to the user and made an assertion to check that it matches our expected text. We have then decided that we don’t want to provide any further information and dismissed the alert.

What about if we decided that we did want to send over some information? In that case we would have written the following code:

            @Test
            public void triggerJavaScriptPrompt() throws Exception {
                driver.get(popupFileLocation);

                WebElement generatePromptPopup = driver.findElement(By.id("js_prompt"));
                generatePromptPopup.click();

                Alert prompt = driver.switchTo().alert();

                assert(prompt.getText().equals("This is a JavaScript Prompt, it wants to get some information."));

                prompt.sendKeys("Some information");
                prompt.accept();
            }

As you can see this time we have sent some information over using the sendKeys() command and then accepted the prompt. We could of course have still decided to dismiss the prompt after entering some information, but that’s a scenario we haven’t written. You could always try that one yourself.

It is worth mentioning once more that different driver implementations can act in many different ways, and different driver implementations can support different parts of the WebDriver API. If you are having problems it’s always worth trying a different type of driver to see if your code can work.