2023

Oct 16 2023

Prompt-Driven Test Automation

Bridging the Gap Between QA and Automation with AI

In the modern software development landscape, test automation is often a topic of intense debate. Some view it strictly as a segment of Quality Assurance, while others, like myself, believe it intersects both the realms of QA and programming. The Venn diagram I previously shared visualizes this overlap.

Historically, there’s a clear distinction between the competencies required for QA work and those needed for programming:

Skills Required for QA Work:

Critical Thinking: The ability to design effective test cases and identify intricate flaws in complex systems
Attention to Details: The ability to ensure that minor issues are caught before they escalate into major defects.
Domain knowledge: A thorough understanding of technical requirements and business objectives to align QA work effectively.

Skills Required for Programming:

Logical Imagination: The capability to deconstruct complex test scenarios into segmented, methodical tasks ripe for efficient automation.
Coding: The proficiency to translate intuitive test steps into automated scripts that a machine can execute.
Debugging: The systematic approach to isolate issues in test scripts and rectify them to ensure the highest level of reliability.

We’re currently at an AI-driven crossroads, presenting two potential scenarios for the future of QA. One, where AI gradually assumes the roles traditionally filled by QA professionals, and another, where QAs harness the power of AI to elevate and redefine their positions.

This evolution not only concerns the realm of Quality Assurance but also hints at broader implications for the job market as a whole. Will AI technologies become the tools of a select few, centralizing the labor market? Or will they serve as instruments of empowerment, broadening the horizons of high-skill jobs by filling existing skill gaps?

I’m inclined toward the latter perspective. For QA teams to thrive in this evolving ecosystem, they must identify and utilize tools that bolster their strengths, especially in areas where developers have traditionally dominated.

So, what characterizes such a tool? At Loadmill, our exploration of this question has yielded some insights. To navigate this AI-augmented future, QAs require:

AI-Driven Test Creation: A mechanism that translates observed user scenarios into robust test cases.
AI-Assisted Test Maintenance: An automated system that continually refines tests, using AI to detect discrepancies and implement adjustments.
AI-Enabled Test Analysis: A process that deploys AI for sifting through vast amounts of test results, identifying patterns, and highlighting concerns.

When it comes to actualizing AI-driven test creation, there are two predominant methodologies. The code-centric method, exemplified by tools like GitHub Code Pilot, leans heavily on the existing codebase to derive tests. While this method excels in generating unit tests, its scope is inherently limited to the behavior dictated by the current code, making it somewhat narrow-sighted.

Contrarily, Loadmill champions the behavior-centric approach. An AI system that allows QA engineers to capture user interactions or describe them in plain English to create automated test scripts. The AI then undertakes the task of converting this human-friendly narrative into corresponding test code. This integration of AI doesn’t halt here – it extends its efficiencies to areas of test maintenance and result analysis, notably speeding up tasks that historically were time-intensive.

In sum, as the realms of QA and programming converge, opportunities for innovation and progress emerge. AI’s rapid advancements prompt crucial questions about the direction of QA and the broader job market. At Loadmill, we’re committed to ensuring that, in this changing landscape, QAs are not just participants but pioneers. I extend an invitation to all attendees of the upcoming conference: visit our booth in the expo hall. Let’s delve deeper into this conversation and explore how AI can be a game-changer for your QA processes.

For further insights and discussions, please engage with us at the Loadmill booth.

Author

Ido Cohen, founder and CEO of Loadmill.

Ido Cohen is the Co-founder and CEO of Loadmill. With over a decade of experience as both a hands-on developer and manager, he’s dedicated to driving productivity and building effective automation tools. Guided by his past experience in coding, he continuously strives to create practical, user-centric solutions. In his free time, Ido enjoys chess, history, and vintage video games.

Loadmill are a Gold Sponsor at AutomationSTAR 20-21 Nov. 2023 in Berlin

Oct 06 2023

Recommended Talks for New Automation Developers

This selection of recommended talks has been compiled by Chris Loder, Programme Chair of the 2023 AutomationSTAR Conference in Berlin, 20-21 November.

Hi again, everyone! After my last blog post about what I as an “advanced automator” would attend, I was asked to write another from the perspective of someone who is fresh and new to this wonderful world we are all lucky to be a part of called test automation. I will also add the same caveat that I did in my last post, if I don’t mention a particular talk, it doesn’t take anything away from it as ALL of the talks on the program are AWESOME!!!

Newbie Chris’s Hypothetical Agenda for AutomationSTAR 2023.

Day 1

Tutorial: As a programmer at heart, but a newbie to automation, I would probably choose Andy Knight’s tutorial Awesome Web Testing with Playwright. But if I didn’t have the programming background, I’d probably be attending Anne Kramer’s tutorial Getting Started with Visual Test Design for Automation.
Keynote Number 1: For this, I’d rush into the hall for the Keynote to get a good seat and ensure I don’t miss anything. Janna is a great speaker and I look forward to her keynote Streamlining Your CI/CD Pipeline with Test Automation.
Lunch: First thing for conference newcomers to know, it’s all about the food!! Be sure to enjoy some lunch to get fueled up for the great afternoon ahead! And it’s a great opportunity to network with fellow attendees and to take a moment to visit some of the great vendors that are present in the expo hall.
First Track Session: Just like I mentioned in my advanced agenda, I would be interested in attending Sophie Küster’s talk The Impostor’s Guide to Tooting Your Own Horn. Who of us hasn’t felt like an impostor at some point in our careers. But if I wasn’t’ feeling up for some soft skills instruction, I would then most likely choose to attend Gunesh Patil’s talk Behold MBT: A Guide To Improved Coverage, Collaboration, Mindset & You.
Break: Like the program says, mix and mingle in the Expo Hall. Grab a snack and a drink and do a little more networking. And don’t forget those vendors!
Second Track Session: As always, it’s hard to decide. But since AI is all the rage these days, I would most likely attend Marek Lof’s talk Do We Automate Everything for AI Testing!? But if my programmer side is winning on this day, I would then probably be attending Christoph Wolf’s talk Unit Testing for Test Automation Engineers.
Expo Party: Who wouldn’t want to enjoy “bites, beers and games!” ? This is another great opportunity for meeting new people and doing some excellent networking. And this takes place in the Expo Hall, so all of the vendors will be there for you to see demos and learn about their various services and tools and answer any questions you may have. Oh, and did I mention the bites and beers? Don’t worry, there are non-alcoholic options as well. (Yes, I know I copied this from my other blog post, but I honestly couldn’t think of a better way to put it. 😊)

Day 2

Morning Keynote: Again, be sure to get there early and get a great seat. Ryan Volker is a great speaker and his talk with Bastian Schürmann about their team’s journey from Confusion to Clarity is not to be missed!
Day 2 Track Session 1: Like I mentioned in my previous blog post, my inner 8 year old that dreamed of going to space would be attending Andrew Brown’s talk Apollo 13 and the Dangers of Automation Bias.
Break: Grab a coffee and a snack and continue to build up my network. Or take time to visit some of the vendors if I haven’t already had a chance.
Day 2 Track Session 2: As a new automation developer, Andrew Knight’s talk What Framework should I use for Web Testing? would be where I’d go.
Lunch: Remember I mentioned the food? Yeah, more of it today for lunch! And of course, networking as always. These connections you make today could benefit you for years to come.
Day 2 track Session 3: Like I said in my advanced blog post, it’s all the rage. So my new automation self would be attending Geosley Andrades’ talk Unlocking New Possibilities in Test Automation with Chat GPT.
Break: Just like in the morning, it’s a great chance to meet some new people while getting a beverage and a snack. Oh, and be sure to check out the vendors while there since this is the last chance to do so.
Lightning talks: Several speakers, each with 5 minutes to share a mini keynote! Sign me up!! Always a great opportunity to hear a little something on several topics in one go.
Final Keynote: Be sure to get every last minute from the conference and be front and center for Marie Cruz’s keynote about “intersectionality”. It’s going to be great!
Closing remarks: This is where that program chair guy Chris gets up and closes out the conference. I hear he’s a great speaker, so I better not miss that! 😉

As I mentioned previously, this is just one combination of talks and everyone’s journey will be different. Whichever path you choose, there is a wealth of knowledge to be gained at AutomationSTAR 2023 and I hope to see you there!

Regards,

Chris

Chris Loder – AutomationSTAR 2023 Programme Chair

Sep 22 2023

Data-oriented reporting for black box performance testing, part 2

This is part 2 of AutomationSTAR 2023 speaker Jakub Dering’s article on data-oriented reporting for black box performance testing.

In my previous article you’ve learned how to expand your visibility of test reports by adding variables to transaction names. After trying it out, you’ll soon learn that the complexity of your report may grow beyond comprehension and the more variables you add for comparison, the less readable the report becomes.

This happens because the number of transaction groups would grow linearly and final number of rows in your report would equal to (t * v) where t is a number of transaction groups and v is the number of possible variable combinations.

In this article I’ll show you how to deal with this complexity so you can still make use of the data, without spending sleepless nights on analyzing the reports which were once easy to digest, and now they look like this:

Example of a the expanded report with only 1 variable added as a parameter.

The idea is simple: we’re adding some variables we think may be significant to the service and check what happens when we increase those values, either during test runtime or by pre-fetching those variables, and use them inside the test scripts. In my example I’ll be using the same variable as in my previous article: “items in cart” because we know already my service would be affected by this number.

Other examples can be (from user perspective): number of accounts the user owns, duration of the user’s session, number of items in sent messages, number of items displayed on the screen, etc. Anything that we can quantify and rank and varies to some extent.

Due to the nature of performance test we usually end up with a large number of samples, and because the amplitude of response times is unpredictable, I’ve found the Spearman’s correlation rank coefficient to be suitable to limit the correlation to a single value.

Now, how to do it? I’ve used a Python script that extracts the number of items in cart, and ran the correlation check against the response times for any sampler group that contains this variable.

The output of the function for my report was:

The next question is, how to interpret this data? Spearman’s correlation rank coefficient measures strength of the monotonic correlation, in a range of <-1, 1>. The closer the value is to 0, the weaker the correlation of the data is. As a rule of thumb, the absolute values between 0,3 and 0,7 represent some correlation and the absolute values between 0,7 and 1 represent a strong correlation . In case of my report, you can see that the correlation for transactions “Open Product” and “Order Product” is close to 0,97 – this means a strong relationship.

The correlation is also positive, that means the response time grows. The remaining transaction response times are close to 0, that means there is no correlation between number of items in cart and the response times of the service.

If you want to prioritize your validation, you only need to sort the absolute values of the correlation factors, and remove the values below 0,3 from the list. This gives you a list of potential culprits slowing your services down.

Sep 14 2023

New tutorial added to the lineup

Exciting news! The AutomationSTAR programme has just gotten bigger, and we can’t wait to get everyone together in November.

Your passion for AutomationSTAR is awesome! There are so many of you joining us in November that we’ve had to add another tutorial – this thrilling addition to the lineup is a half-day tutorial with Janna Loeffler on test automation strategies. That’s on top of four incredible tutorials already on the programme, covering web testing, visual test design, BDD, automation frameworks, and more.

Drawing on real-world test strategies, Janna will help you create a streamlined, actionable, and impactful test automation strategy; to resonate with your organisation, irrespective of your team’s chosen development methodology. Check out below to learn more about it, and see the rest of our in-depth tutorials.

AutomationSTAR is going to be huge – we hope to see you in November!

see tutorials

Test Automation Strategies: Navigating DevOps, Agile, and Complex Technologies

Janna Loeffler

In the era of DevOps, Agile methodologies, and the ever-expanding landscape of intricate technologies, the traditional bulky test automation strategy document has rapidly lost its relevance. Testers are grappling with the challenge of succinctly communicating intricate testing objectives. Join Janne to get a streamlined, actionable, and impactful test automation strategy tailored for Agile, DevOps, and even the notorious Waterfall approach. Janna draws upon real-world test strategies from her experience, deconstructing the essential elements and unveiling the blueprint for crafting a comprehensive test automation strategy, irrespective of your team’s chosen development methodology. Prepare to revolutionize your approach to test automation strategies.

Awesome Web Testing with Playwright Andrew Knight

Andrew Knight

Everybody gets frustrated when web apps are broken, but testing them thoroughly doesn’t need to be a chore. Enter Playwright, a new open-source browser automation tool from Microsoft. Automate concise yet robust web app tests for a Kanban board web app with Playwright in TypeScript. This tutorial covers how to install and configure Playwright, how to perform interactions through page objects, how to conveniently run different browsers, capture videos, and run tests in parallel. After this session, you’ll be empowered to test modern web apps of any kind with modern web test tools. You’ll also be able to compare Playwright with other tools like Selenium WebDriver and Cypress.

Getting Started with Visual Test Design for Automation

Anne Kramer

Visual test design – better known as “”Model-Based Testing”” (MBT) – undeniably has a lot of advantages, in theory. The shift-left approach helps to clarify and validate requirements at an early stage. Modeling allows complex systems to be decomposed and thus mastered. The graphical representation facilitates understanding, and provides a living documentation of the test idea. In practice, reluctance is high: too complicated, too time-consuming, too expensive. See how to start visual test design in a smart way (namely top-down). Using a concrete example, Anne will show you how to structure the test into suitable packages, create some test scripts and identify the required keywords for automation. You only need a pencil and paper!

Exploring BDD Automation Patterns

Gáspár Nagy & Seb Rose

This workshop focuses on the automation aspect of Behaviour Driven Development (BDD) and demonstrates how you can improve your BDD automation solution with design patterns. Using different design patterns for software development is not new. In this workshop you will explore a couple of automation patterns Gáspár and Seb have collected while working on their book, “BDD Automation Patterns”. Learn about the importance and benefits of using Test Automation Patterns, explore a few test automation patterns that work well with BDD tools, like Cucumber or SpecFlow; and learn how to identify and extract new test automation patterns.

Automation Framework Essentials

Chris Loder

Automation is critical in today’s software delivery lifecycle, and yet many organizations struggle to keep their automation running. How can we mitigate difficulties and get consistent automation runs and results we can trust? The secret is implementing a solid automation framework, but that isn’t as easy as it seems. This tutorial will cover what an automation framework is, the benefits of having one, and the keys to a successful framework, including reliability, repeatability, and maintainability. Come away with ideas and a plan to start your very own automation framework, or improve your existing one!

Get ready to transform your testing with two whole days totally focused on test automation – book your tickets and just 350+ of your peers in Berlin, this November.

get tickets

Aug 24 2023

6 Essential Test Scenarios Using Postman Flows

This blog has been compiled by Beth Marshall, who will be at AutomationSTAR 20-21 Nov. 2023 in Berlin, showing you how to use Flows, Postman’s latest beta feature for low/no code API workflow development and testing.

TL:DR Use the links below to see a video on each Flow:-

📃1. Schema Testing
🔐2. Security Testing
🧑‍🏭3. Workflow Testing
🚫4. Negative Testing
🏃5. Performance Testing
✅6. Positive Testing

I recently volunteered at Testbash. A huge draw for me was meeting the wonderfully talented Julia Pottinger who, with the backstage support of her QA partner Orane Findley wowed the attendees with her characteristically clear and simple explanation of “Next Level API Automation”.

Julia’s talk included the following slide, which detailed 6 important high level scenarios to consider when performing API tests. I thought it might be good to show examples of how, you can execute tests in all 6 areas using Postman Flows.

Scenario #1 Schema Testing

As JSON parsing seems to be done on a field by field basis in Flows, it doesn’t currently seem possible to extract the entire schema from the response body in order to compare it with a stored value (correct me if I’m wrong folks!). However, if you want a visualisation that your schema test stored within the tests tab of your request has worked, then here is a good way to do that.

Scenario #2 Security Testing

You can use the create data block to add different data sets to run against a request. I walk through a very aesthetically pleasing example here. 😻

Scenario #3: Workflow Testing

As highlighted by Julia Pottinger, testing an end to end workflow through API calls is often essential. Think about it logically, how do you know if a delete call has worked if you don’t then check the data isn’t then available?

Using Flows to test workflows is a major use case for this Postman feature, because you can see exactly what’s happening. Here is an example workflow test using Postman Flows.

Scenario #4: Negative Testing

Here I cover 2 different but connected scenarios:-
1. You want to verify the status code and message (e.g. 400 Bad Request)
2. You want to verify a string within the error response (e.g. “the length must be 3-18 characters”

An added bonus is generating test data within Flows using both lists and records, which although might not be ideal for large test data sets, would be very handy for smaller ones such as testing what happens if 3 different invalid inputs are entered for the same request.

Scenario #5: Performance Testing

So we know that scaled and in-depth performance testing is one of the few things that Postman isn’t really built for. Although Flows is no exception to that, if its cheap and cheerful performance testing you’re after (say, executing the same scenario 10 times and checking the response time is under a certain threshold) then that is absolutely doable.

Scenario #6: Positive Testing

Last but certainly not least – time for the happy path! 🙌

In this example, I show how it is possible to execute 6 endpoints at the same time, and apply the same conditional logic to all of them to check:-
* status code is 200
* response body value is correct (“status: UP”)

This gives a really clear indicator when running a health check against several different API’s at the same time. Which is one of the huge benefits of using Flows.

And that’s it! I hope that structuring this post around Julia’s essential test scenario’s made sense to you, and that you were inspired to try Postman Flows next time you are looking to test your API.

Til next time! 👋

What’s Next?

Join Beth at for her workshop on using Postman flows. Connect with 350+ automators at AutomationSTAR 2023.