Welcome to the 330th issue!

You might have heard of Jev already — a new AI model that makes fast, cheap decisions.

But how can it help with test automation?

Today, I want to share with you different views and solutions:

• Playwright CLI + Jev is 98% cheaper and TWICE as fast by Filip Hric
• Jev can't run Playwright tests by Oliver Stenbom
• jevify by Ryan Angilly

So while it's efficient, it may not be great for everything.

Happy testing! 🙂

Dawid Dylowicz

Writing tests with AI is easy, but what about actual verification? Kavitha Namduri shares how she's trying to teach an AI agent to ask the right questions first.

Similarly, Jaclyn Leigh and Srilata Meka remind us that AI Is Changing Quality Engineering, But Engineering Judgment Still Matters.

Want to learn more about using AI in testing? Mohammad Faisal Khatri suggests starting with small steps and repetitive tasks, measuring progress by how much time AI saves.

Also, you may want to check out Jason Arbon's list of 15 Myths About AI and Testing.

Julia Pottinger explains why numbers like test count, pass rate and zero open bugs aren't the best metrics and suggests what to measure instead.

At the same time, Maaret Pyhäjärvi points out that Distance from fixes impacts value of same testing.

Testing is mostly about thinking. Ady Stokes describes several testing mindsets that can help you get better at it.

On that note, Zoltan Csontos explains Why manual testing still matters in the age of AI.

While page objects can provide structure to UI tests, they don't always make them easy to change. Ravi Rawal explains what else we can do.

Similarly, Chris Vo describes when to use Page Object Model (POM) and Component Object Model (COM) in Automation Testing.

Based on her experience, Marina Polkovnikova describes nine ways in which an AI coding agent and its controls can fail and how she dealt with that.

Additionally, Nirawit Best asks: AI writes our tests now. So where does QA go?

If you test the output of AI, Ali Süleyman Topuz has a few lessons to share, including why you shouldn't trust an LLM judge's score alone.

Moreover, Adaeze Atusiuba shares What Building My First LLM Evaluation Framework Taught Me, while Divya Mahalakshmi put together A Practical Guide to Testing Agentic AI.

The classic test pyramid is mostly about the cost and number of tests, but Dmitrii Kulikov shows why we should look at where and when each check runs.

Also, Geoff Cooney explains Why our PR gate has 17 tests, not 288, while Benjamin Bischoff advises on Managing the hidden cost of skipped tests.

Now that AI agents can quickly open and review pull requests, the test suite has become the slowest step. Allen Hutchison explains why they moved code out of a monorepo and what it cost them.

Similarly, Julia Pottinger shares her opinion on How to Do PR Reviews in the Age of AI, while Paul Stack points out that Your Agent-Written Tests Aren't Real Tests.

If you're looking for a better way to do evals, this is a handy repository of skills that can help you achieve that.

Similarly, Peterson Oaikhenah advises on Turning Bad LLM Responses into Regression Checks with LLMCheck

Using an AI agent to fix visual tests? Ilya Rodin explains why text works better than diff images for a model and shares a solution for that.

Speaking of Playwright, Tayguara Dias Reis shows how to do BDD with Playwright: Gherkin scenarios that business people can actually read.

If you're automating mobile apps, Mayvin Ramasawmy advises how to structure Maestro tests and harness, after an earlier article on writing A Passing Mobile Test in Minutes (Android & iOS).

Writing acceptance criteria in Gherkin for AI agents? Bruno Baketarić suggests adding new Provided and Otherwise extensions to help agents write better tests.

In relation to that, here's a good first step Towards Spec-Driven Test Automation: Part 1 by Gal Arav.

This is a great, experience-based overview from Tayguara Dias Reis on setting up quality gates in CI before a blue-green deploy.

In relation to that, Ankita Bhattacharjee describes a case where Our Test Results Lived in Two Places. Here's How We Fixed That.

I liked the way James Thomas used a history book to remind us how to view AI and how older tools can stay relevant alongside the new ones.

Joe Colantonio put together a great 30-minute compilation of insights from over a dozen test leaders on what AI cannot or shouldn't automate yet.

Claude's just had enough... 😂

Thanks for reading!

If you enjoy this newsletter and find it helpful in becoming a better tester, please consider sharing it with others.

Dawid Dylowicz