Verifying Test Data Isolation in a Playwright Framework with Github Copilot

Joining partway through? 

This is Post 6 of Phase 3: Enterprise Scalability, part of my Playwright Automation Testing Udemy course.

Post 1 - Scale Test Automation lays out what this series is about.

Post 2  - Getting Your Playwright Framework Running gets the Playwright framework running on your own machine.

Post 3 - Running in parallel for the first time is where a single shared login collides with itself the moment you turn workers up from one to four- the exact problem this post exists to solve

Post 4  - Designing and Building Concurrency  built three classes, IdentityProvider, AuthenticationManager, WorkerResolver, each with one job, safe up to the size of your account pool. Nothing in that post actually called them yet.

Post 5 - Worker Scope, the Fixture, and Closing Out Concurrency is where those three classes finally got wired in: worker scope versus test scope explained properly, workerStorageState built as a worker-scoped fixture, identity isolation closed and proven.

This post covers a different category entirely, one that worker scope, fixture scope, and identity isolation together do nothing to fix on their own. It also does something the earlier posts in this series haven't: it uses GitHub Copilot directly to review the framework's own code for isolation gaps, with the exact prompts shown, not just the conclusions. That combination- verifying instead of assuming, then knowing how to judge whether an AI review's findings actually hold up- is the same reasoning senior QA and SDET interviews probe for, so this post doubles as practice for that conversation, not just a framework update. It's also the exact kind of investigation covered in the AI-assisted section of the Playwright Automation Testing Udemy course, if you want to build this same skill step by step rather than read the outcome of it here.

The question worth asking, even without a symptom

Post 3 - Running in parallel for the first time  named three isolation problems: identity, shared resources, and test data.

Two categories are still open going into this post, and this post checks both, in order, with the same rule for each: evidence, not assumption.

Test data first, since that's where the honest starting point actually is; I had never actually seen it fail. That's not proof it's fine.

Test data leaks only show up in a specific sequence,

Worth picturing concretely rather than abstractly:

Say a cart validation test on qa-cart.com applies a 20% coupon, checks the discounted total, and finishes. If cleanup didn't fully reset the cart, and a checkout test happened to run right after it on that same worker, that checkout test could start with a coupon it never applied, place an order at a discount it never asked for, and still pass, because nothing about it looks wrong from inside that test. Never seeing that happen is consistent with two very different realities: cleanup is solid, or that exact sequence- coupon test then checkout test, on the same worker- just hasn't happened yet. I didn't know which one was true, and guessing either way would have been dishonest.

Two related questions worth separating, since the rest of this post ends up answering both,

First: clearCart(), called in cart.validation.spec.ts's beforeEach, does it actually reset everything, an applied coupon included, or does it clear line items and just assume the rest follows?

Second, and this one only becomes visible after the first is answered: even if clearCart() itself is thorough, does that protection reach every test that might need it, or only the one file that calls it?

Worth defining the category properly before either question, since it's easy to think of this as "the cart bug" and miss how much wider it actually is.

Test data isolation is the guarantee that one test's actions can't leave state behind that changes how a later test behaves, on the same worker, using the same account, regardless of parallelism.

Cart and coupon state is one shape of it. A test that updates a shipping address shouldn't leave a different starting address for the next test. A test that uploads a review image shouldn't leave that image attached to an account another test also uses. Browser storage, localStorage, cookies, set by one test can get silently read by the next if the context gets reused. A test that edits a shared database record, a product's stock count, a review's rating, can change what a completely unrelated test finds when it queries that same record. clearCart() addresses the first of these, for the one file that calls it. Nothing in this suite currently handles the others, but knowing the full category is what lets you recognise the next one before it becomes a bug instead of after.

Checking it properly, instead of guessing

So, rather than read the code once and form an opinion, I had GitHub Copilot review it directly, with a prompt built to prevent the two failure modes AI review tends to fall into: inventing a problem that isn't there, or hand-waving past a problem that is. Here's the exact prompt:

Review this Playwright + TypeScript test automation framework for state isolation issues, specifically around parallel execution and test data cleanup. Context: the framework recently moved from a single shared login to a worker-scoped identity pool (IdentityProvider, AuthenticationManager, WorkerResolver, workerStorageState in parallelAuth.ts). That part is done and verified. I'm now checking a separate concern: whether tests running sequentially on the same worker can leak state to each other through the shopping cart.

Specifically review pages/ShopPage.ts, in particular the clearCart() method, and any spec files that apply or remove a coupon (search for "coupon" across tests/).

Questions to answer, with evidence, not assumptions:

1. Does clearCart() fully reset cart-related state before each test, including any applied coupon, or only the line items? Point to the exact lines that prove your answer either way, don't infer from method naming.

2. Is there a plausible sequence of tests, run on the same worker, one after another, where a coupon applied in one test could still be present when the next test starts? If yes, name the specific test files.

3. Separately, and don't conflate this with question 1 and 2: are there genuine reliability issues in clearCart() (retry behavior, loop termination, error handling) that could cause flaky failures? These are different from isolation bugs, a flaky failure is visible and fails loudly; an isolation bug can cause a silent false pass. Label each finding as one or the other, explicitly, and justify why.

4. For every issue you flag, state a concrete reproduction scenario, not a general risk statement. "This could theoretically cause X" is not useful without saying which two tests, in which order, would trigger it.

Do not rate severity without justifying it against the actual test suite's structure and execution order, not against generic best practice. If you're not confident something is a real bug given the code you can see, say so explicitly rather than flagging it as a precaution.

What came back

Copilot didn't answer from memory of what clearCart() probably does. Here's the trace, condensed, from reading the code, not recalling it:

Searched for regex `clearCart|coupon` (**/tests/**/*.ts), 13 results
Read pages/ShopPage.ts, lines 1 to 260
Read pages/ShopPage.ts, lines 260 to 347
Searched for regex `applyCoupon|removeCoupon|coupon` (**/tests/**/*.ts), 12 results
Read tests/cart/cart.validation.spec.ts
Searched for regex `clearCart\(` (**/tests/**/*.ts), 1 result
Read fixtures/parallelAuth.ts
Read fixtures/baseFixtures.ts
Read tests/checkout/checkout-hybrid.journey.spec.ts
Read playwright.config.ts
Ran: rg -n "async clearCart|removeCouponButton|while \(true\)|..." pages/ShopPage.ts
Ran: rg -n "beforeEach|applyCoupon|coupon" tests/cart/cart.validation.spec.ts
Ran: rg -n "workerStorageState|browser.newContext|scope: 'worker'" fixtures/*.ts
  (rg not installed in this terminal, switched to nl/grep for numbered output)
Ran: nl -ba pages/ShopPage.ts | sed -n '220,320p'
Ran: find tests -name '*.ts' | xargs grep -n "coupon"
Searched for text `coupon` (**/pages/CheckoutPage.ts), 21 results
Read pages/CheckoutPage.ts

Sixteen separate reads and searches, one of which checks whether the coupon-removal logic might actually live on the checkout page rather than the cart page. That's the difference between a review that inspects the code it's talking about and one that pattern-matches on a method name, and it's worth noticing before getting to what it actually concluded.

On the coupon question: clearCart() does more than remove line items; it explicitly removes an applied coupon too, in a block pointed to by an exact line reference, ShopPage.ts:306, not inferred from the method's name. The concern that started this whole review turned out to already be handled.

On the leak question: no plausible same-worker leak exists in the current suite, and the evidence chain is short enough to walk through directly. cart.validation.spec.ts has the only beforeEach that calls clearCart() anywhere in the codebase, and that hook runs three steps in order, clearCart(), then navigate(), then addFirstProductToCart(), before every test in that file. Only one test in the file is active, the coupon test itself.

On top of that, the fixture chain adds its own layer, every test gets a fresh browser context, and workerStorageState only ever carries the authenticated session, never cart contents. Between the beforeEach and the fresh context, there's no path left for a coupon to survive from one test into the next.

Separately, and worth keeping apart from the coupon question rather than folded into it: clearCart()'s loop looks like this:

while (true) {
  const removeLinks = this.page.locator('a.remove');
  const count = await removeLinks.count();
  if (count === 0) break;
  await removeLinks.first().click({ force: true });
  await this.page.locator('.blockUI').first()
    .waitFor({ state: 'detached' }).catch(() => undefined);
  await this.page.goto('/mycart/');
}

No exit condition beyond the cart looking empty, and a swallowed error on the .blockUI wait, .catch(() => undefined) means a failure there disappears silently instead of surfacing. On this exact code, the one I actually expected to get flagged, Copilot declined to call it a bug. Its own words: "I cannot tie that to a concrete flaky failure sequence in this suite, so I would not count it as a confirmed issue." It would have been easy to flag this and stop there, loops without exit conditions sound risky, and "add a max iteration guard" is generic, plausible-sounding advice that costs nothing to suggest.

That's exactly the shape of an inflated finding, technically defensible, unearned. Anyone can list generic risks against code that looks like this. Correctly declining to call it a bug, when the evidence doesn't support that conclusion, in writing, with the reasoning shown, is the harder and more valuable thing to demonstrate, in a review, in a framework, and in an interview room. If asked how you use AI code review, "I use it, and I still check whether its findings are actually backed by the code" is a stronger answer than either "I trust it completely" or "I don't use it."

The question the first review never asked

A clean answer should prompt the next question, not end the investigation. The first review was scoped to spec files that apply or remove a coupon, correctly, and it correctly found only one file that does that on purpose. It never asked whether a file that touches coupons at all could still inherit one left behind by a completely different test, on the same worker, that ran earlier. That's a genuine gap in the question, not an error in the answer.

Two checkout spec files exist, checkout.journey.spec.ts and checkout-hybrid.journey.spec.ts, and WooCommerce persists cart and coupon state server-side, tied to the account, not just the browser session. Neither fact was in scope for the first review. So I ran a second one, specifically targeting whether checkout tests are protected against inheriting state from an earlier test on the same worker:

Review this Playwright + TypeScript test automation framework for a specific test data isolation question. Context: WooCommerce, the backend for this store, persists cart contents and applied coupons server-side, tied to the logged-in account, not just to the browser session. tests/cart/cart.validation.spec.ts has a beforeEach that calls clearCart() before every test in that file, confirmed and closed already. This review is about a different, unexamined question: whether tests in tests/checkout/checkout.journey.spec.ts and tests/checkout/checkout-hybrid.journey.spec.ts could inherit cart or coupon state left behind by an earlier test, if that earlier test ran first on the same worker.

Specifically review both checkout spec files, and pages/CheckoutPage.ts and pages/ShopPage.ts's addFirstProductToCart().

Questions to answer, with evidence, not assumptions:

1. Do either checkout spec file have a beforeEach, or any other setup step, that clears the cart or verifies it's empty before the test body runs? Point to the exact lines, or state plainly that no such step exists if that's what you find.

2. Does the checkout test body itself assert anything about cart contents, item count, or applied coupon before or after calling addFirstProductToCart(), or does it proceed directly from adding one product to checking out? Again, point to exact lines.

3. Given how tests/cart/cart.validation.spec.ts's beforeEach and the checkout specs are structured, is there a real sequence, within how this framework actually assigns tests to workers today, where the coupon test could run before a checkout test on the same worker? State what you can and can't determine about actual execution order from the config and test structure, don't assume file order equals run order unless the code proves it.

4. If that sequence occurred, trace concretely what the checkout test would observe and whether verifyOrderExists(orderId) or any other existing assertion would catch a wrong item count or an unintended discount, or whether the test would pass regardless.

5. I don't plan to modify the checkout tests based on this review. Don't recommend a fix. Just characterize honestly what is and isn't currently protected, and whether this is a confirmed gap or a theoretical one given the evidence you can see, the same distinction you were asked to make in the last review of clearCart().

Do not soften or inflate the severity either direction. If the risk is real but currently unlikely given how tests are ordered, say exactly that, don't round it up to "critical" or down to "not an issue."

A confirmed gap, not a confirmed bug

This time the answer wasn't clean, and the distinction it drew is worth reading carefully, because it's the same discipline as the refusal above, just pointed the other way.

Neither checkout spec has a beforeEach, or any pre-test cleanup at all, confirmed directly against both files. Neither test body asserts anything about cart contents before or after adding a product, addFirstProductToCart() only verifies the product it just added is visible, nothing about item count, nothing about an existing coupon.

On whether the dangerous sequence, coupon test then checkout test, can actually happen on the same worker: provably plausible, not provably guaranteed. The identity pool assigns one account per worker for its whole shift, confirmed in parallelAuth.ts, but Playwright's own scheduling, which spec file runs on which worker at which moment, isn't hardcoded anywhere in this config, so it depends on how a given run happens to schedule things, not on anything the code fixes in place. And if that sequence did occur, the existing checks wouldn't catch it, the UI journey only confirms an order exists in history, the hybrid journey only checks that line_items.length > 0, and neither one verifies the order contains the right item, or the right total, or no unintended discount.

The review's own closing line states the distinction precisely: "Confirmed gap in checkout test protection/detection: yes. Confirmed always-happens contamination bug: no. Most accurate statement: real risk path, conditionally triggered, with weak detection once triggered." That's not a bug being hidden or a risk being invented. It's a category the earlier finding never claimed to cover, held to the same honesty standard as everything else in this post.

Checking whether shared resources even applies here

Post 3's third category, shared resources, two different workers colliding on something scoped to neither account, a coupon with a global usage cap, a product's stock count, was always the one piece of this series left unexamined. So I checked, with the same rules as before: evidence only, no assuming WooCommerce's general behaviour applies just because it plausibly could.

Review this Playwright + TypeScript test automation framework for whether a shared-resource isolation risk, distinct from the test-data isolation already reviewed, is even applicable here. Context: two prior reviews confirmed test-data isolation, whether one test's leftover state affects a later test on the same worker and account. This is a different question: whether two different workers, two different accounts, running at the same moment, could collide on something that isn't scoped to either account, a coupon with a global usage cap, or a product's stock count being decremented by concurrent checkouts.

Questions to answer, with evidence, not assumptions:

1. Does data/coupon.json, or any coupon referenced in tests/, have a usage limit configured, or are these single-use-per-customer or unlimited codes? Point to the actual coupon data used in tests.

2. Does any product used across tests/, particularly anything referenced in checkout or checkout-hybrid specs, have limited or tracked stock, based on what the framework's own code or test data shows, not on general WooCommerce behavior?

3. Do any two currently active tests, across any spec files, actually exercise the same coupon code or the same product concurrently, given how workers are assigned today? Name the specific tests if so.

4. Based only on what you find in questions 1 through 3, is this a scenario that could occur with the current test suite and test data, or is it not applicable at all because no shared, limited resource is currently being exercised by more than one test? Don't default to "theoretically possible with WooCommerce in general," answer only for what this specific suite's data and tests actually do.

If the honest answer is that no current test exercises a limited-usage coupon or limited-stock product, say that plainly, that closes the question rather than leaving it open. If the answer is genuinely uncertain from the code alone, say that too, and state exactly what would need to be checked in the WooCommerce admin to settle it.

The coupon fixture, coupon.json, has no usage-limit field at all, just code, discount percentage, product name, and expected total.

That code is used by exactly one active test, cart.validation.spec.ts. On products, the fixture used by name-based tests, product.json, carries no stock field either, and the only place a stock or inventory concept even appears in the codebase is a single commented-out line inside api-basics.spec.ts, inert, not active suite behaviour. One overlap did turn up: the same product, "Assorted Coffee," is used by two separate active tests, product.validation.spec.ts and product-review.seed.spec.ts, and those two files are eligible to land on different workers at the same time under the current config. But nothing in the repository tracks that product's stock or caps its availability, so being eligible to run concurrently isn't the same as being capable of colliding; there's no shared, limited counter for them to collide on.

The bottom line, in the review's own words: "For the current suite and data, this is not a confirmed shared-resource isolation problem." Not because the category is fake, Post 3 was right to name it, but because nothing in this specific framework currently exercises a resource that's actually shared and actually limited. The two checks that would make this provable beyond code alone, whether that coupon has a usage cap in the WooCommerce admin, whether that product's inventory is tracked there, are outside what a code review can settle, and worth naming as exactly that, an open administrative question, not an open test gap.

Why this connects back to worker scope, not test scope

None of this happened by luck, either, and it's worth tracing back to Post 5 to see why.

Post 5 - Worker Scope, the Fixture, and Closing Out Concurrency  spent genuine time on why loggedInPage stays test-scoped while workerStorageState is worker-scoped. That decision is doing more work here than it looked like at the time. If loggedInPage were worker-scoped instead, the same browser context, the same page, would be reused across every test on that worker, not just the same account. Cart contents, applied coupons, cookies, all of it would carry over automatically, by construction, no leak required, no clearCart() capable of fully undoing it, because the context itself never resets between tests.

Test scope isn't just correct for the workspace metaphor from Post 5, it's the mechanism that makes most of test data isolation free. A fresh context per test clears browser-side state automatically. clearCart()'s job is narrower than it looks, resetting the one thing that does persist across tests on purpose: server-side cart and coupon state tied to the account, since the account itself is meant to survive the whole shift through workerStorageState. Two layers, doing two different jobs: test scope resets what should never survive a test, worker scope preserves what should survive the whole shift, and clearCart() exists specifically for the one thing that lives in the gap between them.

What actually handles each of these, and where

Six examples were named earlier, and it's worth closing with what actually addresses each one, because they don't all get solved the same way, or at the same level.

Handled automatically, by architecture alone. Browser storage, localStorage, cookies, session state, none of this needs a line of cleanup code. Post 5's decision to keep loggedInPage test-scoped means a fresh browser context gets built for every test, and that alone wipes anything the browser itself was holding. This is the cheapest kind of isolation, because it costs nothing beyond a scope decision made once.

Handled by explicit reset, tied to the account, in beforeEach. Cart and coupon state is the example this post spent the most time on, clearCart(), called before every test in its file, verified, not assumed, to actually reset both. A shipping address and an uploaded file both fit this same pattern, they're account-level state that a fresh browser context does nothing to touch, since the account itself persists across tests through workerStorageState. The fix, if either of these needed one, looks like clearCart(), an explicit step, verified with an assertion, not inferred from a method name.

Not solvable by account-level reset at all. A shared database record, a product's stock count, a review's rating, and this post's own confirmed gap, checkout inheriting cart state, sit in a different category entirely. These aren't scoped to one account, resetting your own account's state doesn't protect against another account colliding on the same record, or against a test that doesn't call the reset step at all. This level needs either unique, disposable test data per test rather than a shared fixture record, or an explicit assertion at the point of use, exactly what checkout in this framework currently lacks, rather than trusting an earlier step that may or may not have run.

Three different problems, three different fixes, and the mistake worth avoiding is treating all six examples as if one pattern, one clearCart()-style reset, would cover them all. It wouldn't. Knowing which level a given piece of state lives at is what decides whether the fix is free, one function call, or a genuinely different strategy.

Closing the loop from Post 3

Which brings the loop back to Post 3, and it's worth closing honestly rather than tidily. Three categories, three different endings, and they shouldn't get flattened into one tidy sentence.

Identity: closed. Built in Posts 4 and 5, proven with a genuine run, nothing conditional about it.

Test data: this is the one with a confirmed gap, worth naming plainly. Within cart.validation.spec.ts, isolation is verified as clean, confirmed with evidence, not assumed. Between that suite and checkout, a confirmed gap exists: checkout tests have no protection against inherited cart or coupon state, and no assertion that would catch it if it happened. That's the finding from this post that matters most, and it's the one that counts if anyone's evaluating this framework's readiness today.

Shared resources: not a gap, and worth being precise about the difference. Checked directly, and for the suite as it exists right now, no active test exercises a coupon with a usage cap or a product with tracked stock, so there's nothing currently shared and limited enough to collide on. That's a closed answer for today, not a warning to act on. It becomes relevant the moment the test data changes, a limited-stock product gets added, a capped coupon gets introduced, and at that point it's worth checking again, the same way, with evidence, not from memory of this post.

I'm not proceeding further on either the checkout gap or the shared-resources question.

Not because they don't matter, but because deciding what's in scope for a given piece of work, and documenting what's out of scope honestly instead of quietly ignoring it, is itself the point being demonstrated here. If you're working through this framework yourself, both are genuinely worth trying: add a beforeEach to the checkout specs that verifies a clean cart, the same pattern clearCart() already establishes, and separately, add a limited-stock product to the test data on purpose, then see whether the shared-resources question this post closed as "not applicable" starts applying.

This distinction, actual gap versus future-facing risk versus genuinely out of scope, is worth carrying into an interview directly. Anyone can list every possible risk in a codebase; that's not the hard part. The harder question, the one that actually gets asked in a senior conversation, is how you decide what to fix now, what to document and defer, and what to rule out entirely, and whether you can explain that reasoning with evidence instead of gut feeling. This post is as much a demonstration of that decision as it is of the specific bug it found.

Two workers, one machine, solved. The next post breaks out of that box entirely, sharding this same framework across multiple machines at once, and packing it into Docker so it runs identically anywhere it lands. Same architecture, same guarantees, now proven at a scale a single laptop was never going to reach.

This is Post 6 of the Phase 3: Enterprise Scalability series.  Post 1 · Post 2 · Post 3 · Post 4 · Post 5· Repo link · Course link