I built a mobile test automation practice for a team that had no automated tests at all. Today that practice validates 90+ critical customer journeys every night through CI/CD, and its job is simple to state: keep the codebase releasable, every day, with evidence.

The production side of that system, overnight regression runs that end in GO/NO-GO release decisions backed by screenshots and real-app evidence, is written up on the Akkodis blog. This post is the layer underneath it: the methodology. What to automate, to what standard, and how to keep a suite alive as the app changes. Examples are invented and generic, and they use Flutter's integration_test package; the patterns port to any E2E driver.

Automate journeys, not screens

The first decision that shapes everything else: the unit of automation is a customer journey, not a screen and not a widget. A journey is something a user came to the app to do, end to end: sign in, find a product, pay, see the confirmation. If it fails, a real person did not get what they came for. That is the definition of a release blocker, so it is also the definition of a test.

Starting from zero, do not inventory screens. Inventory journeys, then order them by what a failure costs. Money paths first: sign-in, search, purchase, anything that touches payment or entitlements. The first ten journeys on that list protect more of the business than the next hundred.

// integration_test/journeys/checkout_test.dart
void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  testWidgets('guest can search, add to cart and pay', (tester) async {
    final app = await AppRobot.launch(tester, signedIn: false);

    await app.search.query('notebook');
    await app.catalog.openFirstResult();
    await app.product.addToCart();
    await app.cart.checkout();
    await app.payment.payWithCard(TestCards.valid);
    await app.orders.expectConfirmation();
  });
}

A journey test reads like the user story it protects. That is deliberate, and it matters later: when this fails at 3am, the failure name alone tells the morning triage what is broken for users.

The coverage standard

The trap most young suites fall into: a hundred tests, all of them happy paths. The demo works and production still burns, because production users mistype passwords, lose signal in checkout and get their cards declined. So the practice defines coverage per journey, in three lanes, and a journey only counts as covered when all three are automated:

Diagram of the coverage standard: one journey, three lanes. The happy path lane covers valid input on a healthy backend, the negative-path lane covers declines, wrong passwords and expired sessions, and the edge-case lane covers process death, airplane mode and slow networks. A journey counts as covered when all three lanes are automated.

  • Happy path: valid input, healthy backend. Necessary, and the least likely place to find a release blocker.
  • Negative paths: the backend or the user says no. Declined payment, wrong password, expired session, empty results. The app must explain what happened, offer a way forward, and lose nothing the user built up.
  • Edge cases: the world misbehaves. Process death mid-flow, airplane mode at checkout, a slow network, first launch on a clean device. This lane is where real-world crash reports actually come from.

A negative-path test asserts recovery, not just error text:

testWidgets('declined card keeps the cart intact', (tester) async {
  final app = await AppRobot.launch(
    tester,
    backend: FakeBackend()..declineNextPayment(),
  );

  await app.cart.fillWith(items: 2);
  await app.cart.checkout();
  await app.payment.payWithCard(TestCards.valid);

  // The user is told what happened, in words, on screen.
  await app.payment.expectDeclineExplanation();
  // And loses nothing: the cart survives, retry is one tap away.
  await app.cart.expectItemCount(2);
});

Holding this standard is the practice. Anyone can automate a happy path in an afternoon; a team that automates the decline, the timeout and the relaunch has decided what quality means and written it down as executable truth.

Tests that survive the app changing

E2E suites die of maintenance, not of technology. The killer is coupling: ninety tests that each know how the payment screen is built become ninety edits every time it changes, and six months later someone quietly deletes the suite. The countermeasure is old and boring: every screen gets one robot (call it a page object if you prefer), and only robots touch finders.

// integration_test/robots/payment_robot.dart
class PaymentRobot {
  PaymentRobot(this.tester);
  final WidgetTester tester;

  // One place knows how the payment screen is built. When the
  // UI changes, one file changes, not ninety tests.
  Future<void> payWithCard(TestCard card) async {
    await tester.tap(find.byKey(const Key('payment-saved-card')));
    await tester.tap(find.byKey(const Key('payment-pay-button')));
    await tester.pumpAndSettle();
  }

  Future<void> expectDeclineExplanation() async {
    expect(find.textContaining('declined'), findsOneWidget);
  }
}

Two companion rules keep the suite honest. Finders target stable keys, not display text that copywriting will change next sprint. And there are no blind waits: a test that sleeps for three seconds is a test that flakes on a slow emulator and wastes two seconds on a fast one; wait for conditions instead. Flakiness gets treated as a defect with a root cause, never rerun until green, because the first time the team stops trusting a red suite, the suite is dead.

Nightly, not on every commit

A full journey suite on real builds is too slow for a pull-request gate, and pretending otherwise ruins both. So the practice splits the work honestly: pull requests run the fast lanes (analysis, unit and widget tests, minutes at most), and the complete journey suite runs at night, unattended, against a release build:

Diagram of the nightly loop: merges land all day behind fast PR checks, the nightly suite runs 90+ customer journeys against a release build on schedule, mornings start with screenshot-backed results and triage, feeding GO/NO-GO release decisions that keep the codebase releasable, with confidence flowing back into the next day's merges.

# nightly-regression.yml (pattern; any CI provider works)
on:
  schedule:
    - cron: "0 1 * * *" # while the team sleeps
  workflow_dispatch: # and on demand before a release

jobs:
  regression:
    runs-on: macos-latest
    strategy:
      matrix:
        platform: [android-emulator, ios-simulator]
    steps:
      - run: flutter test integration_test
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evidence-${{ matrix.platform }}
          path: build/evidence/

The morning ritual is the other half of the design: results come with screenshots, failures get triaged before new work starts, and every failure is either a real regression (fix today), a legitimate product change (update the journey), or suite debt (fix the test properly). Nothing gets muted.

Scale with QA, not around QA

Automation efforts fail socially more often than technically, usually by treating QA as the thing being replaced. This practice scaled precisely because it was built with QA: their journey knowledge is the suite's backbone. QA owns the journey inventory, the priority order and the definition of done for each lane, because nobody knows better how users actually break an app. Engineering owns the harness, the robots and the CI wiring. Both look at the morning results together.

That split also changes what QA's day looks like. The repetitive regression pass that used to eat every release cycle runs at night without them, and their time moves to the work automation cannot do: exploratory testing, new-feature risk, and feeding the next journeys into the inventory.

Releasable is the metric

None of this exists to make a dashboard green. The point is that "can we ship today?" stops being a feeling and becomes a reading: last night, the 90+ journeys that define the product either passed on release builds or they did not, with screenshots either way. Release decisions turn into GO/NO-GO calls made on evidence, and the codebase stays releasable instead of drifting into "we should probably test around a bit first".

That evidence-based release system, what runs overnight, what the GO/NO-GO output looks like and what it changed about shipping, is the subject of the companion piece on the company blog: Automating Mobile App Regression: Overnight Testing and Evidence-Based Releases. Akkodis Middle East also shared it on LinkedIn.

Where to start on Monday

If your team has zero automated E2E tests, the whole practice compresses into a first month that looks like this: list your journeys and sort them by the cost of failure. Automate the top three, happy path only, behind robots from day one. Then, before adding journey number four, take those three to the full standard: negative paths and edge cases. Wire the suite into a nightly job with artifacts, and hold the morning triage without exceptions. Ten journeys held to a real standard beat a hundred happy paths every time, and unlike the hundred, they will still be running in a year.