A/B testing for mobile app onboarding in Indian fintech
This case study follows an anonymised Indian payments and lending app whose growth depended on more than downloads. New users arrived through paid campaigns and agent referrals, but many abandoned the mobile app before completing identity checks, linking a bank account, and reaching a usable dashboard. The team treated that journey as a measurable product problem rather than a marketing problem.
The redesigned flow replaced a seven-screen sequence with a progressive journey, clearer value explanations, resumable uploads, and a persistent primary action. Instead of assuming the shorter flow would win, the product and analytics groups ran a randomised A/B test against the existing experience. The results offer useful lessons for Australian teams working with bank-grade security, trusted credentials, and high expectations for mobile usability.
Activation improved from 41.2% to 46.8% among eligible users, while median completion time fell from 6 minutes 18 seconds to 4 minutes 51 seconds. Fraud indicators and duplicate-account signals stayed within the predefined tolerance, so the team could scale the treatment without weakening risk controls. The stronger lesson was methodological: local friction should shape the hypothesis, while behaviour—not intuition—should determine the rollout.
For Australian readers, the numbers are not a template to copy uncritically. A Sydney or Melbourne commuter may complete sign-up between trips, a regional customer may depend on a weaker mobile connection, and a local fintech must earn trust in a market sensitised to authorised push-payment scams. The transfer is the disciplined experiment loop, adapted to Australian dollars, privacy expectations, devices, credit checks, and banking habits.
Why onboarding needed a controlled experiment
New users encountered a dense sequence: mobile number verification, PAN and Aadhaar details, a selfie, bank-account selection, UPI setup, consent screens, and a credit assessment. Each request raised the chance of failure. A poor camera image caused another capture; an OTP delay stalled the page; a slow document upload appeared frozen. Support teams saw the symptoms, but the existing funnel mixed causes, device types, languages, and referral sources, making a single explanation difficult.
The proposed treatment shortened copy, separated document capture from other choices, saved progress locally, and displayed one dominant action. Its hypothesis was specific: reducing effort and uncertainty would raise successful activation without increasing account takeover, identity mismatches, or suspicious device rates. A related profile of data science routines in Indian e-commerce shows the operational context in which analysts turn broad business questions into measurable experiments.
Designing a reliable mobile onboarding test
The experiment enrolled 28,000 first-time Android users across Bengaluru, Mumbai, Delhi NCR, and Hyderabad. Eligible visitors were assigned at the edge of the mobile app to the current flow or the redesigned flow with persistent, anonymous assignment IDs. Returning users, employees, known test accounts, and traffic from a small list of instrumented campaigns were excluded. This reduced repeat exposure and prevented campaign-level differences from overwhelming the treatment effect.
The primary metric was activation: verified identity, linked bank account, completed consent, and a successful first dashboard visit. The team calculated the required sample size from a 41.2% baseline, 80% statistical power, and a two-sided 5% significance threshold. Secondary measures covered completion time, upload retries, support contacts, and abandonment by screen. Guardrails included duplicate accounts, fraud referrals, complaints, and later credit-account delinquency.
Both variants used the same eligibility rules, pricing, notifications, and customer-support staffing. The analytics team monitored sample-ratio mismatch, missing events, and device-level errors daily, but did not stop early for a favourable metric. A predefined decision rule required a confidence interval excluding no difference, no material guardrail breach, and acceptable data quality. This protected the team from mistaking a noisy daily swing for a durable improvement.
Segmenting users without over-reading the data
Overall averages can conceal where an onboarding change helps. The analysts examined pre-specified cuts by operating-system version, device memory, network quality, language, city tier, acquisition source, and prior banking relationship. They treated subgroup results as directional unless the experiment had enough observations and the interaction was consistent. This discipline prevented a few large cities from making a weak pattern look universal.
The largest relative gains appeared among users on lower-memory Android devices and those entering through agent-assisted referrals. Simplified rendering and resumable uploads helped when processing speed was limited. Users with a high-quality device and a prepared bank login gained less, which made sense because most of their original steps were already quick. Hindi and regional-language cohorts also showed positive movement, but some cells were too small for firm conclusions.
Qualitative interviews clarified the mechanism. People were less worried when the app explained why a document was needed before opening the camera, and they were more willing to continue after a recoverable error if progress had been saved. These comments did not replace the randomised result; they helped the team identify why the treatment worked and where a future treatment might fail.
Reading the results beyond a single conversion lift
At the end of the fourteen-day acquisition window, 13,904 users had received the control and 14,096 the redesigned flow. Activation reached 46.8% in treatment, compared with 41.2% in control, an absolute lift of 5.6 percentage points and a relative increase of 13.6%. The estimated 95% confidence interval for the absolute difference was 3.2 to 8.0 percentage points, so the result was unlikely to be sampling noise.
| Measure |
Control flow |
Redesigned flow |
Observed difference |
| Activation rate |
41.2% |
46.8% |
+5.6 percentage points |
| Median completion time |
6m 18s |
4m 51s |
87 seconds faster |
| Upload retry rate |
18.7% |
12.9% |
-5.8 percentage points |
| Day-30 retention |
24.4% |
25.1% |
+0.7 percentage points |
| Fraud referral rate |
0.9% |
1.0% |
+0.1 percentage points |
The operational gains matched the activation result. Upload retries fell by 5.8 percentage points, and median completion time dropped by 87 seconds. Day-30 retention moved from 24.4% to 25.1%, but its confidence interval included zero; the team recorded it as encouraging rather than conclusive. Fraud referrals were effectively flat, while duplicate-account creation remained below 1% in both groups.
Most of the lift came from users who had stalled on document upload or identity capture, not from people who completed every screen quickly. That distinction mattered because removing friction for confident users could have added little value while obscuring a real effect. The team therefore kept the original flow as a fallback for a small segment, rather than declaring a universal winner.
Rolling out the winner and adapting to Australia
The team released the redesigned flow through a feature flag in stages of 10%, 25%, 50%, and 100%, holding each stage long enough to compare fraud, support, and activation signals. A kill switch restored the control flow if error rates or suspicious-account indicators breached a threshold. This made the launch reversible and gave operations staff time to update scripts for the new journey.
An Australian implementation should preserve that operating model while changing the content. Identity checks would use locally appropriate documents and Australian credit-reporting information, and financial values would appear in AUD with GST treatment stated where relevant. Copy should reflect trust in Australian banks and regulators, but it should not copy Indian wording about PAN, Aadhaar, or account aggregation that would confuse local users.
Local context also changes test design. Sydney and Melbourne users may sign up while commuting, so a resumable flow and a clear return point have direct value. Australian consumers are alert to authorised push-payment scams, and many expect SMS or app-based multifactor authentication, making it risky to request a one-time code before explaining the bank-linking purpose. In regional areas and on older Android handsets, lower-resolution documents and unstable connections need dedicated usability sessions.
Baseline conversion should be recalibrated using Australian traffic instead of importing the Indian 41.2% rate. Sample size, language mix, device distribution, credit-policy effects, and banking-provider callbacks all need local estimates. A promising Australian hypothesis might compare the redesigned flow with an unchanged control while monitoring complaint themes and account-quality measures separately from activation.
Practical lessons for teams running similar experiments
The case worked because the team linked a customer problem to one primary outcome, protected the integrity of the test, and treated launch as another measurement window. The following points are directly relevant to Australian product, growth, risk, and analytics groups adapting mobile onboarding.
Four measures shaped the release decision:
- Activation used a completed, value-defining event rather than an app open or screen view.
- Guardrails kept fraud, complaints, and duplicate accounts in the launch decision.
- Sample-size planning prevented a promising result from being declared too early.
- Saved progress and clearer copy addressed the commonest sources of abandonment.
Three safeguards constrained interpretation:
- Day-30 retention was reported as directional because statistical uncertainty remained.
- Subgroup gains were not treated as universal without enough observations.
- The control flow remained available as a fallback during staged delivery.
The transferable pattern is not a particular button colour or form length. It is a closed loop from observed friction to a falsifiable hypothesis, controlled comparison, segmented interpretation, guarded rollout, and ongoing learning. Australian teams can apply that pattern to identity verification, account opening, lending journeys, or subscription sign-up while remaining alert to local fraud patterns and regulatory expectations.
Turning one experiment into a learning system
After launch, the analytics team moved the redesigned flow into the default experience and retained the control as a holdout for a limited period. A persistent user-level assignment layer meant later campaigns could compare messages, credit features, or banking-provider journeys without redefining earlier cohorts. Analysts tracked the same outcome tree across acquisition source, product version, geography, and language.
Experimentation also became part of quality management. Data-engineering alerts caught event loss before it distorted a readout; risk reviews examined unusual account chains alongside conversion; and customer-operations dashboards linked repeated complaints to specific screens. These practices reduced the chance that a short-term acquisition gain would conceal a costly downstream problem.
The lasting result was a mobile onboarding process that helped more eligible Indian users reach value while preserving the platform’s risk posture. It also gave the organisation a repeatable method for resolving uncertain product questions: define the mechanism, measure the right event, respect uncertainty, and scale in stages. That method remains relevant as Australian fintech products compete on speed without compromising trust, security, or transparent customer treatment.
