How Uber Tests Fraud Rules Before They Go Live — and How to Do It in Your Business
Picture a fraud analyst at Uber with a hunch. A pattern of signups looks like promotion abuse, and a rule could stop it. But Uber’s fraud work covers payment fraud, account takeover, collusion between drivers and riders, and promotion abuse, and the same rule that catches abusers can also catch real riders on their way to the airport. A wrong rule doesn’t fail quietly. It fails on customers.
Uber’s answer was not to write rules more carefully. It was to make sure no rule touched a user until it had been watched on real traffic first.
What Uber built: Mastermind and the Evaluate status
Uber described its system in a March 2017 engineering post by Yifu Diao and Isabel Geracioti, Mastermind: Using Uber Engineering to Combat Fraud in Real Time. Mastermind is a rules engine. It runs fraud checks at what the post calls checkpoints, user events such as signup, and each checkpoint holds one or more independent rules. A rule can return actions like rejecting a trip request, adding to a blacklist, asking for additional validation, or slowing down a moderate-risk user’s ride requests.
Why rules and not only machine learning? The post gives the reasons plainly. Fraudsters act faster than a model can be trained, there are cases where heuristics are more accurate than training data, and Uber’s fraud system hooked into about two dozen user events, more than a small team could build a model for each. So Uber builds models for the most important and complicated events, like trip requests, and uses rules for the rest.
That choice puts a lot of weight on rules, which makes the next part important. In Mastermind, a rule can be Active or Inactive, and there is a third status:
“A status of Evaluate means that the rule is being run but won’t actually affect the user.”
The rule sees real traffic and records what it would have done, and that data is logged to analyze later. In the post’s words, this “allows analysts to see how often a rule fires and how it performs in general.” Only “once they know the rule is working” do they switch it to Active.
Uber used the same idea one level up. When it rebuilt Mastermind, the second version ran in shadow alongside the first: it “didn’t actually make decisions, but rather recorded the results of the decisions it would have made.” The team compared the two systems’ results and retired the first version once they were sure the second was making accurate predictions.
The other change in the second version was who could write rules. Before Mastermind, analysts couldn’t write rules themselves, and it took an engineer about an hour to write and deploy one. In the second version, analysts author rules in a front end that validates the syntax and lets them test a rule with feature values. As the post puts it, engineers “don’t need to get involved.”
The numbers Uber published
All of these are from the Mastermind post:
- Under the first version, it took analysts a year and a half to implement 300 rules. Under the second, they implemented 200 new rules three months after the front end launched.
- Running 300 complex rules takes 30 milliseconds.
- A rule change in the front end reaches production in about a minute, compared with about an hour before.
- The second version took four months to build and two months to migrate.
- The fraud system hooked into about two dozen user events.
The post does not say how long a rule stays in Evaluate, or what threshold makes a rule “working.” That decision belongs to the analyst, informed by what the logs show.
What transfers to your business, and what does not
You do not have Uber’s traffic. A rule in Evaluate at Uber’s scale collects evidence quickly. Yours might fire twelve times in a week. You also probably don’t have a team of fraud analysts; you have a founder, a support lead, maybe one engineer.
That changes the details but not the method:
- Low volume means look at cases, not only counts. With twelve hits you can open each one and ask whether it was a real customer. That is often more informative than a rate.
- Waiting costs less than you think. A rule that runs in evaluate for two weeks while abuse continues costs you two more weeks of abuse. A rule that goes live wrong costs you real customers, support tickets and trust, and those are harder to win back.
- Start the whole system in shadow, not only each rule. Uber shadow-tested a complete new version before trusting it. When you add a fraud tool, the same logic applies: let it watch first.
How to do it in your business
These steps map to features in Prynt Control.
1. Begin with a mirror period. In mirror mode, Prynt only observes for the first 14 days. Nothing changes for your users. On day 14 you get “what we would have stopped”, case by case: each signup, trial or checkout Prynt would have challenged or blocked, with the reasons. You review it the way Uber’s analysts reviewed Evaluate logs. It is also the moment to find the customers you would never want to lose and check that none of them are on the list.
2. Put every new rule in evaluate first. After the mirror period, new rules still start in evaluate mode. The rule runs on live traffic and records its hits without affecting anyone. Before you activate it, answer three questions:
| Question | What to look at |
|---|---|
| How often does it fire? | Hits per day at the checkpoint it sits on |
| Who does it hit? | Open the cases: how many are paying or long-standing customers? |
| What would it do? | The action you plan to take, and whether a softer one would do |
3. Attach rules to a use case, not to “everything.” Mastermind organizes rules by checkpoint. You can do the same today: call Prynt separately at signup, free trial, bonus or promo, login, checkout and extra seat, and keep each checkpoint’s rules apart. A rule written for bonus abuse should not quietly affect logins. In Prynt Control, use-case decisions do this for you: send the checkpoint as useCase and each one returns allow, step-up or block, with its own reasons.
4. Prefer step-up over block when you activate. Uber’s own actions include slowing a user down and asking for additional validation, not only rejecting. When a rule graduates from evaluate, start with step-up and move to block only for the patterns you are sure about. Prynt never blocks on its own: it returns the decision, and your code chooses what happens.
5. Keep watching after activation. A rule that was right in its evaluate period can drift as fraud changes. Check its hit rate and the cases it catches every few weeks, the same way you checked them before turning it on.
A short, honest closing
Uber built Mastermind for a scale few businesses will ever see, with a team to match. But the habit at the center of it is small and cheap: let a rule watch before it acts, and let the logs, not the hunch, decide when it goes live. You can start that habit on day one.
Prynt is not affiliated with Uber; this post describes Uber’s published work.
Try it free
Prynt is device intelligence with a free plan: 5,000 identifications a month, visitor IDs with confidence and bot detection — no credit card. Upgrade to Pro for VPN, proxy, tampering and behavioral signals.