How Uber manages billions of dollars in payments

Uber handles tens of billions of financial transactions every quarter. They run between earners, spenders and merchants.

Each one has to be stored completely and correctly. Uber keeps them in LedgerStore, its storage for ledgers.

Why a log?

A ledger is the source of truth for any financial event at Uber. A record in it can't change after it's written.

So LedgerStore is append-only. A new record goes at the end, and nothing before it is touched.

Individual records are immutable. Cryptographic signatures let you check that none of them was altered.

Uber has moved more than a trillion entries into it from DynamoDB. That was a few petabytes of data.

What is wrong with a log?

An append-only log is cheap to write. It isn't cheap to read.

Say you want one rider's payments. In a plain log they're scattered among everyone else's, so you scan it.

Uber builds indexes on top of the ledgers instead. An index is a second table that answers one kind of lookup.

It needs trillions of indexes to index hundreds of billions of ledgers. Uber has built over 2 trillion unique ones.

Now every record lives in two places: the ledger and its index. What happens when the index hasn't caught up?

When the index must never be behind

Take the start of a trip. Uber places a hold on the rider's credit card.

If the trip is taken, the hold becomes a charge. If the trip is cancelled, the hold is voided.

Suppose the index serving the hold lags. The hold takes a while to show up on a read, and the card can be charged twice.

For this kind of index Uber uses a 2-phase commit. It writes an intent for the index first, then the record.

If the record write succeeds, the intent is committed. If it fails, the intent is rolled back.

Once a record is written, a later read can't miss its index.

When a few seconds of lag is fine

That guarantee has a price. The intent and the record are written one after the other. Every write gets slower.

Availability drops too. If the intent write fails, the whole write fails.

Not every index needs the guarantee. Uber's example is the payment history page.

A lag of a few seconds is acceptable there, as long as the payment appears on the page.

These indexes are eventually consistent. A separate process builds them in the background, isolated from the write path.

Uber generates them with Materialized Views, a feature of its own Docstore database.

So they add no write latency. They don't lower LedgerStore's availability either.

The log takes the writes. The indexes answer the reads.

Summary

1. Uber handles tens of billions of financial transactions every quarter.
2. LedgerStore is append-only, and its records are immutable.
3. Uber moved more than a trillion entries into LedgerStore from DynamoDB.
4. Uber has built over 2 trillion unique indexes on hundreds of billions of ledgers.
5. A strongly consistent index is written with a 2-phase commit: the index intent first, then the record.
6. The credit card hold at the start of a trip needs a strongly consistent index.
7. An eventually consistent index is built in the background with Docstore's Materialized Views.
8. The payment history page accepts a lag of a few seconds.

Sources

1. How LedgerStore Supports Trillions of Indexes at Uber. Kaushik Devarajaiah. Uber Engineering blog, 2024.
2. Migrating a Trillion Entries of Uber's Ledger Data from DynamoDB to LedgerStore. Raghav Gautam, Erik Seaberg and Abhishek Kanhar. Uber Engineering blog, 2024.
3. How Uber Migrated Financial Data from DynamoDB to Docstore. Piyush Patel, Jaydeepkumar Chovatia and Kaushik Devarajaiah. Uber Engineering blog, 2021.