Measuring delivery quality with numbers instead of feelings

A supplier comparison built on feelings produces a different answer every week. One slow order makes a supplier look bad; one fast order makes it look good; a big order that arrived early hides three small orders that never finished. The fix is not more intuition. It is four numbers, a fixed sample and a rule for which orders count.

This guide defines the four numbers, shows the formula for each, and works through a small sample so you can check the arithmetic. It also explains why an open order is not a failure and why the median is more useful than the mean.

The four numbers

Fill rate. Delivered units divided by ordered units, across the orders in the sample. An order that delivered 7,300 of 10,000 contributes 7,300 to the numerator and 10,000 to the denominator. A completed order contributes its full quantity. A canceled order contributes zero delivered against its quantity if you want the strict reading, or you can exclude it and report cancellations separately. Pick one rule and keep it.

Start latency. The time between the supplier accepting the order and the first movement in the count. If you do not have an acceptance timestamp, use the payment time and label the number as payment-to-start, not supplier start. The two are different and mixing them inflates the supplier's latency with the panel's placement time.

Completion time. The time between the first movement and the final state. For a drip order, this includes the pauses between runs, which is the point: the customer experiences the whole span, not the average run.

Retention. The count at day seven or day thirty, divided by the delivered units. Use the start count plus the delivered quantity as the baseline, not the public count alone. A retention reading without the baseline is a number without a denominator.

Metric	Formula	Source	A weak reading looks like
Fill rate	Delivered units divided by ordered units	Quantity and remains	Several partial orders in the same sample
Start latency	First movement minus acceptance	Order events and your poll	A median far above the published start
Completion time	Final state minus first movement	Completion time and first movement	A long tail on ordinary quantities
Retention	Count at day seven or thirty divided by delivered units	Your own count checks	A steady fall after completion

Definitions, not benchmarks. The thresholds this panel applies are in the policy section below.

Which orders count

Use orders that have a result. A completed order has a delivered quantity. A partial order has a delivered share and a remains. A canceled or refunded order has a final state and a money outcome. An order that is pending, placing, queued or in progress has no result yet, and counting it as a failure makes a supplier look worse during a busy hour.

There is one exception. If an order has been in progress for longer than the service's normal completion time and the remains has not moved, it is a stalled order. Report it separately as a stalled count, not as a zero. The distinction keeps the fill rate honest and still makes the stall visible.

Keep the sample homogeneous. Compare the same platform, category, quality label and quantity band. A 100 unit order and a 50,000 unit order on the same service are different products for measurement purposes. A sample that mixes them measures the mix, not the supplier.

The symptom that brings people here

What you see: one bad order makes you want to change suppliers, or one good order makes you want to buy a large package, or the average looks fine while the last three orders were late.

What it means: the sample is too small, the metric is an average with a long tail, or the open orders are being counted as results.

Next action: write the four numbers for the last five terminal orders on the same service and quantity, use the median, and compare the reading with the published start and the refill window.

The feeling is not wrong. It is a signal that something changed. The numbers are how you find out what changed and whether it is worth acting on.

Worked example

Take a hypothetical sample of ten Instagram followers orders, each for 10,000 units. Nine complete in full. One is partial with 7,300 delivered and 2,700 remains. The fill rate is the sum of delivered units divided by the sum of ordered units: 97,300 divided by 100,000, which is 97.3%.

The single partial order has a fill rate of 73.0% on its own. The sample fill rate is 97.3% because the other nine were full. Both numbers are true, and they answer different questions. The sample number tells you what a customer experiences on average; the single number tells you what happened on that order.

The money side uses the same quantities. At the median rate, a full 10,000 unit order costs $7.35. The partial order delivered 7,300 units, so the missing 2,700 is the refund base. The fill rate and the refund are two views of one result: one counts units, the other counts money.

Now add the time dimension. Suppose the nine full orders started in a median of twelve minutes and completed in a median of four hours. The partial order started in forty minutes and stopped after six hours. The start median is twelve minutes, not the average of twelve and forty. The completion median is four hours, not the average of four and six. The partial order is visible in the tail without moving the median enough to hide the other nine.

Why the median, not the mean

Delivery times have a long tail. One order that sat in a queue for a day moves a mean by hours and a median by minutes. If you report the mean, one bad order can make a supplier look unacceptable when nine of ten orders were fine. If you report the median and the count, the bad order is still visible in the tail and the typical experience is not distorted.

Report both when the sample is small. The median tells you the typical order; the maximum tells you the worst case. A supplier with a good median and a bad maximum is a supplier that usually works and occasionally fails hard. That is a different risk from a supplier with a bad median and a tight range.

The same logic applies to retention. A single order that lost a large share can move a mean far more than a median. Keep the individual readings next to the summary so the tail is not hidden.

How to compare two suppliers

Hold four things constant: the service, the quantity band, the link type and the period. Then collect at least three terminal orders per supplier. Five is better. Compute the four numbers for each and compare the medians side by side.

Do not compare a supplier you tested during a platform limit with a supplier you tested during a quiet week. The period is part of the sample. If the two tests happened in different conditions, the comparison is a comparison of conditions, not suppliers.

Do not compare a supplier on a service you have never tested with a supplier on a service you know. The first reading is a test, not a baseline. Run the same test on both, then compare.

Write the comparison down. A table with four rows and two columns is enough. The act of writing it prevents the next conversation from starting from memory.

The policy thresholds this panel uses

The panel applies three thresholds as orders accumulate. A supplier with a delivery rate below 95.0% is not preferred. Below 90.0% it is switched off. A service with a start time above 30 minutes is a fallback, chosen only when nothing faster fits.

Those are policy choices, not market statistics. They say what this panel is willing to sell, not what the market average is. A different panel can set different thresholds. The useful part is that the thresholds are written down and applied to the same four numbers, so a supplier's standing is not a matter of taste.

Use the same discipline in your own panel. Pick a floor, write it down, and apply it to a fixed sample. A floor that moves after every bad order is not a policy; it is a mood.

The table you can build

One row per order. The columns: public id, service, quantity, charge, paid time, accepted time, first movement, final state, remains, completion time, count at day seven, count at day thirty. The first ten columns come from the order record; the last two come from your own checks.

Then compute the four numbers over the rows that have a result. A spreadsheet is enough. A small script over SQLite is enough. The important part is that the raw rows are kept, because a summary without the rows cannot answer the next question.

Update the table when the state changes, not only when the order is placed. A row frozen at placement is a record of an intention, not a delivery. The final state, the remains and the completion time are the fields that make the measurement possible.

Four mistakes that make the numbers lie

Counting open orders as failures. An order in progress is not a zero. If it is included in the denominator and excluded from the numerator, the fill rate falls every time a supplier is busy. Exclude open orders, and report stalls separately. The sample stays honest and the busy hour stays visible.

Mixing placement latency with supplier latency. The payment-to-start number includes the panel's placement step. The acceptance-to-start number does not. If you compare a supplier on payment-to-start while the panel was slow that day, you are measuring the panel. Use the acceptance timestamp when you have it, and label the number when you do not.

Using the public count as the delivered quantity. The count can be lower than the delivered quantity because accounts dropped, or higher because the platform is still validating. The delivered quantity is the supplier's report, and the count is the display. Use the report for the fill rate and the count for retention. Mixing them makes both numbers wrong.

A fourth mistake is comparing different periods. A supplier tested during a platform limit and a supplier tested during a quiet week are not in the same experiment. Write the period next to the sample. A reading without a date is a reading that cannot be compared with the next one.

What to do next

If one order is bad and the sample is one, wait for two more before you change suppliers. A single order is an anecdote, and the cost of switching on an anecdote is a new supplier with an unknown baseline.

If the median is fine and the tail is bad, keep the supplier for ordinary orders and route the deadline-sensitive orders to a faster row. The tail is a routing problem, not necessarily a supplier problem.

If the fill rate is below the floor across a real sample, stop using the service. The refund recovers part of the money, not the customer's time. A service that repeatedly stops short is a service that costs more than its rate.

If retention is the weak number, check the refill window before you change suppliers. A drop inside the window is a refill question; a drop outside it is a new order question. The measurement tells you which one you have.

For the order states that feed the table, read the order lifecycle. To follow a live order, open the tracking page. For the platform's current availability, open the status page, and for the price side of the same comparison, read how prices are made.