Every model in meryn reports its metrics through the same vocabulary of columns, split by period: train and test where holdout validation was turned on, plus a validation split that only Robyn produces — it's the only engine that cuts the series in three (see each engine's holdout setting: Robyn, Meridian, PyMC). Where an engine doesn't produce a number, the Compare page says why in the cell rather than leaving a blank. Compare is reached from a Benchmark node: it reads every connected model (up to 30) and lines up three at a time side by side — you pick which three from the dropdown above each column.
Reading the metrics
| Metric | What it tells you |
|---|---|
| R² | How much of the variation in your outcome the model explains. Closer to 1 is better, but a very high R² on the training split alone doesn't guarantee the model generalizes — check it alongside a validation/test split if you turned holdout on. |
| NRMSE (Normalized RMSE) | The model's typical error size, scaled so it's comparable across models with different outcome scales. Lower is better. |
| MAPE / wMAPE (Mean Absolute Percentage Error) | The average size of the model's error as a percentage of the actual value. Lower is better, and it's the same fit definition for all three engines. (A calibrated Robyn model has a separate number, Calibration MAPE, shown on its own detail page — that one measures how well the model matches your lift tests, not day-to-day fit, and never appears in this column.) |
| ROI | Total return per dollar of paid media spend, across the whole model — contribution from baseline, organic, and non-media factors is excluded from both sides of that ratio, so it's specifically "what paid media returned," not a blended number. |
| RSSD | How far each channel's modeled share of the outcome is from its share of spend — a solution where one channel gets 5% of spend but is credited with 60% of the outcome is exactly what a high RSSD flags. All three engines report it, with different standing (the cell note says which): in Robyn it's an objective the optimizer minimizes; in Meridian and PyMC it's a diagnostic that was merely observed. |
| Convergence | Whether the engine settled on a stable answer, as a verdict: Yes / Partial / No. The three engines answer it with methods that don't compare — Meridian uses its own built-in reviewer, PyMC is judged against our thresholds (ArviZ guidance), and Robyn (no MCMC) derives it from whether its two optimizer objectives converged — so compare the verdicts, not the machinery. Treat anything short of Yes as not yet trustworthy. |
The readiness panel
Reading the table above and deciding "is this model good enough to act on?" takes knowing what a bad number looks like for each column. Every finished Robyn, Meridian or PyMC model carries that judgement on its own detail page, as a short panel near the top (EDA nodes don't — there's no model to judge):
| Verdict | What it means |
|---|---|
| Ready to use | Every check we could run came back clean. |
| Use with care | Something is worth a look first — the panel lists what, and what to do about it. |
| Not ready | At least one check came back far enough out of range that the numbers shouldn't drive a budget decision yet. |
| Not enough to judge | The run didn't save enough for any check to apply. This is not a pass — re-running the model usually fills it in. |
The checks fall into two groups. Some ask whether the model holds up: does it explain your outcome at all, does it explain the held-out period nearly as well as the one it trained on, and how big is its error there. Others ask whether the answer is believable: is media credited with an implausible share of the outcome, is a channel's return far outside what the rest of the model sees, does a small channel take much more of the effect than of the spend.
Those first two are worth separating, because they catch different failures. A model can fit the training period beautifully and fall apart on the held-out one — that's overfitting. But it can also fit both of them badly, which looks fine to an overfitting check (nothing degraded, it was never good) and is its own kind of unusable.
Three things are worth knowing about how to read it:
- Not every threshold means the same thing, and each one says which it is. Some come from the statistics literature and don't depend on your data at all — whether the sampler converged, or produced enough effective draws, has a standard answer. Others are our own call about what's plausible. And a few have no universal answer at all (what counts as an implausible return depends on your business), so those are set against the spread of runs that already existed. Each finding shows where its line comes from, because a threshold from the literature carries more weight than one calibrated against a handful of past runs.
- A clean panel is not a guarantee. It means we didn't find any of the problems we know how to look for. A model can pass every check and still be wrong about your business — the panel narrows where to look, it doesn't replace reading the decomposition and the response curves.
- Without holdout, two checks can't run at all. Generalization and held-out error need a test split to exist. Rather than quietly pass a model on those, the panel leaves them out — which is its own argument for turning holdout on.
Comparing across engines
A few things make an apples-to-apples read harder than the numbers alone suggest:
- Holdout methodology differs. Robyn and PyMC hold out the final stretch of the date range; Meridian holds out scattered geo × time cells across the whole period. Two models with a similar validation score aren't necessarily being tested the same way.
- Organic vs. contextual isn't split the same way. Meridian distinguishes organic media from other non-media covariates; Robyn and PyMC treat both as plain covariates. A channel-by-channel contribution comparison between Meridian and the other two won't line up perfectly for anything mapped as Organic.
- A higher R² isn't automatically the better model. Prefer whichever model has the more honest validation setup (holdout turned on, converged if applicable) over whichever one simply reports the highest training-set fit. The readiness panel on each model's page is the short version of this check: a model with the best R² and a Use with care panel is usually a worse pick than a slightly lower one that came back clean.
Choosing a winner
On the Compare page, each column has a Mark as winner button — pick the model (and its allocator, if it has one) you consider the right answer. (The Benchmark node's dialog is the shortcut that takes you there.) This is purely internal curation — it doesn't show anything to anyone outside your team by itself. See Sharing & publishing for the separate step that actually makes a winner visible outside the team.