Trace a Prometheus relabel_configs block rule by rule: what each one matched, what it changed, and whether the target survived. The regex is anchored, which is the reason most rules do not do what they look like they do.
Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.
The answer appears here
Paste on the left and press Trace the rules. Nothing leaves this tab.
Nothing else to flag.
No formatting problems, and nothing the rules object to. Worth remembering what that covers: this reads the file you pasted, not the account or cluster it will be applied to.
No finding matches that filter.
Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.
A keep rule written as a substring search. Prometheus anchors it, so it matches nothing and every target is dropped, with nothing logged to say why.
labels:
__meta_kubernetes_pod_name: api-server-7d9f
__meta_kubernetes_namespace: production
job: kubernetes-pods
relabel_configs:
- source_labels: [__meta_kubernetes_pod_name]
regex: api
action: keepA mistyped regex. The rule does not match, so the target label keeps its old value, nothing errors, and the wrong value ships.
labels:
__address__: 10.0.0.1:9100
instance: old-value
job: node
relabel_configs:
- source_labels: [__address__]
regex: '([^;]+);.*'
target_label: instance
replacement: $1A value built into a __ label, which is discarded when relabeling finishes. Only __name__ survives the prefix.
labels:
__meta_kubernetes_pod_annotation_team: payments
job: kubernetes-pods
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_team]
regex: (.+)
target_label: __team
replacement: $1One of the few relabeling mistakes Prometheus refuses to start on rather than doing something surprising. These actions match label names and take no source_labels at all.
labels:
__meta_kubernetes_pod_label_app: web
job: k8s
relabel_configs:
- action: labeldrop
source_labels: [job]
regex: __meta_.*These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Prometheus anchors every regex as `^(?:...)$`, so `regex: api` does not match `api-server`. On a `keep` rule that drops every target, with nothing logged.
Instead:Write `.*api.*` when you mean a substring, or match the whole value. This is the single most common relabeling mistake.
It is a silent no-op. The target label keeps whatever it already had, nothing is logged, and the config reads as correct.
Instead:Check the source value against the regex in the trace. An empty source value usually means the source label does not exist on that target.
The default `replacement` is `$1`. With no group in the regex, `$1` expands to an empty string, and an empty replacement DELETES the target label rather than setting it.
Instead:Add the capture group, or set `replacement` explicitly to the literal you want.
Both match against label NAMES and ignore values. Prometheus rejects the whole configuration at load rather than ignoring the field, so the server does not start.
Instead:Use the regex alone. It selects which label names are removed or kept.
Every `__` label is discarded once relabeling finishes, with `__name__` the only exception. That is why `__tmp_` is the convention for scratch values.
Instead:Copy the value to a name without the prefix in a later rule if it needs to be stored.
Relabeling is a small language with one rule that catches almost everybody. Paste the labels a target arrives with and the rules you intend to apply, and this shows what each rule matched, what it changed, and whether the target survived.
This is the one to internalise. `regex: api` does not match `api-server`, because Prometheus anchors the pattern at both ends. On a `keep` rule that means every target is dropped, with nothing logged and nothing to see except an empty target list, which sends people to look at service discovery. Write `.*api.*` when you mean a substring search.
regex: api does NOT match "api-server"
regex: .*api.* matches
regex: api-.* matches
regex: api|db matches "api" or "db", nothing elseIt does not clear the target label and it does not error. The label keeps whatever it already had, the rule appears in the config looking correct, and the only symptom is a value that is subtly wrong or unchanged. The trace above shows matched or not for every rule, which is the fastest way to find the one that quietly did nothing.
That is the documented way to remove a label, and it is also how one disappears by accident. The trap is that `$1` is the DEFAULT replacement: a regex with no capture group expands `$1` to nothing, so a rule that looks like a straightforward copy removes the target instead. A reference to a group that does not exist, like `$2` against a one-group regex, does the same.
source_labels: [job]
regex: node # no capture group
target_label: copy # replacement defaults to $1
# -> $1 is empty -> "copy" is DELETED
regex: (node) # with the group, "copy" = "node"They ignore `source_labels` entirely and test the regex against each label NAME. This is one of the few cases where Prometheus refuses to start rather than behaving unexpectedly: a config that gives either of them `source_labels` is rejected at load. `labelkeep` is the more dangerous of the two, because it removes everything the regex does not match, including `__name__` if you forget it.
Every one of them, with `__name__` the single exception. That is what makes `__tmp_` the convention for intermediate values, and it is why a carefully constructed `__my_label` is missing from the stored series. If a value needs to survive, copy it to a name without the prefix before the end. The trace shows the label set both before and after this discard, because the difference is where the confusion lives.
They are concatenated with the separator, which defaults to a semicolon, and the regex is matched against the whole joined string. A rule with two source labels and a regex written for one value never matches. A label that does not exist contributes an empty string rather than failing, so a missing name gives you `value;` and matches the default `(.*)` perfectly well while producing nothing useful.
It applies the rules you paste to the labels you paste, and that is all. It does not know what your service discovery actually produces, so a `__meta_` label you invented will be traced exactly as if it existed. It also does not distinguish `relabel_configs` from `metric_relabel_configs`: the semantics are identical, but `__meta_` labels only exist during the first, so a rule that works in one and not the other is a stage problem this page cannot detect. Prometheus uses RE2, which has no backreferences or lookahead; a pattern using either would be rejected by Prometheus and accepted here.