Instrumentation doctor

Verifies the instrumentation, not the network. SmokePing will happily tell you a target is at 0% loss when nothing is measuring it, and Grafana will happily render an empty panel that looks like a quiet night. This tool catches the case where a newly added measurement or panel looks fine and silently charts nothing.

Every check compares what one stage of the pipeline produces against what the next stage expects:

exporter source → dashboard query → datasource → provisioning → Grafana
      config (DB) → generated Targets → SmokePing → RRD → exporter → TSDB → panel

Source: shared/modules/doctor/.

Running it

# From a checkout — auto-detects the repo root
PYTHONPATH=shared/modules/doctor python -m doctor

# Explicit root, machine-readable, or with findings on passing checks too
PYTHONPATH=shared/modules/doctor python -m doctor --repo-root . --json
PYTHONPATH=shared/modules/doctor python -m doctor --verbose

Exit code is 1 if any check fails; warnings do not fail the build. The Grafana dashboards & provisioning CI job runs the static checks on every PR.

instrumentation doctor — static checks against /home/smokingpi/smoking-pi

[ok  ] dashboards-parse              17 dashboards parse
[ok  ] provisioning-yaml-parses      4 provisioning files parse
[ok  ] dashboard-uids-unique         17 uids unique per tree
[ok  ] one-default-datasource        exactly one of 4 datasources is default
[ok  ] datasource-uids-resolve       45 references resolve
[ok  ] datasource-plugins-installed  4 datasources have a usable plugin
[ok  ] dashboards-are-scanned        17 dashboards sit under a scanned path
[ok  ] panel-measurements-written    90 measurement predicates match …
[ok  ] panel-tags-written            98 tag references match …
[ok  ] text-stats-name-their-field   7 text stats name the field they show

9 ok, 0 warn, 0 fail, 0 skipped

The static checks

Each one exists because the corresponding failure actually happened here.

Check	Catches	Real incident
dashboards-parse	Unparseable dashboard JSON	—
provisioning-yaml-parses	Unparseable provisioning YAML	—
dashboard-uids-unique	Two dashboards sharing a UID in one tree; a dashboard with no UID at all (provisioning assigns a random one, breaking every deep link on reprovision)	—
one-default-datasource	Two datasources claiming isDefault	v2.5.0: ClickHouse was set default, InfluxDB mode provisions both files from a read-only mount, and Grafana refused to start at all
datasource-uids-resolve	A panel pointing at a datasource uid nothing declares	Eight dashboards referenced clickhouse-alt, which never existed
datasource-plugins-installed	A datasource whose plugin the image does not install — it provisions fine and then cannot answer a query	ClickHouse before the plugin was baked in
dashboards-are-scanned	Dashboard JSON in a directory no provider walks	The entire ClickHouse dashboard set, on disk and never loaded
panel-measurements-written	r._measurement == "X" where no exporter writes X	—
panel-tags-written	r.measurement_type and friends — a filter that can never match; on the ClickHouse dashboards, a category / measurement_type value the ClickHouse exporter never writes	DNS panels filtered on measurement_type='latency' / category='DNS_Resolvers', where the exporter writes dns_latency / dns; the panels were fixed in v2.5.0; four ClickHouse $target variables kept the raw names, unnoticed, until the check read SQL too
text-stats-name-their-field	A stat panel whose query answers with text (a tag kept, or _value built from a tag or a string) and whose reduceOptions.fields is empty: Grafana reduces numeric fields only, so the panel shows its "no value" text over data that is there	The Overview's whole first row ("unknown", "not collected yet") and the Wi-Fi dashboard's SSID and BSSID ("No data"), v2.14.0–v2.15.6. Recognized from the query alone: a string field shown as-is (r._field == "owner") looks like any number and is not caught
overrides-match-a-series	A field override matched byName on a yield(name:) value the query does not otherwise produce: Grafana names a Flux series by its field and tags, so the override never applies and the panel draws with the wrong unit, axis or scale. Panels that set displayName are skipped	The CPE microcuts' TX failures drawn in dBm on the signal's axis, and six Wi-Fi Link overrides (noise/SNR units, channel busy's 0–100 % scale, link quality's axis)
links-name-provisioned-dashboards	A dashboard uid named in common/links.py (the deep links in every alert and assistant answer) or the web admin's category map that no dashboard in the InfluxDB tree carries: the link opens Grafana's "not found" page	The alerts' graph still pointed at the SmokePing Latency twins retired after the 2026-10-15 audit, and the Overview's index linked one of them
dashboard-definitions-match	A dashboard literal that restates a shared definition with another number than common/: the loss-event bar (1.5 pings' worth, EVENT_LOST_PINGS), the cut-window bar and its drawn line (MICROCUT_LOSS_PCT), the weak Wi-Fi step of a signal panel (WIFI_WEAK_DBM). A dashboard cannot import Python, so a value retuned in common/ would leave the panel saying the old thing while every alert and answer says the new one	— (added with the evidence-consistency audit, 2026-10-18: every literal matched, nothing checked it)
service-taxonomy	templates/taxonomy.yaml that does not validate (a rule category outside the vocabulary, a match with two keys, a missing source), or a seed target in templates/targets.yaml no rule labels (Service taxonomy)	— (added with the taxonomy, 2026-10-22)

Where the vocabulary comes from

panel-measurements-written and panel-tags-written do not compare against a hand-maintained list. They read the exporter source with ast and collect what it actually writes — the string literals passed to Point(...) and .tag(...). A copied list is exactly the kind of thing that drifts silently, which is the bug class this tool exists for.

rrd2influx.py writes its measurement indirectly (measurement = measurement_for(...), then Point(measurement)), so the extractor resolves one level of indirection through the helper's string-literal returns. Without that it sees only cpe_latency and reports every ping panel as broken — which it did, on the first run, before the resolution was added.

Panel queries and template-variable queries are both checked, including panels nested inside collapsed rows. The DNS_Resolvers mistake lived in a variable query, not a panel.

A pivot() turns values into columns, so r.loss after pivot(columnKey: ["_field"]) reads a field the query selected, not a tag. A query that joins streams (the Overview's gateway cut windows, split by the radio's deaf samples) can pivot more than once. Each pivot exempts the columns it made from there on: the values selected with r.<key> == "x" or named with set(key: "<key>", value: "x") before it. For text-stats-name-their-field, only the final statement's keep() counts (a statement starts on a line outside every bracket that does not continue a |> pipe); a keep() that shapes an assigned stream for a join says nothing about the answer. _value: r.<column> after a pivot reads as a number only when the query named that column itself with set(); a stored field picked with r._field == "ip" may be a string, so it still counts as text.

ClickHouse queries are SQL, and a column name there is always valid, so the check compares values instead: every category / measurement_type literal a query over smokeping.* compares with =, !=, <> or IN must be one the exporter's category_for() / measurement_type_for() can return, read from rrd2clickhouse.py the same way (literal returns, and every value of a CATEGORY_MAP.get(..., "unknown") table). PostgreSQL variables on those dashboards speak the database's vocabulary (dns_resolvers) and are skipped. A valid value on the wrong dashboard is not caught.

Live checks (--live)

python -m doctor --live

These checks need the running host. Each exists because the failure happened here, and they share a shape worth naming: the broken thing keeps looking healthy, so nothing goes red and nobody looks.

deployed-code-current

Compares the sha256 of every deployed .py — the module's own source and the shared common package — against the repository, for each running container built from an image that copies source in.

This is commit dde5e36 ("the flap fix never reached the deployed container"), and it recurred: an image failed to build, the failure was masked by a shell pipeline's exit code, docker compose up -d recreated the container from the three-week-old image, and every surface reported success while the fix sat only on disk. A stale container starts, logs cleanly and serves requests; there is no symptom to notice.

A container that is not running is not drift — a profile switched off is a deployment choice. A container it cannot read is reported as "cannot verify", never as drift: claiming a difference it did not measure would be the same class of bug.

container-dns-fresh

Compares each running container's /etc/resolv.conf nameservers against the host's. Docker writes that file once, at container creation, so a container created while a VPN was up keeps that VPN's resolver forever — and it dies silently when the VPN goes away.

That cost nine of eighteen targets for ten days: every hostname target read 100% loss, every raw-IP target was fine, and "100% loss" is indistinguishable from "the target is down". It reports a warning rather than a failure, since a resolver pinned deliberately via compose dns: is legitimate.

Loopback resolvers are ignored. 127.0.0.11 is Docker's own embedded DNS, present on every container attached to a user-defined network, and it forwards to whatever the daemon currently resolves with — so it is fresh by construction. The first real run of this check flagged all six healthy containers before that exclusion existed, which is how a check gets ignored.

config-manager-database

The config-manager uses the database it is configured with. From v2.13.0 to v2.13.6 it did not. SQLAlchemy 2.1 changed the driver a bare postgresql:// URL selects to psycopg (v3), which the image does not carry. Every connection failed with No module named 'psycopg', and the API fell back to YAML mode without a word. The database's targets stopped being what SmokePing measured: on the reference Pi, five of its own targets went unmeasured for a day and a half. The API answered, the Targets file was written, and graphs kept coming.

Inside the config-manager container, with the app's own code and DATABASE_URL, the check opens a connection with a 5 s timeout:

• fail when the URL is set but the connection fails, naming the error type;
• ok when it connects, or when no database is configured;
• warn when the probe itself did not answer;
• skip without Docker or without the container.

targets-measured

Every configured target has SmokePing data, and recent data. A clean install once came up with every container healthy and every other check green while SmokePing measured the image's sample Targets: 22 of 22 configured targets had no RRD file at all (#303). InfluxDB held points and the dashboards were new, so nothing here looked wrong.

The check asks config-manager's /measurements, the same answer as the web admin's Measurements card: the configured targets against the RRD files SmokePing writes. It runs inside the config-manager container, so its API token never reaches the command line.

Basic has no config-manager. There the check reads the same files from the SmokePing container (Targets, Probes and the RRD files' times) and classifies them with config-manager's own rules, so both editions give the same answer. The only difference is that Basic times a target's first step from SmokePing's start, because Basic changes targets by restarting SmokePing. Standard and Pro take the same path when config-manager is stopped, and then a target added since SmokePing started may read as missing rather than pending.

• fail when no configured target has data (the hint is sudo smoking-pi restart smokeping), naming the targets that never got data, or naming those that stopped getting it;
• ok when every target is measured. A target within two steps of its own start, or of SmokePing's, is pending there, not missing, so a stack that has just started does not fail; the summary counts those;
• warn when config-manager does not answer, cannot tell (no generated Targets file yet), or lists no target at all: "all 0 targets measured" would be the silence this check exists to end. On Basic, the same applies when SmokePing's files cannot be read;
• fail too when SmokePing's container exists but is not running;
• skip without Docker, or without any SmokePing container.

targets-labeled

How many active targets resolve to unknown in the service taxonomy: no rule and no override says what service they are. Unknown is allowed (a target you added may name a service the vocabulary has no rule for), so this only warns.

• warn listing them (the first 12), each a smoking-pi taxonomy set away from a label;
• ok when every active target has one;
• skip without Docker, without config-manager (Basic), or when its image predates the taxonomy.

silent-series

Every series that answered nothing for a whole day: at least 200 rounds measured (a target added this morning is new, not silent), and every one lost every ping. Covers ICMP, DNS, HTTP and TCP, from InfluxDB. A host that does not serve a probe's layer charts a flat 100% loss forever: an outage that never ends, on a dashboard and to the assistant. The Standard and Pro seed shipped one, bare amazon.com, which drops ping, and the DNS wizard had adopted 72 such layers on the reference Pi before anything said so.

Only targets SmokePing still measures count: the check reads the Targets file the SmokePing container loads, plus the files it @includes (the CPE gateway's CPE_Targets), and drops a series whose target is in none of them. A target deactivated an hour ago keeps a day of dead points in the window, and naming it would advise the command that just removed it. If the file cannot be read, every silent series is reported, as before.

• warn, naming configured targets and the DNS wizard's layers apart. For the wizard's, sudo smoking-pi dns adopt deactivates them (history kept). For a configured target, check that the host answers that probe;
• warn when InfluxDB holds no latency point from the last day at all: the exporter is not writing. Nothing is silent then because nothing was written, which this check used to report as "every series answered". The web admin's Measurements card reads SmokePing's RRD files, so it still says Measuring;
• ok when every series answered;
• skip without Docker or without InfluxDB (ClickHouse mode).

uplink-interface

Names the interface every measurement actually crosses, and says what kind it is:

[ok  ] uplink-interface              measuring over wlan0 (wireless, IPv4)

Nothing said this before, and the omission cost a year: the reference Pi has eth0 with no carrier and its default route on wlan0, so every latency figure recorded since the stack went up had crossed a Wi-Fi hop nobody was measuring. Three states are worth a line rather than silence:

• wireless — the Wi-Fi collector applies, its dashboards have data, and the verdict can say "it's your Wi-Fi, not the ISP" (Wi-Fi uplink stats).
• wired — there are no Wi-Fi statistics, deliberately. Without this line an empty Wi-Fi dashboard is indistinguishable from a broken collector.
• virtual — a warning. The default route is on a Docker bridge, a VPN tunnel, Tailscale or WireGuard, so the latency figures describe that path, and the Wi-Fi verdict is off (it requires the wireless interface to carry the default route) with nothing anywhere saying why. This is the failure the check is for; the other two are context.

With Ethernet and Wi-Fi both connected the line also names the standby, still ok, because both up is normal:

[ok  ] uplink-interface              measuring over eth0 (wired, IPv4); wlan0 also has a default route, at a higher metric, and takes over if eth0 goes down

The day the first one drops, every series moves to the standby. The dashboards mark that moment ("Uplink changed", from host_uplink), and Choosing the interface says how to pick which one carries the measurements.

It reads /proc/net/route, falling back to /proc/net/ipv6_route on a v6-only host, and compares metrics rather than trusting the file's order: with Ethernet and Wi-Fi both up there are two default routes and only the lowest metric carries traffic. The kernel does emit them metric-ascending — verified by adding the high-metric route first in a throwaway namespace and reading the file back — but nothing documents that, and naming the wrong one would report the wrong uplink on exactly the host this check exists for. In /proc/net/ipv6_route, ::/0 also appears twice on lo as an unreachable route, so the flags decide, not the destination — and RTF_UP alone, not RTF_UP | RTF_GATEWAY. A default route installed on-link, with no gateway — which is what wg-quick writes, and what a PPP peer route looks like — is a real default route: observed on a real kernel, ip -6 route add default dev wg0 produces flags 0x00000001. Requiring a gateway dropped it and turned "your uplink is a tunnel", the most useful thing this check can say, into "no default route on this host". The rows that must be excluded do not set RTF_UP at all (lo's are 0x00200200), so nothing is lost. IPv4 has the same shape: ip route add unreachable default is listed in /proc/net/route with the interface name literally * and RTF_REJECT set, and reporting * as the interface being measured would be worse than reporting nothing.

avahi-host-name

Whether the host's Avahi still answers for <hostname>.local. On a name conflict Avahi renames itself to <hostname>-2.local and stays there until it restarts, so the plain name resolves nowhere; the reference Pi did this eight seconds after boot, twice. The check reads Avahi's process title (avahi-daemon: running [smokingpi-2.local]), so it needs no journal, and no root unless /proc is mounted with hidepid (then it skips and says why). Only that renaming pattern warns; a name set on purpose in avahi-daemon.conf (host-name=, domain-name=) is reported, not judged:

[warn] avahi-host-name               1 problem
          - Avahi answers for smokingpi-2.local, not smokingpi.local: it renamed itself after a name conflict, so smokingpi.local resolves nowhere. `sudo systemctl restart avahi-daemon` takes it back until the next conflict; smoking-pi.local (the mdns service) still works (docs/mdns.md)

It skips when Avahi is not running or cannot be seen. smoking-pi.local, the mdns service's name, is not this check's business: that service says which name it holds itself (smoking-pi url).

disk-free

How much of the filesystem under Docker's data root (docker info, DockerRootDir) is free. Warns under 10 % or under 5 GB, fails under 2 % or 1 GB, and the message names the mount and the free space. When images of the project's registry carry tags no container runs, it says so and names the command that removes them:

[warn] disk-free                      1 problem
          - / (Docker's data root) has 4.1 GB free of 117 GB (3%): a full card stops PostgreSQL, Grafana and InfluxDB, and InfluxDB keeps refusing writes until its container is restarted; images of 24 other release(s) are on disk (tags 2.13.8, 2.14.0, 2.15.0, 2.15.1, 2.15.2, …): `smoking-pi upgrade` removes them

On 2026-10-16 the reference Pi's upgrade to v2.38.2 did exactly that: every release's images since 2.13.8 were still on the card (513 images, 92 GB of 117), the pull filled it, PostgreSQL crash-looped on postmaster.pid, Grafana's SQLite reported the disk full and InfluxDB refused every WAL write until its container was restarted, sixteen minutes of measurements lost. smoking-pi upgrade now removes the images of other releases once the new ones run (Upgrades); this check is for the card that fills anyway. It skips without Docker, without a daemon, or where the data root's filesystem cannot be read.

targets-in-tsdb

Every target SmokePing measures has recent points in InfluxDB. targets-measured reads SmokePing's RRD files and silent-series reads InfluxDB, so a target whose RRDs are fresh but whose points never reach the TSDB (the exporter skipping a file, a write refused) passed both while every dashboard, alert and assistant answer saw nothing for it. This takes the targets fresh in the RRDs (the same /measurements classification as targets-measured) and fails when one has no point in InfluxDB, or a newest point older than three of its own steps plus ten minutes for the exporter's cycle (the window it reads is that, and at least 30 minutes):

[FAIL] targets-in-tsdb                 1 problem
          - 1 target(s) SmokePing measures have no point in InfluxDB in the last 30 min, so no dashboard, alert or answer sees them: Netflix_1

Within that time of SmokePing or InfluxDB starting (smoking-pi upgrade runs the doctor right after recreating them) a finding is a warning, since the exporter may not have caught up. A target added a minute ago can show as absent until the exporter's next cycle; run the doctor again. It skips without InfluxDB (Basic, Standard, and Pro on ClickHouse) and when no target has fresh RRDs (that is targets-measured's finding).

monitor-radio

Whether the radio carrying the default route heard throughout the last day. A Wi-Fi radio can hang with the link up and associated: it hears nothing, every probe times out, and the charts draw a network outage that is this host's own. The reference Pi did it twice in October 2026 (1 h 44 min and 5 h 31 min), and five of six confirmed microcuts were its radio. Deaf is the definition the alerter and the assistant use, common/microcuts.uplink_flux: the received-packet counter did not move between two samples, or the radio lost its association, while its interface carried the default route. Runs of consecutive deaf samples (the sampling interval is measured from the day's samples) of at least one SmokePing step are reported; a lone quiet sample is not a hang.

[warn] monitor-radio                   1 problem
          - wlan0 heard nothing for 331 min in the last day (1 span(s), longest 331 min from 2026-10-12 15:15 UTC): the measurements in those minutes describe this host, not the network

It fails when the newest sample is inside a deaf run and the default route has not moved to another interface (the radio is deaf now; a radio that lost its association counts, though its samples then say it no longer carries the route), warns on deaf runs in the last day (naming smoking-pi watchdog enable when the watchdog is off) and when the Wi-Fi collector's newest sample is over five minutes old, and skips without InfluxDB or when Wi-Fi is not the uplink.

dashboard-queries-run

Opens every provisioned dashboard the way a browser would, through Grafana's API on the host (127.0.0.1:3000, the admin user and password from the env file, read-only): each template variable is resolved by running its query (the saved value when the variable still offers it, as Grafana does on open, otherwise the first one), and every panel and annotation query is then run through /api/ds/query with the dashboard's default time range, against the datasource the panel names. Sequential, ten seconds per query and two minutes in all, with the elapsed time in the summary line:

[warn] dashboard-queries-run  11 provisioned dashboards, 254 queries in 39 s; 2 empty panel(s)
          - DNS Wizard: 2 panel(s) returned no rows in now-7d to now (often by design: no IPv6, no change in range, a target without that layer): 'Top-10 churn', 'Top services by presence'

It fails on a query that errors (a Flux compile or runtime error, an SQL error, with Grafana's message), a variable whose query errors or offers no value at all (every panel using it opens empty), a /d/<uid> link to a dashboard this Grafana does not have, and a dashboard of the repository's tree (the InfluxDB one, or the ClickHouse one when TSDB_TYPE=clickhouse) that Grafana does not provision. It warns on a query with no answer in ten seconds, on running out of the two-minute budget, and with the list of panels that returned no rows: many are empty by design, so they are listed, not failed; panels inside a collapsed row and panels that set their own "no value" text are left out of the list. Dashboards a person created are not run, and the summary counts them. One value is checked per variable (the one the dashboard opens with), so a query that fails only for another value, or only under All, is not caught. A variable that cannot be resolved is one finding, and the panels that use it are not run. If Grafana stops answering mid-run, the check warns with how far it got.

On 2026-10-15 a hand-run audit doing the same found the DNS Wizard's Top services by presence failing on production with pivot: specified row key column does not exist in table: cdn, and the CPE dashboard opening blank because its variable picked an address no longer measured, with every other doctor check green: a file cannot show a runtime error. The check needs the env file, so run it as sudo smoking-pi doctor --live; it skips without Docker, where no Grafana runs (Basic and Standard), when Grafana does not answer, and without an admin password.

What each live check covers, by edition and backend

Check	Basic	Standard	Pro, InfluxDB	Pro, ClickHouse
deployed-code-current	yes	yes	yes	yes
container-dns-fresh	yes	yes	yes	yes
config-manager-database	skips (no config-manager)	yes	yes	yes
targets-measured	yes (SmokePing's own files)	yes	yes	yes
targets-labeled	skips (no config-manager)	yes	yes	yes
silent-series	skips (no TSDB)	skips (no TSDB)	yes	skips
targets-in-tsdb	skips (no TSDB)	skips (no TSDB)	yes	skips
monitor-radio	skips (no TSDB)	skips (no TSDB)	yes, on a Wi-Fi uplink	skips
uplink-interface	yes	yes	yes	yes
avahi-host-name	where Avahi runs	where Avahi runs	where Avahi runs	where Avahi runs
disk-free	yes	yes	yes	yes
dashboard-queries-run	skips (no Grafana)	skips (no Grafana)	yes	yes, the ClickHouse tree

The three checks that read the TSDB read InfluxDB only: on ClickHouse they skip and say so, rather than pass. The ClickHouse column of dashboard-queries-run is covered by its tests, not yet by a ClickHouse host.

The Docker checks skip cleanly when Docker is absent, so --live is safe to run anywhere; uplink-interface asks the kernel rather than Docker and answers on any Linux host, skipping only where /proc/net is not there, and avahi-host-name skips where Avahi does not run. Without the flag the behavior is exactly as before, and CI is unaffected.

What is not covered yet

The remaining live checks are not built. They are the other half of the plan:

• Active database rows against the generated Targets: the generated file against the RRDs (targets-measured) and the RRDs against InfluxDB (targets-in-tsdb) are compared; whether every active row reached the generated file is not, beyond config-manager-database proving the database is the source.
• Tag values nothing writes: the static checks verify InfluxDB measurement and tag names against the exporter source (and ClickHouse category / measurement_type values); whether category == "topsites" ever actually occurs in InfluxDB can only be answered by the TSDB.
• Measurement written but never displayed: the reverse direction — a new metric someone added and forgot to chart.
• Unit/semantics drift: loss outside its declared range per measurement (ratio 0–1 for latency/dns_latency, percent 0–100 for cpe_latency), which is how both loss-denominator bugs would have announced themselves.

The TSDB checks above run their Flux inside the InfluxDB container (docker exec, the token never on the host's command line); the remaining ones can take the same path.

Exposing the doctor as an MCP tool, so it can be asked for from chat, comes with that work.