Build the dashboard before the incident. Per domain: what to measure, what "trouble" looks like, how to read the panel at a glance, and how to wire it up in Dynatrace (the tool we use). A good dashboard answers one question — "is this healthy?" — in three seconds.
Start here
The rules that make every domain dashboard below readable — including by someone seeing it for the first time.
Measure the signals, not everything
Golden signals (services): Latency, Traffic, Errors, Saturation.
RED (request-driven): Rate, Errors, Duration.
USE (resources): Utilization, Saturation, Errors.
Pick 4–8 tiles per screen. If everything is on the dashboard, nothing is.
Make it readable in 3 seconds
Most important tile top-left; green/amber/red so status reads before numbers do.
Show rate of change, not just absolutes — "climbing" matters more than "high".
Annotate deploys on the timeline. Half of all incidents line up with one.
One dashboard = one question. "Is the AG healthy?" not "everything about SQL."
Dynatrace — how to reach it
OneAgent auto-discovers hosts, processes, and services — start from the entity, not a blank chart.
Davis AI does baselining + anomaly detection; let it flag the deviation instead of hand-setting every threshold.
Build custom dashboards from metric tiles; query with DQL (Grail) for logs/events.
Use management zones to scope a team/brand, and SLOs for the "are we within budget" tile.
Databases & middleware come via Hub extensions (deployed on an ActiveGate) — add the SQL Server / PostgreSQL / Oracle extension to get their metrics.
SQL Server Always On + engine health. Version/edition changes what's available — confirm first.
Measure
AG health: synchronization_state, log send queue, redo queue, failover readiness
Redo queue growing → DR falling behind; RTO at risk
sync_health ≠ HEALTHY, or a replica NOT SYNCHRONIZING
PLE dropping fast → memory pressure
High signal waits → CPU pressure; long PAGEIOLATCH → storage
Blocking chains, rising deadlocks
log_reuse_wait stuck, backups aging out
What the dashboard is telling you
Top row = AG health (send/redo queue + sync state) — your DR truth. Second row = load (batch req/sec, CPU, PLE). A queue climbing while throughput is flat means the secondary can't keep up, not that you're busy.
Dynatrace tips
Deploy the Microsoft SQL Server extension (Hub → ActiveGate) for DMV-based metrics; OneAgent gives host + the SQL process.
Chart AG send/redo queue and sync state as status tiles; alert on redo queue trend, not a fixed number.
Use database service views for top statements; correlate slow app services to their SQL calls via Service flow.
PostgreSQL / RDS replication + engine health. On RDS many of these arrive via CloudWatch.
Lag + slot state is the DR story. A flat lag with a rising retained-WAL slot means a replica/consumer stopped reading — fix that before the disk fills. Cache-hit dropping with rising read IO = working set outgrew RAM.
Dynatrace tips
Deploy the PostgreSQL extension (Hub → ActiveGate) for pg_stat_* metrics; RDS/Aurora via the AWS integration (CloudWatch).
Alert on lag trend + inactive slots; let Davis baseline TPS/latency.
Oracle Data Guard + RAC. Don't assume the release or OS — 12c/19c/21c and Linux/Windows differ.
Measure
Data Guard: apply lag, transport lag (v$dataguard_stats), MRP state, archive gap
Transport: v$archive_dest_status, FRA % used
DB time / top waits: log file sync, db file sequential read
Resources: CPU, SGA/PGA, redo generation rate
RAC: gc (global cache) waits, interconnect latency, instance status
Space: tablespace + ASM diskgroup usage
Look for
Apply/transport lag growing → standby behind; may break FSFO
Archive gap, or archiver stuck (FRA full)
High log file sync → commit latency (SYNC transport / storage)
gc buffer busy / cr block busy → interconnect or hot blocks
ASM diskgroup filling; tablespace near full
What the dashboard is telling you
Lag + gap + FRA% is the DG health triangle. Lag rising with a full FRA on the standby usually means apply stalled on space, not network. High log file sync on the primary with SYNC transport points at the standby/network round-trip.
Dynatrace tips
Deploy the Oracle Database extension (Hub → ActiveGate); OneAgent covers the host + listener process.
Data Guard lag isn't standard — feed v$dataguard_stats as custom metrics and chart apply/transport lag as tiles.
Tiles: lag, gap, FRA %, top waits, ASM %; alert on lag + FRA trend.
Application services — the RED signals plus saturation. This is where Dynatrace is strongest out of the box.
Measure
Rate: requests/sec per service and endpoint
Errors: error rate, HTTP 5xx, exception count
Duration: p50 / p95 / p99 latency
Saturation: CPU, memory, GC pauses, threads, connection pool
Dependencies: latency + error rate to DB, cache, queue, 3rd-party
Apdex / SLO burn
Look for
Error rate spike aligned to a deploy
p99 climbing while p50 is flat → tail latency / GC / a slow dependency
Connection pool exhausted → "timeouts" that aren't the network
One dependency slow → cascading timeouts + retry storm
SLO burn rate accelerating
What the dashboard is telling you
RED per service, with a deploy overlay, answers "what changed and where does the time go." p99 up + p50 flat = a subset of requests hurting (tail), often GC or a hot dependency — not everything.
Dynatrace tips
OneAgent auto-instruments services — you get RED signals with no config.
Use Service flow + PurePath traces to see where a request spends time; Davis auto-surfaces the root cause.
Set SLOs on key services and put the burn-rate tile on the dashboard; overlay deploy events.
Dashboards & Metrics Field Kit — a ReliabilityOps tool. Tips are tool-agnostic where possible and Dynatrace-specific where noted; confirm entity/metric names against your Dynatrace version and extensions. Offline, single-file; nothing on this page calls the network.