AG Fundamentals

how Always On actually works — with the queries to prove it
On this page
  1. The moving parts — what an AG actually is
  2. How data moves — send queue, redo queue
  3. Synchronous vs asynchronous, properly
  4. The listener & DNS — what really happens on connect
  5. Endpoint, replica & seeding permissions
  6. Service account & storage permissions — how to check
  7. Moving database files to a new drive — zero downtime
  8. Pre-exercise validation checklist

1The moving parts — what an AG actually is

An availability group is five separate systems cooperating. When something breaks, knowing WHICH layer failed is most of the diagnosis.

┌─ Windows Server Failover Cluster (WSFC) ──────────────────┐ │ quorum votes · owns the AG "role" · listener resources │ │ │ │ ┌─ NODE A (primary) ─────┐ ┌─ NODE B (DR secondary) ─┐│ │ │ SQL instance │ │ SQL instance ││ │ │ AG: [YourAG] │ │ AG: [YourAG] ││ │ │ ├─ db1 (read/write) │ log │ ├─ db1 (restoring/RO) ││ │ │ └─ db2 │────▶│ └─ db2 ││ │ │ Hadr_endpoint :5022 ◀──┼─────┼──▶ Hadr_endpoint :5022 ││ │ └────────────────────────┘ └─────────────────────────┘│ │ │ │ Listener: AGL-name = Network Name + IP resources │ │ → registered in AD (VCO) and DNS (A records) │ └────────────────────────────────────────────────────────────┘
/* the whole anatomy in four result sets */
SELECT name, is_distributed, automated_backup_preference_desc
FROM sys.availability_groups;                       -- the AG

SELECT ag.name AS ag, ar.replica_server_name, ar.availability_mode_desc,
       ar.failover_mode_desc, ar.seeding_mode_desc, ar.endpoint_url
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id;  -- replicas

SELECT name, state_desc, port = (SELECT port FROM sys.tcp_endpoints t
       WHERE t.endpoint_id = e.endpoint_id)
FROM sys.endpoints e WHERE type_desc = 'DATABASE_MIRRORING';   -- endpoint

SELECT l.dns_name, l.port, ip.ip_address, ip.state_desc
FROM sys.availability_group_listeners l
JOIN sys.availability_group_listener_ip_addresses ip
     ON l.listener_id = ip.listener_id;                        -- listener

2How data moves — send queue, redo queue

Every change is a log record. The pipeline per secondary database:

PRIMARY SECONDARY write txn → harden to local log → SEND QUEUE ──(endpoint :5022)──▶ receive → harden to its log → REDO QUEUE → redo thread applies to data pages
SELECT ar.replica_server_name, DB_NAME(drs.database_id) AS db,
       drs.log_send_queue_size AS send_kb,      -- RPO exposure
       drs.log_send_rate      AS send_kb_s,
       drs.redo_queue_size    AS redo_kb,       -- RTO exposure
       drs.redo_rate          AS redo_kb_s,
       CASE WHEN drs.redo_rate > 0
            THEN drs.redo_queue_size / drs.redo_rate END AS redo_catchup_sec
FROM sys.dm_hadr_database_replica_states drs
JOIN sys.availability_replicas ar ON drs.replica_id = ar.replica_id
WHERE drs.is_local = 0 OR drs.is_primary_replica = 1;

3Synchronous vs asynchronous, properly

Synchronous-commit

The primary does not acknowledge a commit to the application until the secondary confirms the log record is hardened to ITS log (not applied — just written to disk in the log). That handshake is what HADR_SYNC_COMMIT waits measure. The price: every transaction pays the round-trip to the slowest sync secondary. The prize: a SYNCHRONIZED secondary is a zero-data-loss failover target, and only sync replicas can use automatic failover.

Asynchronous-commit

The primary commits locally and ships log on its own schedule. Apps feel zero replication cost, distance doesn't matter — but the send queue is unshipped data that dies with the primary. Async replicas show SYNCHRONIZING forever (never SYNCHRONIZED), allow no automatic failover, and a manual failover to one is always a forced failover with data-loss acceptance.

SynchronousAsynchronous
Commit latency+ round trip to secondary (every txn)none
Data loss on failoverzero (when SYNCHRONIZED)= send queue at that moment
Failover modesautomatic or manual, no data lossforced only (data-loss ack)
Steady state showsSYNCHRONIZEDSYNCHRONIZING (that's normal!)
Typical usesame site / metro HA pairDR site over distance
Wait signatureHADR_SYNC_COMMITnone on primary; queues carry the story
A DR replica showing SYNCHRONIZING is healthy async behavior — don't chase it. The number that matters is the send queue trend. Flat/small = fine; growing = the link or the secondary can't keep up.
/* check current modes */
SELECT ag.name, ar.replica_server_name,
       ar.availability_mode_desc, ar.failover_mode_desc
FROM sys.availability_replicas ar
JOIN sys.availability_groups ag ON ar.group_id = ag.group_id;

/* switch DR replica to sync (before a planned no-loss failover) */
ALTER AVAILABILITY GROUP [YourAG]
  MODIFY REPLICA ON 'DRSERVER'
  WITH (AVAILABILITY_MODE = SYNCHRONOUS_COMMIT);
-- wait until sync state = SYNCHRONIZED, then fail over.

/* set it back to async after (protect app latency) */
ALTER AVAILABILITY GROUP [YourAG]
  MODIFY REPLICA ON 'DRSERVER'
  WITH (AVAILABILITY_MODE = ASYNCHRONOUS_COMMIT);

4The listener & DNS — what really happens on connect

The listener is three things wearing one name: a WSFC Network Name resource, one or more IP address resources (one per subnet), and the AD/DNS objects the cluster creates for them.

When the AG fails over, WSFC moves those resources to the new primary's node, brings the right subnet's IP online, and re-registers DNS. When a client connects:

  1. Client asks DNS for listener-name → gets one or more A records.
  2. Client connects to that IP on the listener port; the node that currently owns the resources answers.
  3. With ApplicationIntent=ReadOnly, the primary can bounce the session to a readable secondary via read-only routing.

The two settings that decide DR behavior

SettingWhat it really doesFailure mode
RegisterAllProvidersIP=1 (default)ALL subnet IPs go into DNS at once; clients are expected to try them in parallel (MultiSubnetFailover=True)Legacy clients without the flag try the dead site's IP first and hang for their full timeout
RegisterAllProvidersIP=0Only the ONLINE IP is registered; DNS is updated on failoverEvery client waits on DNS propagation — HostRecordTTL (default 1200s = 20 min!) plus any DNS server caching
Pick your poison consciously: =1 + fixed connection strings (best) or =0 + HostRecordTTL lowered to ~60–120s (for app estates you can't touch). The worst state is the default with legacy clients — which is exactly what a DR exercise exposes.

Validate the whole chain

-- SQL's view: listener, port, IPs and which is online
SELECT l.dns_name, l.port, ip.ip_address, ip.state_desc,
       ip.network_subnet_ip, ip.network_subnet_prefix_length
FROM sys.availability_group_listeners l
JOIN sys.availability_group_listener_ip_addresses ip
     ON l.listener_id = ip.listener_id;

-- read-only routing (if you use readable secondaries)
SELECT ar.replica_server_name AS when_primary_is,
       ar2.replica_server_name AS route_reads_to,
       rl.routing_priority, ar2.read_only_routing_url
FROM sys.availability_read_only_routing_lists rl
JOIN sys.availability_replicas ar  ON rl.replica_id = ar.replica_id
JOIN sys.availability_replicas ar2 ON rl.read_only_replica_id = ar2.replica_id
ORDER BY when_primary_is, rl.routing_priority;
# PowerShell on a cluster node — the cluster's view
Get-ClusterResource |
  Where-Object ResourceType -in 'Network Name','IP Address' |
  Format-Table Name, State, OwnerGroup, OwnerNode

# the two DNS-behavior settings, per listener Network Name
Get-ClusterResource "YourAG_Listener" |
  Get-ClusterParameter RegisterAllProvidersIP, HostRecordTTL

# force DNS re-registration after fixing either setting
Get-ClusterResource "YourAG_Listener" | Update-ClusterNetworkNameResource

# what clients actually receive — run FROM AN APP SERVER
nslookup your-listener-name
Test-NetConnection your-listener-name -Port 1433

Who creates the AD/DNS objects (and why it fails)

The cluster's own computer account (CNO) creates the listener's computer object (VCO) in AD and its DNS records. If listener creation throws error 19471, the CNO lacks Create Computer Objects on its OU or write access to the DNS zone — that's an AD fix, not a SQL fix. Alternative: pre-stage the VCO and grant the CNO Full Control on it.

5Endpoint, replica & seeding permissions

Almost every "can't join / can't seed / not connecting" is one of three grants missing. Here they all are, plus how to verify each.

Grant 1 — endpoint CONNECT (replica ↔ replica auth)

Each replica connects to the others' endpoints as its SQL service account. On EVERY replica, every OTHER replica's service account needs a login and CONNECT on the endpoint:

/* on each replica — once per other replica's service account */
CREATE LOGIN [DOMAIN\svc-sql-dr] FROM WINDOWS;
GRANT CONNECT ON ENDPOINT::Hadr_endpoint TO [DOMAIN\svc-sql-dr];

/* if the endpoint doesn't exist yet */
CREATE ENDPOINT Hadr_endpoint
  STATE = STARTED
  AS TCP (LISTENER_PORT = 5022, LISTENER_IP = ALL)
  FOR DATABASE_MIRRORING (ROLE = ALL, ENCRYPTION = REQUIRED ALGORITHM AES);

/* VERIFY — who can connect to the endpoint on this replica */
SELECT e.name AS endpoint, sp.name AS grantee,
       p.permission_name, p.state_desc
FROM sys.server_permissions p
JOIN sys.endpoints e ON p.major_id = e.endpoint_id AND p.class = 105
JOIN sys.server_principals sp
     ON p.grantee_principal_id = sp.principal_id;
Instances running as NT SERVICE\MSSQLSERVER or local accounts can't authenticate across machines — cross-server endpoint auth then needs certificate-based endpoints or (better) domain service accounts. If prod uses domain accounts and DR was built with local ones, this is your 1418.

Grant 2 — seeding permission on the new host

Automatic seeding has the AG itself create databases on the secondary. That requires an explicit grant on the secondary, and it's the single most-forgotten step when standing up a new DR replica:

/* on the NEW SECONDARY (the replica being seeded) */
ALTER AVAILABILITY GROUP [YourAG] GRANT CREATE ANY DATABASE;

/* verify + watch it work */
SELECT ag.name, ds.database_name, ds.current_state,
       ds.failure_state_desc, ds.start_time
FROM sys.dm_hadr_automatic_seeding ds
JOIN sys.availability_groups ag ON ds.ag_id = ag.group_id
ORDER BY ds.start_time DESC;

Grant 3 — the cluster's hook into SQL

WSFC drives AG online/offline through the local [NT AUTHORITY\SYSTEM] login. If someone "hardened" it away, the AG can't come online (error 41131):

/* required for AG operations — verify SYSTEM has these */
GRANT ALTER ANY AVAILABILITY GROUP TO [NT AUTHORITY\SYSTEM];
GRANT CONNECT SQL TO [NT AUTHORITY\SYSTEM];
GRANT VIEW SERVER STATE TO [NT AUTHORITY\SYSTEM];

SELECT sp.name, p.permission_name, p.state_desc
FROM sys.server_permissions p
JOIN sys.server_principals sp ON p.grantee_principal_id = sp.principal_id
WHERE sp.name = N'NT AUTHORITY\SYSTEM';

6Service account & storage permissions — how to check

SQL touches disk as its service account, never as you. When seeding or restore fails with access-denied (5120, 15105, 3201, 17204), test the path as SQL sees it — not from your own session.

/* 1 · who IS the service account on this box? */
SELECT servicename, service_account, status_desc
FROM sys.dm_server_services;

/* 2 · where do new files want to go? */
SELECT SERVERPROPERTY('InstanceDefaultDataPath') AS default_data,
       SERVERPROPERTY('InstanceDefaultLogPath')  AS default_log;

/* 3 · probe a path AS THE SERVICE ACCOUNT
   (these run under the engine's identity — the honest test) */
EXEC master.sys.xp_dirtree  'D:\SQLData', 1, 1;
EXEC master.sys.xp_dirtree  '\\dr-share\backups', 1, 1;
EXEC master.sys.xp_fileexist '\\dr-share\backups\seed.bak';
-- empty result / 0s = the SERVICE ACCOUNT can't see it,
-- even if it opens fine in YOUR Explorer window
# PowerShell side — confirm and fix NTFS/share
Get-CimInstance Win32_Service -Filter "Name LIKE 'MSSQL%'" |
  Select-Object Name, StartName, State

icacls D:\SQLData              # who has what on the data dir
icacls D:\SQLData /grant "DOMAIN\svc-sql-dr:(OI)(CI)F"   # grant if missing

# share-level ACL matters too, not just NTFS:
Get-SmbShareAccess -Name backups   # on the file server
Rule of thumb for the exercise: before any restore/seeding step, run the xp_dirtree probe against the exact path in the command. Ten seconds, and it separates "SQL can't see it" from "the backup is bad" before you burn 40 minutes on the wrong theory.

7Moving database files to a new drive — zero app downtime on an AG

The AG itself is the migration tool: move files on each secondary while it's not serving, fail over to a migrated node, then finish the old primary. Apps see one brief failover blip at most — with sync-commit and client retry, effectively nothing. Example: 4 nodes SQL-1-01…04, moving from D:\ to E:\.

Pre-flight — do all four of these before touching anything

/* 1 · document logical names + physical paths — run on EVERY node.
   Paths live per-instance in master; never assume they match. */
SELECT @@SERVERNAME AS node, DB_NAME(database_id) AS db,
       name AS logical_name, type_desc, physical_name,
       size*8/1024 AS size_mb
FROM sys.master_files
WHERE database_id = DB_ID(N'<db>');
# 2 · target path exists on ALL nodes + NTFS for the service account
$nodes = 'SQL-1-01','SQL-1-02','SQL-1-03','SQL-1-04'
$paths = 'E:\SQLData','E:\SQLLog'
foreach ($n in $nodes) {
  Invoke-Command -ComputerName $n -ScriptBlock {
    param($ps)
    foreach ($p in $ps) {
      if (-not (Test-Path $p)) { New-Item -ItemType Directory -Path $p | Out-Null }
      "== $env:COMPUTERNAME $p =="; icacls $p
    }
  } -ArgumentList (,$paths)
}
# grant where missing:
# icacls E:\SQLData /grant "DOMAIN\svc-sql:(OI)(CI)F"
/* 3 · verify AS THE ENGINE, on each node (§6 — the honest test) */
EXEC master.sys.xp_dirtree N'E:\SQLData', 1, 1;
EXEC master.sys.xp_dirtree N'E:\SQLLog', 1, 1;

/* 4 · note failover modes, then set MANUAL for the migration window
   (no surprise auto-failover onto a node mid-move) */
SELECT ar.replica_server_name, ar.availability_mode_desc,
       ar.failover_mode_desc
FROM sys.availability_replicas ar;
ALTER AVAILABILITY GROUP [<ag>]
  MODIFY REPLICA ON 'SQL-1-01' WITH (FAILOVER_MODE = MANUAL);
-- repeat for every replica currently AUTOMATIC

Method A — rolling move through the secondaries (recommended)

Per secondary, one node at a time — say 02, then 03, then 04 while 01 is primary:

/* on the SECONDARY being migrated */
ALTER DATABASE [<db>] SET HADR SUSPEND;

ALTER DATABASE [<db>] MODIFY FILE
  (NAME = N'<logical_data>', FILENAME = N'E:\SQLData\<db>.mdf');
ALTER DATABASE [<db>] MODIFY FILE
  (NAME = N'<logical_log>',  FILENAME = N'E:\SQLLog\<db>_log.ldf');
-- repeat per file from your pre-flight inventory
# still on that node - stop SQL (it's a secondary; apps don't notice)
Stop-Service MSSQLSERVER
robocopy D:\SQLData E:\SQLData <db>.mdf     /copyall /j
robocopy D:\SQLLog  E:\SQLLog  <db>_log.ldf /copyall /j
Start-Service MSSQLSERVER
/* verify the paths took, resume, wait for green */
SELECT name, physical_name FROM sys.master_files
WHERE database_id = DB_ID(N'<db>');
ALTER DATABASE [<db>] SET HADR RESUME;
-- wait for SYNCHRONIZED (AG health query) before the next node

When all secondaries are moved: confirm your failover target is SYNCHRONOUS_COMMIT + SYNCHRONIZED, then planned failover to it (right-click AG › Failover…, or on the target: ALTER AVAILABILITY GROUP [<ag>] FAILOVER;). The old primary is now a secondary — repeat the same move steps on it. Fail back if you care which node is primary.

"Zero downtime" honestly stated: the planned failover is a connection blip of a few seconds. Sync-commit means zero data loss, and clients with MultiSubnetFailover=True + retry logic reconnect invisibly. If even that blip is unacceptable, schedule the failover inside your quietest window — everything else in this method is genuinely transparent.

Cleanup — don't skip

-- after ALL nodes verified on E:\ and SYNCHRONIZED:
--   delete the old D:\ copies (they are stale, not backups)
-- restore failover modes:
ALTER AVAILABILITY GROUP [<ag>]
  MODIFY REPLICA ON 'SQL-1-01' WITH (FAILOVER_MODE = AUTOMATIC);
-- re-run: AG health query, backup job check, post-failover smoke test

Method B — ADD FILE / EMPTYFILE (fully online, data files only)

No service stops, no failover — but it changes the file layout instead of moving files, and it cannot move the log or the primary .mdf. Good when the goal is "get the data onto the new drive," not "same file, new path."

/* file operations replay on EVERY replica - the E:\ path must
   already exist on ALL nodes or that secondary SUSPENDS (why
   pre-flight #2 and #3 matter for this method too) */
ALTER DATABASE [<db>] ADD FILE
  (NAME = N'<db>_data2', FILENAME = N'E:\SQLData\<db>_2.ndf',
   SIZE = <current_used_mb> MB) TO FILEGROUP [PRIMARY];

DBCC SHRINKFILE (N'<old_secondary_file>', EMPTYFILE);  -- IO-heavy, online
ALTER DATABASE [<db>] REMOVE FILE [<old_secondary_file>];
Which method? Log or .mdf must move → Method A. Extra .ndf files, or you can't get service-stop windows on secondaries → Method B. Both stand on the same pre-flight: paths on all four nodes, NTFS for the service account, xp_dirtree proof.

8Pre-exercise validation checklist

#CheckTool
1Endpoints STARTED on all replicas; CONNECT granted to every service account§5 queries
2Seeding mode known per replica; GRANT CREATE ANY DATABASE in place if automatic§5 / Seed tab
3NT AUTHORITY\SYSTEM has the AG grants on every replica§5
4Service account can see data paths and backup shares (xp_dirtree)§6
5Logins/jobs/linked servers/credentials/certs synced to DRTriage page → Detect
6Listener: RegisterAllProvidersIP + HostRecordTTL match your client reality; nslookup from an app server§4
7Send/redo queues small and draining; note baseline numbers§2 query
8Fleet audit clean (or findings logged)Invoke-HadrAudit.ps1

Companion pages: SQL HADR Triage (decision tree, Detect audit, Seed runbook) · Ops Console (fuzzy search, wait board, live scripts). Personal reference — defer to your site's approved runbook where they differ.