High Availability and Disaster Recovery

Concept

High availability (HA) handles the failures you expect: a server dies, a disk fails, a zone loses power. The service continues, usually without anyone noticing.

Disaster recovery (DR) handles the failures you hope never happen: an entire region is unreachable, data is corrupted, or someone deletes the production database. The service is restored, and there is usually a visible interruption.

They are not alternatives. HA without DR loses everything to one bad deletion. DR without HA means every routine hardware failure becomes an incident.

RTO and RPO

Two numbers drive every DR decision. Learn them precisely - they are asked in almost every cloud interview.

          last backup            disaster            service restored
              |                      |                       |
  ------------+----------------------+-----------------------+------->  time
              |<------- RPO ------->|<-------- RTO -------->|

RPO  Recovery Point Objective - how much DATA you can afford to lose,
     measured backwards from the disaster. Set by backup frequency.

RTO  Recovery Time Objective  - how long you can afford to be DOWN,
     measured forwards from the disaster. Set by how ready the standby is.

The four disaster recovery strategies

Choose the cheapest strategy that meets the agreed RTO and RPO. Buying active-active for a system that could tolerate four hours of downtime is a common and expensive mistake.

Important terminology

Real world example

A payments team agrees RPO of 5 minutes and RTO of 30 minutes.

• Backup and restore is rejected: restoring the database takes ninety minutes on its own.
• Pilot light is chosen: the database replicates continuously to a second region, the application definitions exist there, and no application servers run.
• On failover, automation starts the application tier from a pre built image, promotes the replica, and updates DNS.
• Rehearsals show 22 minutes end to end, and replication lag stays under 20 seconds.

The plan is credible because it was measured, not estimated. The first rehearsal took 74 minutes and revealed a missing security group rule and an expired certificate in the second region.

Commands

# Take a logical backup of a MySQL or MariaDB database
mysqldump -u YOUR_DB_USER -p --single-transaction --routines --events YOUR_DB_NAME > backup-YOUR_DB_NAME.sql

# Restore into a scratch database - always restore somewhere safe first
mysql -u YOUR_DB_USER -p YOUR_SCRATCH_DB < backup-YOUR_DB_NAME.sql

# Verify the restore rather than trusting it
mysql -u YOUR_DB_USER -p -e "SELECT COUNT(*) FROM YOUR_TABLE;" YOUR_SCRATCH_DB

# Copy a directory to another host, resumable and verifiable
rsync -avh --partial --progress /var/www/ YOUR_USER@YOUR_HOST:/var/www/

# Check replica status and lag on MySQL or MariaDB
mysql -u YOUR_DB_USER -p --vertical -e "SHOW REPLICA STATUS" | grep -i -E "Seconds_Behind|Running"

Command options worth knowing

Hands on lab

1. Take a backup of any non production database using the mysqldump command above.
2. Create an empty scratch database and restore into it. Never restore over the source.
3. Count rows in two or three tables in both databases and compare.
4. Time the whole restore with a stopwatch. That measured number is your RTO for this component.
5. Delete one row from the source, wait, then check whether your last backup still contains it. The age of that backup is your RPO.
6. Write both numbers down and compare them to what you would have guessed.

Expected output

$ time mysql -u YOUR_DB_USER -p YOUR_SCRATCH_DB < backup.sql
real    3m48.112s

$ mysql -u YOUR_DB_USER -p -e "SELECT COUNT(*) FROM orders;" YOUR_DB_NAME
+----------+
| COUNT(*) |
+----------+
|   184203 |
+----------+
$ mysql -u YOUR_DB_USER -p -e "SELECT COUNT(*) FROM orders;" YOUR_SCRATCH_DB
+----------+
| COUNT(*) |
+----------+
|   184203 |
+----------+

Measured RTO for this component: about 4 minutes, restore only.
Add DNS change, application start and verification for the real figure.

Common mistakes

• Backups that were never restored. An untested backup is a hope, not a plan. The failure is usually discovered during the disaster.
• Confusing HA with DR. A synchronous replica faithfully replicates a DELETE in milliseconds. Redundancy does not protect against mistakes or malice.
• Backups stored beside the thing they protect. Same account, same region, same credentials means the same disaster takes both.
• Ignoring failback. Teams rehearse the switch away and discover the return path is undefined.
• Quoting RPO from the schedule, not from the lag. A five minute schedule with twenty minutes of lag is a twenty minute RPO.
• No plan for corrupted data. If corruption replicates, you need point in time recovery, not another copy.

Troubleshooting

Security considerations

• Backups contain everything the database contains. Encrypt them at rest and in transit, and restrict who can read them as tightly as the database itself.
• Keep at least one copy immutable and in a separate account, so a compromised administrator cannot delete both the data and its backups.
• Restore rights are powerful: restoring an old backup can reinstate deleted users or old permissions. Treat restore as a privileged operation and log it.
• Never place real credentials in a runbook. Reference a secret store using placeholders such as YOUR_DB_USER.

Best practices

• Agree RTO and RPO with the business in writing before choosing a strategy.
• Follow the 3-2-1 idea: three copies, two kinds of storage, one somewhere else.
• Automate the restore, then run it on a schedule and alert if it fails.
• Rehearse a full failover at least twice a year, and write down the measured time.
• Document failback with the same care as failover.
• Keep the runbook where it is readable when the primary system is down.

Interview questions

1. Define RTO and RPO, and give an example of a system with a low RPO but a high RTO.
2. Name the four DR strategies in order of cost and state a workload suited to each.
3. Why does replication not protect against accidental deletion?
4. What is your real RPO if backups run every 5 minutes but replication lag is 20 minutes?
5. How would you prove a disaster recovery plan works?

Mini assignment

Pick any application. Write a one page DR plan stating the agreed RTO and RPO, the chosen strategy with a justification, where backups live and how they are protected, the numbered failover steps, how you verify success, and how you fail back. Finish with the date of the next rehearsal.

Conclusion

HA keeps you running through ordinary failure; DR brings you back from extraordinary failure. RTO and RPO turn a vague wish for safety into a design you can cost, build and - most importantly - test.