Scalability and Elasticity

Concept

Scalability is whether a system can handle more load by being given more resources.

Elasticity is whether it does so automatically, in both directions, as demand changes.

A system can be scalable without being elastic: you can add servers, but a human has to decide and act. Elasticity is scalability plus automation plus the willingness to shrink again. The shrinking half is where the savings live and where most teams stop.

The two directions of scaling

Architecture

VERTICAL                       HORIZONTAL

  [ 2 CPU ]                     [ app ] [ app ] [ app ]
     |  resize                           |     /
     v                            [ load balancer ]
  [ 8 CPU ]                              |
                                    [ database ]
one machine, bigger            many machines, shared state behind them

Elasticity = a controller watching a metric, adding and removing
             machines on the right hand diagram without a human.

Important terminology

Real world example

A ticket booking site sells seats for a concert at 10:00.

• At 09:00 the fleet is four instances at fifteen per cent CPU.
• At 09:58 traffic begins to climb. A target tracking policy aiming for fifty per cent CPU starts adding instances.
• Each new instance takes about ninety seconds before it passes its health check.
• At 10:00 the spike arrives. Because the policy reacted at 09:58 and instances warm in ninety seconds, capacity is roughly in place.
• By 10:40 demand has fallen. The policy removes instances one at a time with a cooldown between removals.

Note what did the real work: reacting early enough to cover warm up time. A scaling policy that reacts at 10:00 exactly is a policy that arrives late.

Commands

You can observe scaling behaviour without a cloud account by generating load and watching the machine.

# Watch CPU, memory and run queue while load changes
vmstat 2

# Load averages, compared against the number of cores
uptime
nproc

# Generate CPU load on 2 cores for 30 seconds (install with your package manager)
stress-ng --cpu 2 --timeout 30s

# Send 200 requests with 20 concurrent connections to a local service
ab -n 200 -c 20 http://127.0.0.1:8080/

# Per process CPU and memory, sorted, refreshed
top -o %CPU

Command options worth knowing

Hands on lab

1. Run nproc and note the core count. Run uptime and note the load average.
2. In one terminal start vmstat 2.
3. In another, run stress-ng --cpu 2 --timeout 30s on a machine you own.
4. Watch the r column rise and the id column fall.
5. Run uptime again. The one minute load average should approach the number of busy workers.
6. Write down: at what load average would you want a scaling policy to add capacity, given a ninety second warm up?

Expected output

Before load:
 r  b   swpd   free  ... us sy id wa
 0  0      0 4071232      3  1 96  0
$ uptime
 14:02:11 up 6 days,  load average: 0.14, 0.19, 0.22

During stress-ng --cpu 2 on a 4 core machine:
 r  b   swpd   free  ... us sy id wa
 2  0      0 4065108     51  2 47  0
$ uptime
 14:03:04 up 6 days,  load average: 1.71, 0.62, 0.35

Reading: 2 of 4 cores busy, idle roughly halved, load average
climbing towards 2. On this machine, 2.0 is 50 per cent utilised,
not overloaded - load average must always be read against nproc.

Common mistakes

• Confusing the two words. Scalability is capability, elasticity is automatic behaviour. Interviewers ask this specifically.
• Reading load average without the core count. A load of 4.0 is saturated on four cores and comfortable on sixteen.
• Scaling out an application that stores state locally. Sessions or uploaded files on the instance disk break the moment a second instance appears.
• Only scaling up. If the policy never removes capacity, you have bought a bigger fleet, not elasticity.
• Ignoring warm up time. Reacting at the moment of the spike guarantees you are late by exactly the boot time.
• Forgetting the database. Ten application servers pointed at one database that cannot take more connections have moved the bottleneck, not removed it.

Troubleshooting

Security considerations

• Set a maximum instance count. Without one, a traffic flood or a runaway loop becomes an unbounded bill.
• New instances must be built from a trusted, patched image. Scaling multiplies whatever is in that image, including its vulnerabilities.
• Rate limiting is part of scaling. Some spikes should be rejected, not served.

Best practices

• Make the application stateless first. Everything else is easier afterwards.
• Prefer horizontal scaling for the application tier, and reserve vertical scaling for the database and for emergencies.
• Set both a minimum and a maximum, and be deliberate about the minimum during off hours.
• Scale on a metric that reflects user pain - request latency or queue depth - rather than CPU alone.
• Test the scale down path. It breaks more often than the scale up path.

Interview questions

1. Explain the difference between scalability and elasticity in two sentences.
2. When would you scale vertically rather than horizontally?
3. What must be true about an application before it can scale out safely?
4. Your fleet scales out but latency does not improve. What do you check first?
5. Why is load average meaningless without knowing the core count?

Mini assignment

Design a scaling policy on paper for a service that receives a five times traffic spike every weekday at 09:00, where an instance takes two minutes to become healthy. State the metric, the target value, the minimum and maximum instance counts, the cooldown, and how many minutes before 09:00 the policy must react. Justify each number in one line.

Conclusion

Scalability is a property of your design; elasticity is a property of your automation. Make the application stateless, scale on a metric that reflects user experience, and remember that scaling down is the half that pays for the whole exercise.