A sliding window is a rate limit kept on the client: it moves forward as responses come back, which limits the number of requests in flight.
The size of the window is a trade-off. A larger window causes more congestion, so the last request waits a long time. A smaller window causes more round trips, because the requests beyond it are rejected and tried again later.
The size is therefore not fixed:
- The window grows. A system that handles 3000 requests per second is given a little more, and while requests succeed the window keeps increasing.
- The window shrinks. When requests start failing, the window size is reduced.
With 200 requests, the 200th waits a long time. With a window of five, the sixth request is bounced back.
A static limit of 200 does not adapt. When the hardware improves, the window sets itself with no code change.
A very large window does not help either. A user who waits three seconds hits refresh, and memcached gets two requests on 95 keys.