Traffic Resiliency Features

With this plain configuration, we plugged the Gradient capacity limiter, to manage capacity for the service endpoint we are building. The limiter, will react to RTT time changes (as we explained above), by evaluating all the requests that go through this filter.

We discussed how to set up the Traffic Resilience filter, using default configurations of the Gradient capacity limiter. We also discussed what is the basic contract of the Capacity Limiter API, but we skipped some concepts on purpose because it didn’t make sense to discuss them out of context. One of these concepts is the Classification of a request.

The classification of a request, is a simple API that allows users of ServiceTalk assign a different weight to a request.

What that means is that requests with a specific weight assigned, will have slightly different treatment when they are evaluated for capacity purposes. To understand this better, you can imagine the following example. If a capacity limiter, has a limit set to 10 concurrent requests (fixed or dynamic), then a request with a weight of 10 will only be allocated 10% of that limit, meaning that only 1 concurrent request will be allowed with that weight classification. In contrast, a request with a weight of 90, will be allocated 90% of that limit. This feature, helps with prioritization of requests, according to importance, even though the actual ordering of the requests isn’t considered in this offering.

To use this feature, we need to provide a mechanism to classify incoming requests before they are evaluated by the capacity limiter. In the following example, we classify requests that target to a health endpoint with a weight of 20, while everything else with the max value of 100.

The Capacity Limiters we discussed in the APIs so far offered a universal approach on protecting a server against overload degradation. The solutions we covered in the Capacity Limiters chapter, rely on either time based feedback (see RTTs) or loss based feedback (see Rejections/Cancellations).

Time (or duration) is a fundamental concept in our core offerings, and can have different origins. Treating them all equally may not result in the best possible experience in some occasions. ServiceTalk in the defaults takes some liberties of assuming a fair distribution of RTTs among all flows in a server, and any extremes (e.g. severe tail latencies) ideally need to be prevented (see. Gradient).

There are however use-cases that have quite different RTT characteristics. Imagine a WRITE API (i.e., HTTP POST/PUT) that takes multiple seconds to complete, where on the same service a READ API (i.e., HTTP GET) takes a few millis to complete by relying on caches. These two APIs guarded by a universal Capacity Limiter can result in poor dynamic limits and false positives (rejected requests that don’t present an overload risk for the server).

To support this use case, the Traffic Resilience filter allows for partitioning schemes. Below an example that has a different limiter for different HTTP methods.

Another interesting use-case that these APIs support is a way to manage quotas. By default, the Fixed Limiter can act as a quota controller, by allowing N ammount of concurrent requests for certain customer / API etc. but with a custom implementation a ServiceTalk user could define an alternative solution that applies a quota based on incoming content-length universally or per client.

A way to support this but at the same time be overload protected by a Capacity Limiter, is to use the composite APIs to form a complex Capacity Limiter.

Ordering of the limiters passed to the composite factory matters. We generally want the root limiter to be evaluated first and make sure there is no overload before we evaluate individual quotas.

Let’s now focus on what happens when a request gets rejected through a Traffic Resilience filter. By default rejections from a capacity limiter through the Traffic Resilience filter will result in "canned" responses of:

This behavior can be customized by providing a different ServiceRejectionPolicy

So far, we discussed rejecting requests to meet capacity limits, but when the system is stressed, accepting new connections can make things even worse. A client-side load-balancer for example, could decide that because existing connections are busy, more connections are needed, which can make matters worse pretty fast; consider also that new connections usually entail expensive handshakes (e.g. TLS)

For the purposes of capacity, a control mechanism needs to be applied as early as possible in the application flow. The ideal spot for a Netty based networking application would be inside Netty itself, before every interaction with a socket.

In ServiceTalk we chose to allow this controller to take place a bit later. This offers us access to the HttpMetadata of a request which allows for these additional features we covered above to be supported at a small cost of processing the first part of an incoming request. Along with the features it helps us build, it provides API familiarity being yet another filter.

The Traffic Resilience filter provides a way for capacity rejections to yield the server socket from accepting new connections. That means that when the limiter in use, starts rejecting requests, then the server socket will also not accept new connections, without that affecting existing connections.

Default behavior is set to not use this feature out of the box, due to the way users are allowed to specify limiters per partition.

All examples we have seen so far, rely on defaults to provide feedback to the capacity limiter in use. Feedback is the mechanism an acquired Ticket can use to hint the limiter about various conditions.

Here are some expected behaviors:

As a user, you could also hint the limiter of certain conditions. For example, imagine that you have an internal Executor Service in your service, that starts throwing RejectedExecutionException when it can not accept any more tasks. You know that this is an indication that your Executor is taking longer to complete tasks and the rate of incoming tasks is greater than that. You can use the Traffic Resilience filter to react on this exceptions by hinting to the limiter that the request was dropped.

In the example below, we demonstrate this case, by translating the RejectedExecutionException to a dropped signal, and all other errors, are using the ignored signal to tell the limiter to not take these flows in-to account.

Feedback for limiters is crucial, and in some cases it can pro-actively benefit the limiter by hinting on a situation before the limiter comes to figure it itself adaptively. Cases like these are when the remote (a.k.a., peer) returns indicative of capacity issues responses. If for example, the remote starts responding with Too Many Requests status code then we should hint the limiter about it and let it adapt its limits on our side (the client).

To do so, the client filter offers APIs that allow the ServiceTalk user to define what a "Rejection" looks like for their upstream dependency. In most cases that would be:

and these are our defaults, but you can modify this Predicate according to your upstream behavior.

The above example, will evaluate the remote rejection, and the request will result in a RequestRejectedException. If you want to still evaluate the remote rejection, but allow the original response from the upstream to be returned to the caller then see the example below.

Rejected requests (both locally or remotely) are good candidates for a retry. We typically want to retry a few times to avail for better chances, local rejections (requests that didn’t touch the wire yet), will automatically be retried by the RetryingHttpRequesterFilter. Remote rejection on the other hand require some additional coordination to avail retries.

This topic has good potential for improvements. An optimal behavior would be to adaptively learn from the frequency of rejections and retries, and stop retrying when things are not looking like a retry will be beneficial.