ServiceTalk

Motivation

ServiceTalk is intended to provide a common and extensible networking abstraction on top of a lower-level networking framework (e.g. Netty). Netty is a great low-level networking framework, but when used for service to service communication it presents a few opportunities for improvement:

  • Threading Model

    • Fully asynchronous and requires knowledge of EventLoop threading model

    • Executing CPU intensive or "blocking" code requires manual thread hops

    • Subtle out of order execution of tasks when code executes both on and off the EventLoop thread

  • Usability

    • APIs are not tailored towards common application use cases (e.g. request/response, RPC, etc..)

    • Asynchronous programming paradigm presents a barrier to entry in scenarios where vertical scalability is not a primary concern.

    • Error propagation follows multiple paths depending on the event and state of Channel

    • Back-pressure requires manual association between source and sink of data

  • Lacking Feature Set

ServiceTalk addresses these challenges and includes support for multiple Programming Paradigms. It accomplishes this by building on a fully asynchronous non-blocking I/O core and taking care of the threading model complexities internally.

Design Philosophy

ServiceTalk has many different goals driven from the improvement opportunities provided by Netty. Despite ServiceTalk’s broad feature set it follows some key design principles ensuring consistency and reduced complexity across module boundaries. These design philosophies are elaborated in this section to provide context for users and developers of ServiceTalk.

Extensible core

ServiceTalk is designed to provide an extensible core and APIs tailored to network protocols. It does not intend to provide abstractions for low-level networking primitives (e.g. Channels, EventLoop, TLS, etc…​) but instead uses these primitives to provide a higher level API in multiple Programming Paradigms.

Low barrier to entry across richer feature set

One of the primary objectives of ServiceTalk is to reduce the barrier to entry relative to lower level networking frameworks (e.g. Netty). This must be achieved all while adding a richer feature set, or else the value proposition for ServiceTalk is less compelling. The combination of low bar to entry and enhanced feature set is the primary challenge for ServiceTalk developers and benefit for end users. The richer feature set can be observed in areas such as our more accessible APIs that are more tailored toward common request/response protocols (eg: HTTP/1.x, HTTP/2, gRPC), extensibility at the protocol level through client/server filters, our asynchronous sources that account for cross cutting concerns (eg: retry/cancellation/back-pressure), and "smart client" features that enables decentralized microservices deployments to name a few.

A key component of low barrier to entry is providing users with the programming model that is right for their needs. Historically it is common for frameworks to support a single programming model (eg: asynchronous or synchronous) and/or optionally have support for an alternative paradigm at a higher level with a reduced feature set (eg: JAX-RS provides multiple programming paradigms without fine grained control over the networking layer). However it is not uncommon for applications to have a mixed profile. For example some requests require asynchronous and streaming due to performance constraints (eg: large file transfers) while others are adequately handled by traditional blocking APIs from a scalability perspective and benefit from more familiar APIs (eg: low volume health check). A consequence of not supporting both paradigms with a seamless transition is that users have to decide which layer best suits their need; low level feature rich layer or a high level flexible layer with less features. Typically these decisions have to be made early in the development cycle and changing this choice usually requires a large engineering work to re-design the application, also impacting the parts that do not require this new model.

ServiceTalk embraces the fact that applications tend to be multi-faceted; they have a mixed requirement of high vertical scalability (eg: large number of long-running connections for push based streaming APIs) for some areas and ease of development (low concurrency endpoints) in other areas. ServiceTalk makes writing such applications easier by natively supporting multiple programming paradigms and an ability to switch between these paradigms in the same application. We also support writing blocking code safe by default even with asynchronous programming paradigms. This design philosophy takes the burden off our users such that they can start simple and use advanced features as when they feel comfortable. Thus, making ServiceTalk a library that grows with the user needs.

Cross-protocol API symmetry

ServiceTalk provides different application level protocols to users, eg: HTTP/1.1, HTTP/2.0, gRPC. A design goal is to provide consistent APIs and features across these different protocols when possible to maximize reuse and minimize cognitive overhead making it easier to adopt ServiceTalk across multiple protocols. However, this ease of use does not come for free, it necessitates that ServiceTalk provides generic concepts that abstract protocol specifics. This trade-off introduces some additional abstractions that complicate core ServiceTalk development (e.g. load balancer) for the benefit of reuse and consistency. This trade-off is consistent with the design principles of ServiceTalk to value simplicity and ease of use for end users.

Modularization

The project is divided into many modules to decouple the user-facing API from implementation details. This gives users freedom to choose only the functionality they need, and also allows us to evolve each module independently.

Why ServiceTalk?

Over the past few years we have gained experience through Netty and ServiceTalk development. Deploying into production has also provided some invaluable user feedback and lessons which has influenced our roadmap along the way. In the meanwhile, the industry also identified similar opportunities adding new frameworks on top of Netty. Despite the space being more crowded than when we first started, we believe that ServiceTalk provides unique value in this space.

ServiceTalk provides an extensible core, is safe by default with a low barrier to entry across a richer feature set that evolves with your application , and does this in a consistent manner across commonly used request/response protocols for client/server. This combination of design principals provides a value that is unique within the industry and has demonstrated successful real world deployments across different domains.

Our design philosophies are somewhat unique but we acknowledge overlap with other libraries/frameworks in this space. While providing different features in ServiceTalk, we often face questions around "build vs use" as there are other options available that solve similar concerns. When possible we use an off-the-shelf solution (eg: jersey) but in some cases the trade-offs justify building our own(eg: asynchronous primitives).