Effective Spring Boot Caching Techniques for Horizontal Scaling
Written on
Scaling server applications is vital for ensuring availability and handling traffic surges. However, this process is not as simple as just adding more instances.
This article serves as the first in a series focused on horizontal scaling, addressing potential obstacles that may affect your backend application's scalability and examining efficient caching methods within the Spring Boot framework.
Starting Point
Let’s assume you already have a Spring Boot application running as a single instance and utilizing PostgreSQL as its database. As demands evolve, there is a need for improved resilience and the capacity to manage increased traffic. To meet these challenges, we are considering expanding the application to multiple instances using a platform like Kubernetes.
Before we move forward, we must ask ourselves:
Is Your Application Prepared for Horizontal Scaling?
The answer is NO if an application instance keeps a local state that isn't synchronized with other instances.
To illustrate this, consider a scenario involving two colleagues, Stephen and Randy, who are waiting for an important meeting with their manager. Randy receives a text from their boss canceling the meeting and heads to the pub, while Stephen remains at his desk, unaware of the change.
In this analogy, Randy and Stephen represent application instances. Initially, both shared the same knowledge (the meeting was scheduled), but as soon as Randy received the message, his understanding changed, while Stephen's did not.
Such inconsistencies can manifest in various ways within your application. For instance, if a user uploads an image to one instance and receives an error when trying to access it from another.
When the states of instances diverge, the behavior of your application can become unpredictable, leading to functionality disruptions and challenging bugs in a multi-node environment.
How Do We Tackle This Challenge?
Understanding and managing local state is crucial for scaling server applications. Here are two strategies to consider:
- Eliminate local state at the application level and transition it to a shared external service.
- Implement a state synchronization mechanism.
To better understand the advantages and disadvantages of each option, we should analyze them in the context of specific application components.
Caching
This article will specifically focus on caching, a common local state scenario.
A cache serves as temporary storage for frequently accessed data, enhancing performance by minimizing the need to repeatedly retrieve the same information from the original source, such as a database or external API.
Single-instance applications typically utilize local caches for efficiency, but this approach cannot be directly applied in a multi-instance setup due to potential inconsistencies.
In Spring Boot, caching is generally managed through the CacheManager and Hibernate’s second-level (L2) cache abstractions. We will primarily discuss the CacheManager, as the foundational principles for both approaches are similar.
Approach 1: External Cache
To begin, let's eliminate any local cache providers and replace them with a shared external solution.
By storing data in a single location, all application instances can access the same information regardless of which instance processes a request. This method mimics the behavior of a single-instance deployment and resolves inconsistencies stemming from local caches.
Our preferred choice for a shared external cache is Redis, a reliable in-memory storage solution that is suitable for applications of all sizes. However, other options are also available.
To include Redis as a dependency, add the following:
implementation("org.springframework.boot:spring-boot-starter-data-redis")
Spring Boot will automatically set up the Redis implementation of CacheManager with default settings. To customize the configuration, you will need two additional beans:
@Bean
fun redisCacheConfiguration(): RedisCacheConfiguration =
RedisCacheConfiguration.defaultCacheConfig()
.entryTtl(Duration.ofMinutes(60))@Bean
fun redisCacheManagerBuilderCustomizer(): RedisCacheManagerBuilderCustomizer =
RedisCacheManagerBuilderCustomizer { builder ->
builder
.withCacheConfiguration(
"venues",
RedisCacheConfiguration.defaultCacheConfig()
.entryTtl(Duration.ofMinutes(30)))
.withCacheConfiguration(
"customers",
RedisCacheConfiguration.defaultCacheConfig()
.entryTtl(Duration.ofMinutes(5)))
}
Next, configure your Redis deployment and connection properties in the application.yml file.
Approach 2: Synchronized Local Caches
Is it possible to maintain a fast local cache while enabling horizontal scaling?
To keep caches synchronized across instances, we need a messaging system to communicate changes to all running instances. This can be achieved through a message bus, allowing us to broadcast notifications to all connected nodes.
Various message bus systems are available. Spring Boot integrates well with AMQP brokers, Redis, Pulsar, Kafka, and other cloud messaging solutions. Each system has unique characteristics and optimal use cases, but we will not delve deeply into them here.
Since our application utilizes PostgreSQL, we can leverage its built-in notification system as a basic message bus. The examples below utilize my library built on PostgreSQL's LISTEN/NOTIFY feature, providing real-time messaging without complicating the infrastructure.
Let’s develop a custom Spring Cache interface implementation that delegates operations to the local cache and notifies other instances when cache entries need removal. This is accomplished by broadcasting CacheNotification messages of two types:
Evict: Remove a cached value for a specific key.
Clear: Remove all cached values.
class SimpleDistributedCache(
private val underlying: Cache,
private val messagingTemplate: PostgresMessagingTemplate,
) : Cache by underlying {
companion object {
const val CACHE_CHANNEL = "cache"
}
override fun evict(key: Any) =
underlying.evict(key).also {
messagingTemplate.convertAndSend(
CACHE_CHANNEL,
CacheNotification.Evict(
cacheName = name,
key = key
)
)
}
override fun clear() =
underlying.clear().also {
messagingTemplate.convertAndSend(
CACHE_CHANNEL,
CacheNotification.Clear(cacheName = name)
)
}
}
We also need to implement our CacheManager:
class SimpleDistributedCacheManager(
private val underlying: CacheManager,
private val messagingTemplate: PostgresMessagingTemplate
) : CacheManager by underlying {
private val logger = KotlinLogging.logger {}
override fun getCache(name: String): Cache? =
underlying.getCache(name)?.let { cache ->
SimpleDistributedCache(cache, messagingTemplate)}
@PostgresMessageListener(value = [CACHE_CHANNEL], skipLocal = true)
fun handleNotification(notification: CacheNotification) {
try {
underlying.getCache(notification.cacheName)?.let { cache ->
when (notification) {
is CacheNotification.Clear -> cache.clear()
is CacheNotification.Evict -> cache.evict(notification.key)
}
}
} catch (e: Exception) {
logger.error(e) { "Error during processing distributed cache notification: $notification" }}
}
}
The critical aspect of the code above is the handleNotification method. This method listens for updates from other instances and applies them to the underlying caches. The @PostgresMessageListener annotation designates the method as a message listener for the channel, while the skipLocal = true attribute ensures that messages from the same instance are disregarded.
If you opt for a different messaging system, ensure you use the appropriate annotations, such as @RabbitListener or @SqsListener for RabbitMQ and AWS SQS, respectively.
It's essential to maintain the cache key data type when sending eviction messages; otherwise, instances may fail to clear their caches. For instance, JSON, a commonly used message serialization format, does not differentiate between Int and Long data types. To prevent issues, use strings for cache keys or customize message serialization.
Finally, let’s instantiate our custom CacheManager using Caffeine as the underlying local cache:
implementation("com.github.ben-manes.caffeine:caffeine:3.1.8")
@Bean
fun cacheManager(messagingTemplate: PostgresMessagingTemplate): CacheManager =
SimpleDistributedCacheManager(
messagingTemplate,
CaffeineCacheManager().apply {
setCaffeine(Caffeine.newBuilder().expireAfterWrite(10, TimeUnit.MINUTES))}
)
In conclusion, we've established a straightforward cache synchronization mechanism that combines the benefits of local caching and horizontal scaling.
Limitations to Consider
- There will always be a delay before all instances receive a synchronization message, which means clients may temporarily see outdated data.
- The memory allocated for local caches is constrained to the instance/container running your application.
- If you are using a messaging system that guarantees 'at-most-once' delivery (like PostgreSQL or Redis pub/sub), be aware that messages could be lost due to network issues, resulting in some instances not evicting cache as expected.
Summary: What Strategy Should You Choose?
A shared external cache is the default choice for horizontally scaled server applications. It ensures strong consistency of cached data across instances, is highly scalable, and reduces the load on the primary database more efficiently than local caches. Modern in-memory storage solutions like Redis or Hazelcast offer more than just caching; they can significantly enhance your system's capabilities.
However, if your application experiences significantly more reads than writes and you can tolerate occasional data inconsistencies, consider using synchronized local caches. Local caches generally outperform shared caches due to the absence of network communication, resulting in lower latency for users.
Tips
- Consider a multi-level caching strategy that integrates both approaches to improve cache performance and reduce database load.
- Review the usage of ConcurrentHashMap in your application, as it is often employed as a form of local cache.
- In certain scenarios, load balancers with sticky sessions can enhance local cache efficiency by minimizing cache misses and resolving temporary inconsistencies.
Conclusion
To effectively scale horizontally, it is essential to discern which application data can remain local and which must be shared.
This article has provided general strategies for managing instance local state and implementing an efficient caching layer that enables you to scale your Spring Boot application effectively.
Future articles will delve into managing WebSockets and scheduled tasks in multi-instance backend applications.
Explore the complete code for the sample Spring Boot application designed for scaling on GitHub:
GitHub - aleh-zhloba/spring-boot-horizontal-scaling-example: Example of ready-for-scaling-out Spring Boot application
Happy coding! Cheers!