The way in which a simple fix to one problem can have unexpected effects in other areas is always interesting. Anyone who has done any coding will know that changing one little thing can affect the larger project in ways that aren’t always apparent at the outset. Recently one of our clients experienced a recurring problem in their logstash SQS queue: every couple of days the IAM role that was being used to generate temporary credentials for logstash SQS would become unable to connect to the queue, throwing errors and requiring frequent maintenance. Our answer was to simply switch from temporary to static credentials which has entirely cleared up the issue.
The benefit that we weren’t expecting, although in retrospect it makes perfect sense, is that the client’s CPU usage was significantly reduced. No longer required to make frequent calls back to the IAM role for temporary credentials the client system’s resources were freed up for other tasks and resulted in throughput in the logstash queue. One relatively simple change not only cleared up a recurring problem but increased the client’s message handling from just under 600,000 per ten minutes to just over 700,000 over the same interval.
Never underestimate the power of a minor improvement in a broader system.