Computer screen showing data

Small Changes Can Lead to Big Improvements

The way in which a simple fix to one problem can have unexpected effects in other areas is always interesting. Anyone who has done any coding will know that changing one little thing can affect the larger project in ways that aren’t always apparent at the outset. Recently one of our clients experienced a recurring problem in their logstash SQS queue: every couple of days the IAM role that was being used to generate temporary credentials for logstash SQS would become unable to connect to the queue, throwing errors and requiring frequent maintenance. Our answer was to simply switch from temporary to static credentials which has entirely cleared up the issue.

The benefit that we weren’t expecting, although in retrospect it makes perfect sense, is that the client’s CPU usage was significantly reduced. No longer required to make frequent calls back to the IAM role for temporary credentials the client system’s resources were freed up for other tasks and resulted in throughput in the logstash queue. One relatively simple change not only cleared up a recurring problem but increased the client’s message handling from just under 600,000 per ten minutes to just over 700,000 over the same interval.

Never underestimate the power of a minor improvement in a broader system.

Other posts you may be interested in

How to build a Singer tap: infographic

"How do you eat an octopus? One bite at a time…" When I first encountered Singer, I was armed with basic Python skills and experience with REST APIs. I was familiar with data integration and ETL but I had no knowledge of the Singer platform. After learning how to use...

Looker ERD Generator – from an Explore using the Looker API

Have you ever wanted to create an Entity Relationship Diagram (ERD) from your Looker Model Explores? As a Looker partner consultant, I get asked for an ERD or Data Model all of the time. Data Model diagrams provide a concise, visual way to show how the tables (or...

Plug and Play Data Movement in the Cloud

Any developer who has ever been tasked with consolidating data sources before performing analysis knows the challenge of silos, especially when data is sprawled out across multiple third-party services. But what about if you had to help a client pull application data...

Small Changes Can Lead to Big Improvements

The way in which a simple fix to one problem can have unexpected effects in other areas is always interesting. Anyone who has done any coding will know that changing one little thing can affect the larger project in ways that aren’t always apparent at the outset....