Wow! AWS re:Invent is next week – it’s already been a year since the last event!? Doesn’t time fly fast when you’re having fun in the tech world!
For many businesses and engineers, re:Invent is an opportunity to bring greater efficiency to their operations by utilising newly announced features. Each new feature isn’t just a productivity/cost benefit – it comes with layered principles of security enhancements to keep prospective security teams in their zen posture.
2019 has been a great year in the cloud for me but it hasn’t been a perfect one. Given the time of year I thought I’d put a few predictions in that would have made things a lot smoother.
Here are my top 3 predictions/asks/wants/needs of AWS from re:Invent:
AWS announced the public fork of Open Source Elasticsearch on the 1st July of this year. Music to my ears! We are heavy users of Elasticsearch both in AWS and in Elastic Cloud. The only reason we use Elastic Cloud though is for its’ RBAC (Role Based Access Control) of indices (although there are many other great features!).
Using the non-native SaaS sounds like it shouldn’t be a problem, it actually isn’t a major one but as you compare the integration of the native solution to external SaaS solution, it has lots of little complexities. We have to write more glue code, it’s slower to spin the whole stack up, harder to test, duplicate solutions for indices curation, not as easy to integrate with our monitoring and alerting. Each thing on it’s own is no big issue but it would be better by far if it was all native and with the OpenDistro fork announced it must be on the cards for an AWS Elasticsearch service announcement at re:Invent.
Another thing on the wish list is the release of AWS Timestream. Announced last year it still isn’t generally available. A managed time series DB wrapped up in the AWS ecosystem.
Today we use InfluxDB as our time series DB of choice but we run it on EC2. This takes a lot of management to ensure resilience, zero downtime etc. It is such a stark contrast to the RDS Serverless deployments we have that it leaves me itching for the release of Timestream DB (and hopefully an InfluxDB like API).
I recently built an auto scaling Logstash service that ingested, transformed and pushed UDP syslogs to our SIEM. This solution should have been trivial following the announcement of UDP support on the NLB earlier in the year, but it’s never that simple… The AWS NLB uses a flow hash based algorithm which means requests that originate from the same host and port will be pinned to a given destination/target. Effectively not load balanced when a single IP is sending all the requests (as was my case).
After some POC work this issue was circumvented by introducing a proxy layer of Nginx that could forward each syslog packet to the NLB with a randomised source port. Immediately terminating the connection after each dispatch and thus evenly balance traffic.
It was a hard fought battle to come to that solution. It would have been much much easier if the NLB had the ability to customise the load balancing algorithm. This should be trivial but who knows… It’s a long shot but I’m hoping there’s an announcement for some level of customisation that helps me simplify the solution and remove Nginx.
So that’s it. Just 3 things that would have added great value in 2019 to me and my own delivery efficiency.
The great thing about re:Invent is that they often announce things you didn’t even think about. I trust this year will be no different. It all starts next Monday December 2nd!