Join devRant
Do all the things like
++ or -- rants, post your own rants, comment on others' rants and build your customized dev avatar
Sign Up
Pipeless API
From the creators of devRant, Pipeless lets you power real-time personalized recommendations and activity feeds using a simple API
Learn More
FUUCCKKKK!! I need to hit smth. Or rant..
So that flaky ec2 issue.. These ec2s act as a shared environment for multiple apps. Our app is one of them. I have no access to those ec2s at all.
What I have access to is my app and some monitoring. Now the app randomly starts lagging while nearly idling. At the same random times monitoring stops completely and doesn't come back up. This happens to random app instances at random times.
Reached out to infra support, managed to get attention from the big boys [mgmt]. Today we got the fix deployed. I test it out -- problem persists.
I find this behaviour somewhat familiar. Managed to get some server stats from infra folks. Apparently cpu% is high as well as load avg [cpu queue]. Bingo! I know how to fix it!
So I write a long comment w/ all the commands and all the 'if that, do this'. Send it to one of the infra technitians
and I get a reply: 'we will apply cpu usage limitations to fix the issue'
wait... Cpu% limitations will do nothing but highlight the underlying problem...
'no, instances have high cpu utilisation which is causing those lags. We will limit cpu resources and it will be fixed'
oh ffs... Cpu utilization and cpu queue are VERY different things.. I tried explaining that to them like 7-9 times. And all I get is:
'yes, cpu utilization is the problem. We will limit it and solve the problem'
I would surely escalate all of this through higher channels if only I could get my hands on those ec2s and have a proof. But that is not happening and I'm forced to sit back and watch them break things even worse until they are out of options and mark my query as 'wont fix'....
Fuck that's frustrating....
*thinking to myself* so I've read about that new vulnerability 2 days ago that allows one to escape from docker container to the host... What if <...>
rant