Anyone here at Hack Manchester this weekend?

Spotted a new feature that had just recently being completed: "Disable all Logging due to noise". What.

Us on the Ops team just died a bit inside...

At the beginning of April there were 9 of us in Ops, come May there will only be 3 of us left. :/

They've left because they disagreed with some the CTO was trying to do and he fired them, or just because they hate him. Every week it's a new game of "Who's going this time?".

We recently started using a Redis cache for our application, and our boss has now taken to blaming Redis for every single incident we have - all because he doesn't understand how it works, and doesn't want to >.>

Setting unit tests that fail to ignore.

Dev created a new service for our application that opened SQL connections. The catch? They never closed said connections. How this passed QA/review I've no idea, but after a few minutes on Staging the service would crash with "connection pool overflow".

I raised a urgent defect, and the devs "fix"? To up the maxConnectionPool to Int.Max!! Argh!

Middle of Black Friday, in Change Freeze, and I'm doing a Production deployment. Yay...!

Boss came to me earlier

Boss: There's a critical issue with this release version for this project. Make sure it doesn't get deployed to our test/live environments!

Me: ...

Me: ...

Me: Err, that release went live 3 months ago...

Huh, just created a job in Jenkins to run msbuild over some solutions. The job accepted a parameter called "Configuration" which was just some simple JSON.

Everytime I ran the job, msbuild failed.

Then I realised that the Debug/Release info for a .NET Solution gets put into an environment variable called "Configuration" that msbuild creates and relies on! >.<

Boss just told me that he thought him making changing directly on production was "OK", because it always seemed to automatically update the other environments when his changes appeared "magically" a week later.

Yeah, because the frequent deployments I do and then come across your un-approved changes which I'm constantly flagging up has nothing to do with it!!

*facepalm*

5am now and trying to do a production deployment. It just failed because boss loves making un-approved changes to prod that change entire database schemas. Argh!!

So one of the PMs arranged a meeting last week for today, where he was going to "talk to us about a project we're currently working on".

Today the PM was off, so myself and a few other managers attended said meeting. Once everyone was in, one of the managers looked around, then at me and stated: "Have you not prepared your handover for this meeting?". I was just sat there like "WTF? What handover!?"

Apparently the PM decided to raise a meeting saying he was going to talk about the project, but then told all the other managers I was going to give a handover. He told everyone, but me!!

No wonder he didn't show up for it -.-

So we had a new platform guy start this morning.

*2 hours later*

They fired the only previous other platform guy. Well this'll end well...

It was in May and I had a recruiter call be up about an interview for a dev position. I went to the interview, thought it went all right and awaited feedback.

Nothing came... I called the agency a few days later and he said he'd get me something back by end of the week.

Still nothing. I called again and he was all "oh sorry, I forgot, and I'll get it tomorrow".

You can probably guess; nothing. A couple weeks pass so by now I'm pretty confident I don't have the job, so I continue looking.

Then early afternoon on a Friday in November that agency calls back:

Agency: "Hi, how are you?"

Me: "Hey, I'm fine"

Agency: "Excellent, remember that interview in May for that company?"

Me: "Yeah, why have they go another position available?"

Agency: "Better, you start work with them on Monday"

It took the guy 6 months to get back, nothing from him or the company. Then he calls up out of the blue. No idea what he would be done had I already got a job.

I actually did accept, still work there now 4 years later, for now.

PM: "so I need you to deploy this new application to some new server. The deadline is in 2 days"

Me: "yeah I can do that, is the application ready and has been tested? Have the servers been set up properly by the IT guy?"

PM: "yep, all is set up and good"

Couple of hours later I try locating the server, only to find it didn't exist.

Me: "the server you mentioned earlier, is doesn't appear to exist?"

PM: "it definitely does, IT guy said he set it all up"

I dig around a little more, but this server definitely doesn't exist. The IT guy was on holiday for a week, so we had to wait for him to get back; delaying the release. On the morning the IT guy got back,

PM: " I though you said you set up that server for the application, we've had to delay it now!"

IT: "I just set it up this morning. Like I said in the email to you before I Ieft, I will have to do it first thing when I get back after holiday"

Turns out the PM had asked the IT guy to spin up the server, but never bothered to read his response. Assuming it was done he told the client he'd have it deployed in a couple of days.

The application was deployed successfully later that day, but not before the PM blamed us two for its delay.

Doing the deployment to production, and towards the end one of the support guys looks over at me;

"So, the website in prod is throwing some errors"

Followed by another guy:

"Yeah I'm getting the same, SQL exceptions on the page"

I stare at them panicked for a moment, when one of them goes "just kidding!". Like dude, my heart just skipped several beats!

Any one here ever had something cruel happen to them during a deployment?

When your CTO has full access to the production servers and makes SQL changes directly without following process. Then breaks production and tells everyone "this is why we need to follow process!". Then proceeds to break it again hours later.

When developers fix broken unit/integration tests by setting them to ignore...