we had this guy once, who we gave access to our private repo. everything's all good until we noticed that our amazon bill was USD 8,000+!!! we found out that lots of servers got created and that's bec. this guy forked our private repo and his fork was a public one. our keys were still not in .env files and were part of the commit so some bot got hold of it and accessed our amazon account. we suspected that the servers were used for bitcoin mining. anyway guy was fired on the spot and we also learned our lesson to keep keys out of repos.

This horrifies me so much! I always have to double check I've removed anything private from the config files when creating/commiting

we didn't pay the whole 8k USD but we still had to pay a fraction of it to Amazon bec. of some troubleshooting fee or something (I forgot already as it was years ago)

after this we really restructured our whole project so this won't happen again.

This might just be me, but how was it explicitly that guys fault if he was given access to a repo that was dumb enough to keep its key stored?

@corjaantje well it was a private repo and he also knew why we kept it private in the first place.and he wasn't just any third party guy given access, he was one of the company devs (a new one basically). we were just surprised why our repo went public. it might be an honest mistake but from management point of view we couldn't afford to keep the guy anymore. also the system was a legacy one. this event triggered us to rebuild a better version (old system was built by the previous guys who were outsourced btw).

@daryledesilva, it's moments like these that force is to learn how to rewrite Git history... Lol.

Still, that's a painful lesson to learn hey. Sorry bud..

@ZaLiTHkA not good enough. The key may already be compromised at that point. Only proper thing to do is change the key. Had the same thing happened to me when i posted some config file to stack overflow. Didn't notice that the key was in the file ☹️

@eldamir, I wasn't implying that there was nothing more to do.. But when there's something in the history of a git repo that shouldn't be there, rewriting the history is the only way to remove it.

Obviously, if something like this actually happens, the relevant machines would definitely need new private keys generated.

Same thing happened to us one day. New dev on a project decided to take some work home with him by committing it to his public repo.

Same as you, we have some AWS keys in there for the dev asset buckets.

About a day or two later we received an email from Amazon saying that they have detected that the key was in a public repo, and that since it is most likely compromised, sent instructions on how to cycle it.

I thought that was a great effort towards keeping their customers safe.

I don't understand why you would fire someone after a mistake like that. You just paid 8K to teach him an important lesson, you are now certain he will never make that kind d of mistake again.

@da9lzqyq yes our COO felt the same way. 😁

@ZaLiTHkA we thought about this but then again we just decided to switch to a new framework w/c ended up us creating a new fresh repo

@eldamir we did changed all our keys after that! anything in the leaked repo were all definitely compromised 😁

@fattymiller i forgot to mention they also emailed us. we found relief once they said we don't have to pay the whole thing as this scenarios are not that uncommon

@Zilounay My guess is that this poor guy just hit the fork button on github web interface. Knowing how most of github projects work (fork > update > pull request) I'd say that it was an honest mistake or an accidental missclick.

He just had a lot of bad luck - bad security policies at the company (storing keys in git?!), company not using separated location for critical source code storage (nope, private github repos don't count) and finally, working at a small company, where a 8k$ bill caused heart attacks (despite the fact that only a fraction was paid).

Yup, he made a mistake but who doesn't? And as Zilounay said - by keeping him, you'd have an extremly carefull developer who would be extra motivated to atone for his mistake. Mistake that was only possible because of quite a few slipups on the company side.