An area of my company hired a new director who directed his “DevOps” team to implement a process that would prevent the CI server from running tests on a PR unless the code was reviewed first. He was worried that there would be too many tests executing with 400 developers committing code frequently.

He’s from Yahoo.

That’s not that many.

Too many tests? Is there such a thing?

If you have more than zero tests, there is a possibility you have too many.

@monkeyboy how is that

@alexbrooklyn Well, yes and no. Sure more tests are nice since you can be sure that what's tested actually works but at the same time you need the hardware to execute them in a feasible amount of time and you need the man power to review the results since you can't just ring the alarm bells just because a test decided to behave flaky. That way you can actually get to a point where your organization can't handle the information coming in anymore and completely releasable releases get delayed.

That being said, if you make sure your infrastructure grows with your testsuite more tests are usually better.

@Godisalie negative. You delete flaky tests and make them not flaky anymore, you don’t review them to see if they should have passed. In this case, it was too many tests being run period, not just a single suite. Basically, someone so ignorant of proper software development that they think humans can inspect more efficiently than machines.

@alexbrooklyn Resources at any company are limited. The resources needed for any single test may best be used elsewhere. Also, any single test may be more expensive than not having that test. Lots of other similar considerations.

@monkeyboy I work for a fortune 5 company. I assure you that area makes up a small fraction of our compute footprint. In addition, the products in that area are generally small and loosely coupled. I know. I helped design them originally. :)

@yamatoman I don't think I've made my point clear. How about this... Every test has a cost. Not every test is worth that cost. At your specific company for your specific products, all your tests may very well be worth the cost. There are many factors that contribute to both the cost and benefit, and reasonable people may differ on their opinions all the time.

@monkeyboy I think any reasonable person would accept that delaying PR testing until after code review is a waste of the most expensive resource, developer time.
No reasonable person would do that.
Now, if you’re one of the ilk who think that testing is optional that’s a different conversation. Professional software developers test their code before submitting a PR.

Sigh

This became even funnier.