jonoropeza.com

Making software and software development teams. Mostly the people parts.


Downtime Prevention Technique: Exit if environment variables are not found

I've written about the concept of assertively throwing errors before. This is a specific example:

If the application, service or script you're developing requires an environment variable to be able to do the things it does (fulfill its contract and meet its SLAs, in enterprise-ie terms), then you should prefer a loop at the earliest stage possible in the runtime startup that iterates through the expected environment variables, build an array of any of them that are null / undefined / not found, and if that array is not empty then exits 1 with a console barf of all the required environment variables.

Coupled with a proper deployment pipeline that checks for a live instance before flipping or rotating traffic to the new instance, this pattern completely prevents a whole host of downtimes caused by well it worked on my machine and worked in the staging environment(s), but failed when deployed to prod because we forgot to set the environment variable / didn't have permissions to set or even check if the environment variable was set / fat fingered or copy-pastaed the environment variable.

In my experience, the problems will occur at the first deployment of something like 5-10% of any group's new features that require an environment variable, especially as groups grow and deployments are handled more and more by less experienced engineers.

posted in Downtime Prevention