As part of my development flow on Airflow I use virtualenvwrapper and a long time ago I set up the ability to automatically set environment variables whenever the virtualenv is activated, and to unset them when leaving. (I think this trick would...
I’ve done a lot of work with Apache Airflow over the last two years, but my experience of how people use it is limited to the teams I’ve worked on and a few glances through talks I’ve seen. I have plenty of ideas of things I’d …
Last week I went along to “Breakfast Ops”, a semi-monthly event aimed at CTOs and other technical decision makers, run by The Scale Factory, and held under Chatham House Rule which says When a meeting, or part thereof, is held under the Chatham...
A very common task when launching an instance in AWS is wanting to configure the instance in some way - maybe we want to install a few packages, download and launch the latest release of our project, or perhaps register with Chef or Puppet and...
Terraform is my preferred tool for automating cloud infrastructure deployments. It’s not without it’s problems but in balance I still prefer it over any other tool I’ve tried. When re-useable modules were added fairly early on in the development...