I have had the pleasure of working on a variety projects the past few years. Recent projects have involved creating React-based user interfaces. In the beginning this was exciting. Very exciting indeed. React! Typescript! Components! MobX! Live reloading! Unit tests (Jest)!
Yes, there was some convincing the other developers of the benefits of unit tests. But once I did proof of concept work to show that writing unit tests really isn’t that hard, it was (begrudgingly) accepted. After Jest had proved its worth during refactors it was welcomed as a valued (though sometimes annoying) friend. This is what I like about developers though. Battles may be hard fought, but once they “see the light” and something is proven useful, it is accepted.
Working with security professionals is similarly fraught with conflict. After all, good security practices are frequently at odds with the way developers are used to working.
“I can’t commit this password in the git repo? It’s a hidden file!”
“I can’t run this container as root!?”
“Why do we have to encrypt anything? It’s an embedded device!”
I shudder at each one of those statements too. So yes, I understand you security analyst. I sympathize with the need for your processes security architect. This is why, whenever dealing with security mandates from a client’s IT department, I keep in mind that there is a reason policies are in place.
I ran into a situation though, where the speed of IT reacting to the business’ needs was so glacial, that it was a genuinely frustrating experience. And it was all in the name of security. Let’s say you are a company and have a product (see above). That product has many pieces, only one of which is the user interface (UI). The pieces are all made by different groups within the company. The groups all have their own requirements. None of these requirements includes getting the software packaged together. Tiny, but important details like packaging up different product into a cohesive unit gets lost for a variety of reasons. Primarily, no one wants to be responsible. This is an enterprise company.
I was working within a group which was a part of such a company. As a part of the team working on the UI I felt uniquely positioned to solve, or at least attempt to solve, the packaging problem. I got along with the other groups, was able to work alongside them and understand the problem and thought, “Why not take a shot at providing a solution?” This solution involved docker, or rather, the solution was docker.
My experience is that being in an enterprise company and “getting things done” means doing them a certain way. I started by working my way up the approval chain. Make a proposal, get it approved, do a proof-of-concept. If that seems viable, refine. Run into any issues with “external factors”? You open a ticket. The ticket doesn’t go anywhere and you escalate.
I made a proposal to do the work as a proof of concept, it was approved. The proof was finished in a sprint and found to be viable. The software could be packaged into docker containers. Plot twist! This was docker on windows. Enterprise docker.
I had created windows-based docker images and now I needed to distribute these. And that’s when I ran into my first enterprise IT security problem. I was using images based on windows servercore 2019 which ran just fine on my updated computer. Other computers, as in every single other person, had older versions which required older 2016 servercore images.
And so a one sprint task turns into an epic. Three sprints later, windows 2016 servercore images arrive (along with other UI work).
I’ll skip over the fact that IT not staying up to date on windows releases can be a security issue. There is a balancing act of security vs utility after-all, but do keep that in mind.
But now I try having other people use the images and there’s a problem. You see the 2019 servercore images are much smaller than the 2016 servercore images. So my 5.5 Gb docker image has now become an 11.5 Gb docker image. Worse, something on the enterprise network is preventing these large images from downloading properly. They stop during a pull of the initial windows 2016 base image, retry a few times, then die.
I know what you’re thinking, “It’s the proxy.” I thought so too. So I went home, did the pull and it worked. Then I deleted the image logged into the company VPN at home, did the pull again and it failed. Then I did a wireshark capture while this was happening. Then I went back to work, deleted the image again and confirmed my suspicion (again capturing packets in wireshark).
So now I think I have a pretty good idea of what is going on and, armed with all this information I open a ticket and add all this info (after talking with the very helpful network guy). The ticket is closed the next day. It was a firewall ticket. Traffic gets through the firewall, not their problem. Sigh.
I created another ticket with IT (non-firewall). It’s closed. “Docker is not a supported application.” Keep in mind the business wants to containerize. It’s already using linux containers, it’s just that those containers are significantly smaller in size. After trying to get the ticket to the right bucket a few times I escalate. Nothing changes. Then I write up a summary email detailing my findings and lay out why our network is not suitable for developing with docker on windows. 3 months have passed from when the project began and I’ve moved on. IT has essentially killed innovative use of a technology that has benefits to the company.
The story doesn’t end there however. I am nothing if not persistent. I mention my travails during a meeting another 3 months later and someone else in IT looks closely at the issue. The ticket is re-opened. We find the issue is the proxy (shocker). The solution is simple enough, use a different proxy. However, it took another 3 months to find this solution. 3 more months and many prods to an email chain to keep asking about the progress of the investigation.
Now, I’m not mad at anyone involved in the investigation. Higher priority things can come up all the time in large organizations. Nevertheless, it’s taken 9 months to do something. that should have been finished in 1 month. Welcome to enterprise.
But wait, theres more. A project came up requiring multiple instances of the packaged software to be tested. How many? 50. This could have been done on virtual machines (VMs) at the cost of ~$300 per VM per month. The (IT) VMs we would be mandated to use could not be cloned. Yes, they all had to be setup individually. Let’s be generous and say that this would all have been done in a month. That is ~$15,000 dollars.
Instead I used windows containers and three existing machines to run multiple containers. Instead of spending weeks setting up 50 VMs, (or more realistically, writing code to automate those deployments) I instead wrote a script to launch the containers. It took an hour. I typed very slowly while doing other things.
Not only did this solution save time and money for that specific task, it is a gift that will keep on giving. There is the need for this setup for additional more months; let’s estimate two. If we were using the VMs we’d just keep them up for that time. The windows containers are stopped when not needed and the machines resources available for other tasks. Let’s say that brings the cost savings up to $45,000 over 3 months.
Here’s the kicker though. Given the “loss” of $45,000, IT is still not incentivized to do better. The processes in place are presumably designed to prevent data loss and secure communications on a company wide scale. $45,000 vs the loss of company IP is likely an acceptable tradeoff. But what other development has been lost, truly lost, because people (developers or otherwise) just gave up because of IT instituted roadblocks. How many other $45,000 savings could have been realized across the entire enterprise organization if IT was just a little more diligent. If instead of closing tickets, finding solutions was prioritized.
If you believe the kind of tenacity outlined above would be beneficial to your team, whether in an enterprise (or aspiring) company or not, get in touch. I’d love to help you transform some of your software processes with some innovative problem solving. And save you some money along the way.