Originally published at The New Stack

In May, an old hot topic in software design long thought to be settled was stirred up again, sparked by an article from the Amazon Prime Video architecture team about moving from serverless microservices to a monolith. This sparked some spirited takes and also a clamor for using the right architectural pattern for the right job.

Two interesting aspects can be observed in the melee of ensuing conversations. First, the original article was more about the scaling challenges with serverless “lambdas” rather than purely about microservices. Additionally, it covered state changes within Step Functions leading to higher costs, data transfer between lambdas and S3 storage, and so on.

It bears reminding that there are other and possibly better ways of implementing microservices other than just the use of serverless. The choice of serverless lambdas is not...

Ironically, the move from an engineer role to a manager role is both aspirational and disconcerting. It is also one that usually comes with very meager preparation or training. Over the years, I’ve had the opportunity to stumble, learn, be guided and guide others through this process. Here, I draw from these experiences and hope that this is a helpful start for anyone making this journey.

First, the challenging shifts in mental model:

1. Getting Things Done

The move from being responsible for actually doing things to getting things done is a huge mental shift. Suddenly, what you do individually is a lot less important. You are now accountable for the failures of your team, but more importantly you are responsible to bring them success and work satisfaction.

Contrary to popular FUD, this doesn’t mean you need to be less technical. In fact, many a time its quite the reverse. As an...

We bring a child into this world for our own needs and desires;
the child obviously has no say in it,
even if it is out of a desire to love and be loved.

Having borne a child, we nurture the child
out of our own pleasure and responsibility;
the child is obviously helpless in early childhood.

As the child gains conscious thought, we wish to be “good” parents;
but only because the child is “mine”,
and we define “good” in terms of recognition by our societal norms.

We impose upon elder children what they should be doing;
to become “comfortably” wealthy, to gain recognition in the world,
and to become achievers.

We often justify actions as “for their own good”,
but seldom take into account their own needs and feelings.
We fear that they will be inadequate and our own fears of inadequacy and insecurity spur us.

When children don’t go “our” way, we agonize;
we try to impose our “wish”...

These are testing times. Prolonged isolation and self-quarantine. However with sufficient food supply and a lot of time on hand, the mind wavers towards existential questions. One such is the “essentiality” of work.

Earlier today my wife was recollecting one of her old favourite bangle store which was a 2-storied store selling all kinds of accessories for women. She remarked that in the context of what is happening today, it feels so unnecessary now that such a lot of time, money and resources are typically spent (in normal times) on such frivolous things. I differed with her. My point was that such embellishments and art (when not exaggerated to the point of vanity) add colour to life, which otherwise would become as bland as a black-and-white movie. In a similar vein we both acknowledged that her own craft of classical Indian dance has a huge role to play in making human life much...

Originally published at The New Stack

As Kubernetes turns five, we explore the changing face of DevOps in the K8s world.

DevOps Inception and Shortcomings

In the pre-Kubernetes era, infrastructure and app development were inescapably intertwined. As complexities grew and teams evolved, we saw “DevOps” emerge as a bridge between development and operations in an attempt to resolve the age-old delivery trouble arising from developers throwing things over the wall to ops and then ops having to deal with production issues on the other side. DevOps rose to be a new subculture within existing teams, sometimes yielding new “DevOps teams” and even leading to a new class of tools and methodologies.

In reality, though, bridging skills and evolving an operational culture was not enough. Issues arise in the correct functioning of an application without proper configuration management in place...

Originally published at The New Stack

The last two decades have seen a sea change in the way software is written and delivered. Waterfall to iterative to agile, native to hybrid to responsive interfaces, monoliths to microservices, installed to pay-as-you-go SaaS, data centers to private and hybrid clouds, and virtual machines to containers. As the market constantly evolves, enterprises are facing a ton of choices with increasing complexity.

When microservices came along to break up huge monoliths into manageable parts, it meant teams still had to manage those individual parts, each with their own lifecycle. Containers (you know where that metaphor came from) solved some of that complexity by packaging each part and all dependent information into self-contained, repeatable units. This is where our story begins.

With the advent of containers as an economical portable solution...

Container technology is old. Older than it seems.

In 2008, one of the central concepts behind containers called OS-level virtualization (known as control groups or cgroups in the Linux world) became a mainstream feature of the Linux kernel. OS-level virtualization is not to be confused with whole-system virtualization provided by the likes of VMWare ESXi or Hyper-V. The concept was originally conceived of by FreeBSD developers and made available in the year 2000 as a mechanism called ‘FreeBSD jail’ in the FreeBSD OS. Building on this concept, Google engineers heavily contributed to cgroups in Linux as a way to isolate groups of processes from each other and allocate system resources to these process groups; all without the need of virtual machines or hypervisors. In simplified terms, one group of processes gets its own namespace, limited memory, CPU, network & disk - all of which are...

Fear mongering about machinery and automation has been prevalent since a couple of centuries now! There is a story about the introduction of spinning machines in 18th century England when it was feared that this would end livelihood of thousands of spinning-wheel turners and weavers. There was widespread protest and it had to be put down by force. About 25 years later after the introduction of the machines, a parliamentary committee found that the machines had actually increased jobs in the spinning industry by over 4000%. There are similar stories about fears during the introduction of rail-roads and also when computers made their way to office desks. You would be familiar with some of them. It has been found by numerous studies that machinery & automation has almost always (so far) had a very positive effect on the economy and on people’s lives. Henry Hazlitt’s excellent book Economics...

Two important concepts lie at the heart of understanding the various storage options supported by the Docker engine:

1. The Copy-on-write (CoW) technique
2. Stackable filesystem layers

Copy-on-write (CoW) is a simple pointer-based technique for efficient storage when you have one or more copies of the same data. Lets say you need to make a copy of directory X into directory Y. Here is what a simple copy operation would do:

Employing a copy-on-write technique, the same operation would do something like this:

So this time, instead of copying all the bytes over to directory Y (a time-consuming process and also disk consuming), we only get a bunch of pointers in directory Y - one pointer for every file. So there is just one copy of the file contents on disk and two pointers to it - one from directory X and one from directory Y. This way the copy completes very fast and occupies...