Too much software, not enough engineers - a word on productivity

So much software, not enough engineers. You would think that’s related to the Great Resignation, or maybe that the pandemic made them move to the countryside and convert into farmers. I mean, the latter might be true in some cases, but for sure it is not the reason why the world is experiencing a shortage of software engineers. In 2019 The Wall Street Journal reported that U.S. companies had about 918,000 unfilled IT jobs in the three months leading to the report. The shortage of software engineers is halting innovation, affecting the growth of businesses, and, unfortunately, the situation did not improve in this regard since 2019. But it is not just a matter of the past few years - the inability for enough software to be produced to satisfy the world’s needs is a pattern that repeats itself every few years. Even in 1968, at the first NATO software engineering conference, attendees were bringing up the term software crisis (Naur & Randell, 1969).

Oh no! How can we solve this?

Simply put, there are 2 main ways to address the gap between software demand and supply - we either try to reduce the demand, or we try to increase the supply (Meyer et al., 2014). The first approach is unlikely to succeed since the world’s appetite for software won’t cease any time soon. The second one though could be viewed a little differently than trying to magically make some more SE appear. How about we focus on improving the productivity of the already existing developers?

Some pieces of research tried to conceptualize developers’ productivity based on the number of code lines per hour, but that clearly isn’t an accurate indicator. So what Meyer et al. (2014) did, was to look at the SEs’ perception of their productivity. A shared view amongst them was that a productive day is one when many or big tasks are completed without significant context interruption.

A context interruption or switch means having to stop working on a task to engage in another task or activity. Here are some of the aspects that can influence the impact of a context switch:

• having to undertake another task that is different in nature and has the potential to be taxing on your workflow, for example, having to write a report
• the more focused you were on the initial task, the more switching away from it will turn out to be costly
• a switch from a creative task requires more focus, so it is more expensive than a switch from a routine task
• self-inflicted context switches, like going on a coffee break with your colleagues is less expensive (and sometimes even beneficial according to Epstein, Avrahami & Biehl, 2016) than externally imposed switches, such as interruptions by coworkers

With all that in mind, we can still not assume that what works for some works for everybody. Yes, most people don’t like to be interrupted when they do meaningful work, but what is perceived as a disruption varies from case to case. For example, most SE that participated in Meyer’s study considered meetings as being a waste of time, however, 57% out of the 379 professionals said that meetings can also be productive and beneficial when they have a clear focus, include decision making, and all attendees are well-prepared. In an attempt to better understand the variations between what is considered a distraction and what is not Meyer, Zimmermann & Fritz (2017) came to the conclusion that developers can roughly be clustered into 6 groups based on their perception of productivity:

1. the social developers - these are the ones who love to help their coworkers and are fueled by social interactions. They are the genuine team players that get things done by collaborating and exchanging ideas and they feel fulfilled in tasks like doing code reviews. They usually like to focus on their tasks early in the morning or late at night and dedicate most of their office time to being connected.
2. the lone developers - “DND” is the sign they would wish to put on their desk. They avoid disruptions at all costs and feel most productive when they can work uninterruptedly on complex tasks such as solving problems, fixing bugs, or coding features.
3. the focused developers - efficiency is what they strive for. They prefer taking one task at a time, and they feel that spending too much time on something is making them unproductive.
4. the balanced developer - not really affected by noise or disruption, but more by unclear, irrelevant, or unfamiliar tasks. Oh, and they hate it when tasks are causing overhead! (but we guess nobody really likes this anyway)
5. the leading developers - here it is, the category that is actually not bothered by emails or meetings. They more so enjoy them and might feel less productive with coding activities than other developers. They aim for the team’s overall performance and try their best to facilitate everybody’s processes.
6. the goal-oriented developers - “Where’s my checklist? Another one’s down!” They feel productive when they complete or make progress on tasks, while multi-tasking or goal-less activities are quite a bummer for them. They don’t really mind meetings as long as they can bring clarity in achieving their goals.

Why should we care about these 6 clusters after all?

The answer is pretty intuitive - the better we understand how developers perceive productivity, the more we can contribute to supporting them and finding ways to foster productivity at work, both on individual and team levels. On a different note, knowing how each cluster defines its productivity can help measure the individual productivity more accurately, based on what’s most important to them (take for example the leading developer and the lone one and compare their productivity view after a day with unexpected meetings, calls, and emails).

On a team level, it can help team leaders or managers rethink their processes. If in an office context, maybe it’s a good idea to provide open spaces for the socials, but silent offices for the lones or focused. Being aware of the working styles and preferences of the members of a team can encourage implementing new practices, such as reducing ad-hoc meetings for lone and focused developers or using more asynchronous communication so they can choose when to respond to an inquiry. Similarly, when it comes to assigning tasks, it will serve as an indicator of whom will it bring more satisfaction to working on it - an explorative, open task, that can be a little unclear regarding its goals might not be everybody’s cup of tea (aka the goal-oriented, the lone or the balanced).

On the other hand, there is a trade-off between an individual developer's productivity gain, let’s say by avoiding interruptions from coworkers, and a team's productivity loss. This is where there needs to be a thorough exploration of the real impact a context switch can have on productivity, and not just on the perception of it.

Besides addressing interpersonal differences in work approaches and managing context switches, what else can be done to increase productivity?

Self-monitoring is a great place to start, but in a manner that provides more personal insight on productivity and the way time is spent. Companies should encourage employees to reflect on how focused they felt during work, to consider not just the goals they reached, but also the progress made towards certain goals, to think about the value their work has for the customers, and to keep in mind the bigger picture of the project/product.

The second good practice would be setting goals. As basic as it sounds, and as reluctant developers are to the idea, the participants in Meyers’s study stated that doing so “provides an overview of their task, allow them to prioritize work and to better react in cases of unplanned tasks, which were mentioned to be a major detriment to being and feeling productive.”

Is it too harsh to say “So much software, not enough productivity”? Long story short, yes it is. Indeed productivity would help the growth of the industry, while also providing SE a sense of meaningful and satisfactory work, but the shortage of professionals needed to keep up with the highly-technological times we’re living in is undeniably a reality. To end on an optimistic note, what we can do as an industry is to preserve and stimulate real talent and experienced professionals, to support and facilitate the work of SE in all the ways we can, and to encourage young talent to be dedicated by showing them how meaningful their work can be.

References:

Meyer, A. N., Zimmermann, T., & Fritz, T. (2017). Characterizing Software Developers by Perceptions of Productivity. 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). doi:10.1109/esem.2017.17

Meyer, A. N., Fritz, T., Murphy, G. C., & Zimmermann, T. (2014). Software developers’ perceptions of productivity. Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering - FSE 2014. doi:10.1145/2635868.2635892

P. Naur and B. Randell. Software engineering: Report of a conference sponsored by the nato science committee. Scientific Affairs Division, NATO, 1969.

D. A. Epstein, D. Avrahami, and J. T. Biehl, “Taking 5: Work-Breaks, Productivity, and Opportunities for Personal Informatics for Knowledge Workers,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16, 2016