As a software developer, it’s super weird that I have to manage and train my organizational superiors. It’s a notion that economists and business thinkers in academia generally rule out before the analysis, but it’s a real thing.
Software development firms are sometimes spoken of as flat organizations, but in fact many teams are genuinely inverted structures where senior developers are not only developing, but leading teams and even managing their managers. A senior developer may be more familiar with agile scrum, kanban, or other development processes or tools like JIRA, Trello, Rally, and so on, compared to their manager. Some organizations hire young or inexperienced managers to oversee development teams where some of the developers are substantially older. Developers may be more privy than their manager to things as obscure as contract details, corporate policies, and so on.
While it sounds counter intuitive, but a good developer may well be a better manager than their own manager. One common issue which pops up for the talented developer with a low quality manager is that the manager will adopt arbitrary policies which are bad for team productivity or satisfaction, financially a bad choice for the company, and/or result in a lower quality product. A common example is that a manager will overvalue physical collaboration in a software team, valuing “ass in the seat” time over actual productivity.
I’ve seen managers go into full blown denial about this sort of thing. When confronted with multiple measurements of quality and productivity, managers might simply deny the validity of any such measures. A related issue pops up where teams fail to implement collecting measures in the first place. Developers might complain that rigorous task definition, estimation, time tracking, and so on takes away from development time. Sometimes this is a great point and sometimes it isn’t.
This so-called administrivia of metric collection does take a certain number of minutes per day, but it reaps major rewards in the long term. Much like unit testing, the question should be: What is the estimated lifetime of this product or project? For prototyping projects which are only a few months or shorter, I agree that this formal process can net a loss to productivity, and I would say overtesting is also a real problem in these situations. As a rule of thumb, I would say projects and products lasting more than 6 months should implement these more formal processes. In addition, formal processes help manage complexity and scale, so even if a project is short, say a few months, you might want to implement formal process anyway if the team is large. A large team here being 10 or more developers.
Here’s an interesting wrinkle: Most developers are below average quality. You heard that correctly. Talent is non-normally distributed. It is a skewed distribution in the population of the labor force. Because talent is skewed, most developers are lower quality than average. This makes a problem because low quality developers are also resistant to metric collection. Such measures would show they are low quality! So the manager who is reluctant to collect metrics will find plenty of sympathy among the development team. This is a problem because many teams want to take a democratic 1-person-1-vote approach to establishing process and norms, but such an approach will tend to settle on bad process. Not simply sub-optimal process, but indeed, sub-average process.
Only really top notch developers lose out on missing metrics, because they end up having to pick up slack for the weaker developers and they aren’t given proper credit. I’ve been in multiple projects, including my present project, where I wrote more code, set up more key systems, and knocked out more tickets than other developers by a factor of 1.5x, 2x, 4x, or more. Companies typically see more variance in developer productivity than they see in developer compensation, and this nets a reduction to corporate profit
Why can middle managers get away with this? At the end of the day it reflects on poor organizational leadership or upper management. Some reasons for poor upper management execution which I have heard:
- Principal agent problems.
- I don’t buy this because if the organization is well-formed then every laborer is paid on their marginal productivity, so managers who implement better practices would be better paid.
- Rational ignorance
- Again, I don’t buy it. For the same reason outlined above. Rational ignorance may exist, but it’s not a root cause explanation. It would be the byproduct of something else on this list.
- Bad organizational structure
- I totally agree here. It’s a stylized fact that managers are not paid on their marginal productivity, and organizational structure explains some of this. Research exists showing that differing organizational structures differ in their labor efficiency, and law regulates certain industries, structures, practices, and even particular firm activity in some cases.
- Bad leadership incentives
- Totally possible, although I don’t know how prevalent it is. If a company hires an executive to do a specific task, instead of hiring an executive to act freely and efficiently, they will motivate rational ignorance with respect to all tasks other than the task identified.
- Poor management productivity measurement
- I absolutely agree this is the case, although it’s baffling why such a large problem with such obvious solutions persist. Skip level performance reviews, 360 performance reviews, and continuous performance review are all best practices well known in human resources and management, yet most projects do none of the above. Even when those are implemented, it’s usually done on a rating system instead of including objective measures of performance. The result is that bad management flourishes.
- Plain incompetence
- Obviously this explains some portion of the issue. Managers are not equally talented, and not even the best managers are omnipotent, omniscient, totally benevolent, and so on.
- Imperfect competition / normal profit / normal tolerance
- I think this is probably the key answer. Efficient behavior isn’t really needed for a company to succeed. Companies simply need to do about as well as other companies are doing. Instead of targeting perfection, then, the standard is something like “Well if this manager isn’t doing substantially worse than other managers then I guess that’s good enough.”
- Upper management and leadership will act with rigor motivated by their own incentives, modulated by their ability. Upper management is also generally high paid across society, though, which may result in relatively low motivation toward financial efficiency of particular processes, because highly paid people have diminishing utility of income. This may show one way in which large corporations with multiple levels of management becomes almost necessarily inefficient.
Tim Ferris speaks eloquently on a microcosm of this issue in the video below. He deals with the specific case of allowing employees to work remotely. Managers often resist this despite evidence that remote working is better for productivity and satisfaction. on “How Do You Train Your Boss to Value Performance Over Presence” addresses one of the most common issues I see with my managers, and one of the most common cases for a need to manage up. A sort of availability bias in management.