by on

HelpToyota introduced the andon cord in their manufacturing lines to help people to stop the production line and alert the people around if an abnormal situation arises. It is a cord that hangs above the head within easy reach of any to help to immediately gather the attention of the others. While the rest of the industry was treating production line is something that never should be stopped and let quality control take care, this one put power in the hands of people on the line to take a call to improve quality by attacking the problem at its source.

The idea then widely got adopted in different forms like ‘Help’ buttons in various manufacturing sectors. It was easy to adopt in manufacturing as we can see what is going wrong and gather together to immediately fix the problem. In software development it is not obvious when to pull the andon cord and how to collect thoughts of people on what to solve.

huddle

Dev huddles are the answer to the andon cord in software development. Huddles are very common in any team sports, team members quickly huddle to celebrate a goal or discuss a plan. The team also optimises over time to communicate very effectively in fewer words and quick time.

When to call for a dev huddle?
We need to call for a dev huddle when

  • Our programming has deteriorated from a flow to brute force or trial/error method. Manuals did not help to resolve and even a quick help from another team member did not help.
  • We find a badly developed code and need to bring it to attention of the team when it is still fresh in mind.
  • We are about to make a major change in the code base and every body needs to be aware of the incoming change so they are not surprised.

How to make sure that we don’t waste other’s time?

Team time interruption is expensive, so before calling for a dev huddle we need to make sure that we have all our show and tell pieces ready for discussion. One example is when stuck at a problem, quickly jot down the problem and the steps tried to resolve that did not work on the whiteboard in diagrams or words; then call for the huddle. If the resolution or direction is not found within 10~15 minutes, break the huddle to come and do a detailed research on the problem.

Why should dev huddles work?

Dev huddles work on the idea of crowd wisdom, the average output of a group is always higher than best individual in the group in most cases. Ideas & solutions can come from any one provided they are given a good explanation about the problem. Dev huddles also help in knowledge sharing in a terse & effective manner.

by on

I got a chance to walk on the Golden Gate bridge, could not stop admiring the beauty of the surrondings and the bridge. One can see what the bridge is subjected when taking a walk on this one, I arrived at one edge of the bridge when it was bright & sunny and by the time I reached the other end it was misty & foggy. It is constantly pounded by waves & high speed wind; and takes a lot of traffic. It left me curios to find the engineering behind it, after a while I stumbled on a documentary made by Nat Geo titled ‘Impossible bridges – Golden Gate’.

The engineers thought that it is impossible to build such a bridge because the water was too deep and winds were too fast and the distance is too long to cover. Even if built people wondered about how could it be sustained. Nat Geo’s documentary showed how the crew worked over the years to strengthen the bridge, rework the floor without disrupting much traffic flow. There were innovations like suspended traveling platforms to work on the cables. The bridge withstood an earthquake and was  strengthened to withstand stronger earthquakes. At the end of the documentary, the narrator mentions, he is sure that all parts of the bridge has been replaced at least twice but due to the engineers’ skill, people never knew that they are looking at a new bridge.

Golden Gate

The same holds good for long running software projects, many softwares over the course of time change so much internally based on the people working on it, which Richard Gabriel calls as Habitability in his book ‘Patterns of software architecture’. If the engineers who took over the maintenance of Golden Gate have not replaced the bridge over the course of time we would have a big heap of metal lying on the sea bed due to structural erosion. It is the same analogy that applies to software called ‘big ball of mud’ due to software rot.

The biggest reason given for software rot is that there is not enough time. Architecture is a long term concern, little beyond the learning horizon as mentioned by Peter Senge in his book ‘Fifth Discipline‘. We are not able to observe the effects of software architecture in the near term as there are very urgent business needs that keep coming up and they are never tangible like bridges; visualising them requires experience and skill. The advantage a software has is that it can be made more robust even after it has been built hastily.

How do we find that the architecture & design needs a revamp?

  • If the change to the software looks simple and trivial but takes a bafflingly disproportional amount of time to fix.
  • If it is difficult to visualise the code and explain to others in simple abstractions.
  • If a technology upgrade looks infeasible without a rewrite.
  • If there is too much dependency on individuals and context.

The revamp is not just about modular design & clean code but also adequate test coverage along with a robust continuous delivery mechanism similar to how the Golden Gate’s engineers used suspended traveling platforms to carry out their repair work on the cables without affecting traffic. A successful well maintained app that takes advantage of new technologies and changes hands would definitely been written many times over and no one would have noticed it.

by on

We are inept at perceiving the effects of debt, we grossly underestimate compounding factors and fall into a debt trap. Debt is tempting, it gives that edge to own things which is not possible now but only with the future effort that we can conveniently borrow and buy. There is a saying in Tamil, buying something in loan is like lighting a fire with borrowed hay but to repay it has to be paid in wood for same volume. If instant gratification takes priority then debt that needs to be repaid will hit us hard and bring us to a grinding halt. Corporates love debt, because with a lower cost of interest they will be able to create a high value output. This is where debt helps, if we are able to create something that is more valuable that can repay the debt with interest and still leave us with lots of money then debt is good.

Developers take debt analogy to code and create technical debt to get features fast to production, one major drawback is that most of us don’t understand monetary debt and its impacts well even though it can be quantified; but take decisions on technical debt which is difficult to quantify.

Let us have a look at an analogy, Neo has a trash can in his area that has to be cleared periodically. Clearing the trash can whether it contains little trash or full can of trash takes the same effort, so he prefers till the trash can fills up. The trash accumulates in a way that it doubles every day if not cleared. A new person Brio takes up the place of Neo, Neo explains that the first day only a gram of trash gets generated after clearing, but if left uncleared it doubles every day. He used to clear it every 10th day as it is around 500 grams by that time.

Brio observes that the trash piling up day to day is trivial to care about, around the tenth day he notices that trash is 500 grams and barely fills the can, he decides to come back 10 more days later forgetting the fact that it doubles. 10 days later what was a small can trash turns into an unmanageable heap of mess which is about 500 kilograms. The power of compounding is too easy to miss in the initial days, it will hit us hard who are used to linear maths.

Same is applicable for technical debts in an application. Tradeoff decisions have to be recorded and visited at frequent intervals to see if it is causing any harm than the convenience it provided. Technical debt also paves way for ‘Broken Windows’. Software development is a social activity, it will be very easy to make a broken window the new norm when code changes hands. Run your tech retrospectives; review code, design and architecture as an entire team at frequent intervals, that will help to manage technical debt better. Remember, it is not like corporate debt which is manageable by numbers.