“Ruin in community is not caused by witches with broomstick but it’s something that starts in people’s heads.” – Prof. Preobrazhensky (Michail Bulgakov’s “The heart of a dog”)

The newborn software system looks beautiful inside after the few weeks of development. Several talented programmers put their souls together into this amazing software idea. But week after week the system become older and uglier. It could still look exciting for users, but the system rots inside. It is much more difficult to modify, it breaks often, development time and cost soars. Programmers become scared to work with it. The system becomes Frankenstein. Why could it happen?

Three reasons

There are hundred objective reasons why it could happen – complex technology, changing customer needs, management pressure, time to market and many more. But some teams still deliver a good system under these conditions. And some teams cannot deliver even in the most favorable conditions. Why?

Degradation of the system starts in the people heads and there are three main reasons:

1. Incomprehension – developers don’t understand the purpose, ideas, design or technology behind the system
2. Inarticulateness – developers unable to express ideas through clear and effective architecture, design and code
3. Inconsistency – developers cannot act consistently or don’t care about the system

Incomprehension – don’t understand

“I am so clever that sometimes I don’t understand a single word of what I am saying.” – Oscar Wilde
Every person has limits for understanding:

• We have limited memory and perception, which are far from perfection and distort reality.
• Team members as any individuals have different background, mindset and expertise. It is not a surprise that they could easily misunderstand each other.
• It is even more difficult to understand outsiders as clients coming from completely another world.
• Separated teams, make situation more complicated. Conway’s Law says
  • Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.
  • Corollary: If you have four groups working on a compiler, you’ll get a 4-pass compiler. It is easier to implement this way than to understand what other groups are doing.

It is hard job to reach good understanding. But if you and your team don’t understand the purpose, ideas, design and technology behind the system – you cannot effectively build the system.

Inarticulateness – cannot express

“I didn’t have time to write a short letter, so I wrote a long one instead.” – Mark Twain
Understanding is not enough. The crucial step is to actually write a program. And programming requires special style of thinking: logical, abstract and creative in the same time. Ability to clearly and effectively express oneself is a precious skill for any speaker and writer. Programming requires a similar talent multiplied on complexity of technological means for expression – software platforms, languages and tools. Every programmer is unique – our innate and learned skills determine our potential, our experience shapes our capabilities, and our tools frame our mind. After all, every mortal programmer has limits for the problem he could solve. Many programmers could write programs only for straightforward, defined and standard problems (and some have unique skills to create chaos even with simple programs). Few programmers could tackle complex, wicked and open-ended problems in the elegant, clear and effective way.

The system is doomed if you or your team cannot express ideas through clear and effective code, design and architecture.

Inconsistency – cannot act consistently or don’t care

“Of course, if people could just act consistently, they could keep their desks clean, avoid cavities, lose weight, give up smoking, play a musical instrument, and possibly even produce software on a regular and timely basis.” – Alistair Cockburn
Having full understanding and expertise to create the software system is still not enough. People are not completely rational beings and there are many difficult to explain factors that affect productivity, quality of work and the level of responsibility:

• how happy, motivated and satisfied they are – unmotivated, uncaring or stressed programmers will usually produce bad software.
• peer pressure, conformity and groupthink could substantially alter our behavior and thinking. The group could detach from reality and start believing in wrong ideas.
• environment, structure of work and ability to focus are important. Distractions, inconvenient environment and confusing tasks will have negative influence.
• team dynamics, leadership style and established processes affect team efficiency

We cannot simply tell people: “please, care and be responsible” and they will immediately become responsible, concerned and motivated to keep the system in the good shape. It is very difficult and often against our nature. People should make effort and need right conditions to behave rationally, consistently and meet expectations (which are imposed by other imperfect people :)).

Further complications

Technology Tower of Babel
“Go to, let us go down, and there confound their language, that they may not understand one another’s speech.” – Genesis 11:6-7

Complex technology, levels of abstractions and various components of the system require different perspectives, layers and approaches even for simple applications. Nonetheless, all these different representations create seams between subsystems and components. Mess and rot are often starting to appear around these places without proper care and mastery.

Consider, for example, representation levels and languages in the traditional web application:

• HTML – markup language for layout of web pages
• CSS – stylesheet rule-based language for describing presentation of web pages
• JavaScript – dynamically-typed language for web page functionality on browser
• ASP.NET / JSP / PHP – server-side declarative languages for web pages generation on the web server
• C#, Java – object-oriented languages for domain and middle-tier logic
• SQL – declarative query and data manipulation language

Few web developers master all these levels, because of time, experience and volume of knowledge involved. Web and graphic design, information architecture, usability, database programming, networking, server and infrastructure administration add additional perspectives to the system. This mixture significantly contribute to the complexity and effort to understand , evolve and maintain the system.

Increase in complexity and entropy
“The law that entropy always increases – the second law of thermodynamics – holds I think, the supreme position among the laws of Nature.” – Sir Arthur Stanley Eddington

Entropy affects all the complex systems in the Universe. Amount of disorder in these systems tends toward maximum. Any system degrades and collapses unless external or internal forces keep integrity of structure and adapt it to changing conditions.
The development team is this external force that evolves and maintains the software system. It is tough job, considering limited human ability for mental control over vast and complex information. In addition, more complex systems are less predictable and more difficult to change. There is essential complexity related to complexity of the problem and wasteful caused by inability to deeply understand the problem, find optimal solution or over-engineering. The effective development team should control essential complexity and avoid wasteful.

How to save the software system?

Any complex problem has the perfect software solution that meets two criteria:

• customer is completely satisfied, the system is simple and user friendly
• programmers can easily understand, support and modify the system for the future changes

Unfortunately, we don’t have now any formal methods how to reach such solution. Software development is empirical process and the best we can do is to assemble good team, equip and prepare ourselves for the journey. However, if the team rigorously tries to meet these two criteria from the first day of development – the chances are high that the system will be good at the finish. If the team temporarily abandon these criteria in hope to achieve them on later stages – the chances are high that the system will have serious problems at the finish.

Before start

1. Find right and diverse people – they are the most important factor of success. Experience and skills should match the problem, but there is more. Team should be diverse and members should supplement each other:
   • diverse people will better handle different system representations and knowledge: system architecture, domain expertise, web design, object oriented and database programming.
   • less risk for conformity, groupthink and distortion of reality by the whole group.
   • they could select from wider range of strategies for achieving results and be effective in execution of all tasks, from routine to creative.
   • finally, ideas, brainstorming and solutions will be more balanced, broad and have higher quality
2. Match people, problems and compensation – people will be happy, motivated and dedicated if they have interesting work – not overwhelmingly complex and not too simple. It will boost growth, motivation and productivity. Compensation is another important factor – it should be fair and linked to the end results of the whole project.
3. Give control to the team, define clear goals and set criteria of success – software team members are the best people to make project related decisions. They will make right decision if they understand project vision and what is important for success. The empowered team will take full responsibility, self-organize and really care.

After start

1. Establish excellent information flow – it is the basis of the development team understanding, judgment and decisions. A team will be the most effective in processing project related information if the team is
   • diverse – people will understand information differently and reconciliation of this knowledge will create much better understanding for the whole team
   • small – people will avoid communication overhead and high number of interactions
   • self-organized – people will know what is important and ask what is needed. They will strive for the better information instead of relying on external help and guidance.
   • facing customers directly, understand their ideas and feel their pain. It helps to avoid incorrect information caused by misinterpretation and distortion of information through people chains.
   • understanding and solving problems in the same time with all necessary information available
2. Fight complexity, which is the main reason for the project degradation. Too complex solutions cause poor understanding, fear of change and inefficiency in the development. The software code is the key indicator for the amount of entropy and health of the system. The best systems have easy to understand, clear and well-structured code even for the complex problems. Code is good when
   • it is minimal, readable and relevant
     • no duplication and dead code
     • no over-engineering
     • developers understand each other ideas behind the code
   • achieve optimal representation with
     • constant refactoring to improve design and understanding
     • balance between expansion and consolidation phases, which allow rapid evolution of the system while keeping structural integrity
     • frequent synchronization with design ideas: architecture, design and code evolve and reconcile against reality together. It is two way – architecture and design provide meaning, direction and high level explanation, while code is ultimate and true description of the system behavior. Good code will enable, enhance and provide feedback for the future architecture and design ideas. Bad code will kill even genius ideas.
   • support abstractions and decoupling between subsystems and layers
     • big picture is easy to keep in mind
     • clear boundaries, isolated behavior and minimal dependencies reduce complexity of each subsystem. Poor implementation in some places will have minimal impact on other components.
     • solution overall complexity grows while allowing human brains relatively easy understanding of the individual pieces.
3. Adopt and learn – every project has unique requirements, people and context. It is almost impossible to make right decisions from the beginning. Therefore, the process should have built-in adaptation and correction mechanism as one of the most important components.:
   • Different opinions and alternatives provide more chances for selecting the best strategy
   • Short problem solving cycles (a.k.a iterations) allow fast sync with reality, reconcile mistakes and boost learning
   • People are encouraged to learn, listen to others and change

It is a titanic work to keep the system away from Frankenstanization, disorganization and degradation. But it is possible if we realize – our success depends on struggle with complexity and… ourselves, human programmers.

[UPDATE: As few readers correctly pointed out, the post compares software systems to Frankenstein’s monster. The confusion with Frankenstein is a common mistake. Anyway, we are still a bit far from creation of creators of the monsters as Dr. Frankenstein :). ]

Posted by Andriy Solovey | Permalink | Trackback