From the first sight biological evolution looks too unpredictable to have any value for the constrained software development. We just don’t have time, money and resources for these wild experiment, unlimited trials and errors. It really seems that Nature could learn from us how to make things fast and effectively.

However, there are some principles that helped evolution to come up with amazing and efficient designs that made life flourishing on the Earth. This post will explore what software development could learn from biological evolution. See my previous post for the review of evolution concepts and mechanisms and how they could be applied to software development.

1. Evolution works

The evidence is richness of our flora and fauna which adapted to almost all possible conditions on our planet. Organisms live in polar ice, in desert rocks, in the bottom of Pacific ocean Mariana Trench (11 km deep) and even on the Moon.

Evolutionary approach is one of the most powerful and proven methods known to us for designing effective systems under complex uncertain conditions. There are no reasons why software teams should ignore this approach – they indeed often deal with complex uncertain requirements and environment.

There are 2 areas where evolutionary approach could be used:

1. project management – enhance the way people work and interact for building a system
2. design and build – the growth of this software system itself.
   

Simple Steps:

1. Generate new and modify existing ideas (increase variations).
2. Select potentially best ideas for building software better in a specific environment (form project genotype)
   
3. Make this generation of ideas work for some time (interact with environment).
   
4. Learn from results – keep useful and dump harmful ideas (natural selection – improve fitness).
   
5. Go to the first step and produce the new generation of ideas
   

We can make this process more intelligent and less random and still learn a lot from biological evolution.

2. Increase beneficial variations or how to generate good ideas.

Organisms will not evolve without getting new variations of traits. Some new variations in a gene pool could lead to better survival and fitness, but most of them are harmful. Software development genotype is a pool of ideas applied for building a specific software system.

How could we generate many good ideas and avoid harmful?

Mechanisms of idea’s evolution

• Mutations – continuous invention and modification of the new approaches based on individual creative insights and team brainstorming
  
• Reshuffling – recombination of existing ideas
• Gene flow – outside expert opinions and borrowed ideas from other disciplines (engineering, art, architecture, manufacturing, etc.) and study of complex systems (physical, biological, markets, societies, etc.)
  

Prerequisites for generating many quality ideas:

Powerful software teams with diverse range of

• skills / specialization – they cover broad knowledge areas of software development: customer domain and needs, project management, UI design, development, testing
  
• expertise / experience – combination of different approaches, insights and methods of experienced professionals with fresh ideas of newcomers.
  
• personalities – various traits that complement the whole team: creativity, discipline, pragmatism, attention to details , imagination, focus and leadership.

Decentralized control / self-organization
Centralized control reduces ability to come up with new ideas, which are not compliant with imposed top views. People from bottom, who implement the system, could generate better ideas, because they know environment deeply, experience it closely and receive immediate feedback from it. Independent people, who have control over project decisions and responsible for results will strive to make the best decisions.

Idea incubator

• Free flow of ideas, sharing of knowledge and expertise.
  
• Encouragement for innovative and creative thoughts and tolerance to failures.
• Opportunity for every team member to express ideas, be heard and respected.
  

Good ideas in a software project are necessary foundation of success, lack of them is disaster. Do whatever is possible to generate many good ideas.

3. Improve Fitness – keep useful and dump harmful ideas

Genes passed to the next generation are result of two factors: Fitness (natural selection) and Chance (genetic drift).

How could we improve blind process of selecting best ideas and avoid high failure rate?

Establish clear fitness criteria
Every organism have clear short-term goals – survive and reproduce to accomplish the long term goal – passing and increasing share of their genes in the next generations.
Software projects should have clear fitness criteria, which aligns team members, guide their decisions and actions. Criteria should be relevant to the most important goals of the project. It could reflect the purpose of the system, time to market, quality or cost. It should be clear, short and really important for survival and success.

Maintain selection pressure and increase role of Fitness vs. Chance
Low natural selection leads to more complex solutions. Most mutations which don’t increase fitness will be eliminated in a large population with fierce competition and strong selection pressure. This process could lead to a greater simplicity keeping only useful for fitness traits. In populations without much selection pressure, genetic drift increase chances of new genes survival (even disadvantageous). So population could have increased genes complexity without much advantage for fitness.

Avoid genetic drift
Genetic drift produces random changes in the frequency of traits in a population, which results from the role chance plays in whether a given individual will survive and reproduce. Chance play enormous role in biological evolution (maybe even more important than natural selection) and a great source of inefficiency. We can do better in software development. Transparent and independent decision making, self organization and power of diverse opinions will substantially decrease the role of the Blind Chance.

Therefore, a team with many good ideas shaped by selection pressure and fitness criteria will produce the best results.

4. Pragmatic implementation

Good enough working solutions
Animals shouldn’t be perfectly adapted, they just need to be in par with their competitors. Natural selection requires only requires something to work, not perfectly but work. Evolution is more likely reshape existing structures than invent new ones.
The classic example is the panda’s thumb, which it uses to grasp bamboo. The panda’s true thumb is committed to another role. The lobed fins of early fish have turned into structures as diverse as wings, fins, hoofs and hands.

In software we could be satisfied with solutions which are not perfect at the beginning. We can live with them knowing that evolution forces and pressure will shape ideas behind and we will make these solutions better with more experience.

Practical and immediate purpose

Every new feature in the system should have some immediate purpose – evolution invented only something that has purpose.

Exaptation – structures and behaviors that evolved for one purpose but take on a wholly new one, while remaining useful at every intermediate stage. It means that some features cannot evolve if they are not completely useful immediately. Two way radio could be useful for animal communication, but it didn’t evolve, because half-way radio is useless – it doesn’t tell anything useful for animals about their environment. Visible light is different – any small improvement in vision can provide big advantage in many environments.
There is an interesting question: could we miss great design ideas if we only develop for immediate purpose? There are many useful inventions that came as a result of fundamental science curiosity without much practical need at the beginning. However, I’m inclined to believe that software development is empirical discipline and more effective with developing for immediate purpose only.

Repurposing
Repurposing a structure does not have to involve the loss of the original structure. Reptilian jaw bones turned into mammalian ear bones, without the loss of the jaw. The neural circuitry that allows us to make fine limb movements may have been adapted to produce speech as well.

Sometimes just one aspect of a feature can be co-opted for another use. The first hard mineralised structures to evolve in our ancestors were the teeth of early fishes known as conodonts. Once the ability to form hard hydroxyapatite had evolved, it could be exploited elsewhere in the body and may have been the basis of the bony skeletons of all vertebrates.

Repurposing is an interesting lesson for the software development – successful reuse of existing structures for the new functions. At the first glance, it goes against main principles of good software design – single responsibility, isolation and decoupling of components. Biological organisms don’t have much choice with mutations- which could add only small changes. But efficient and economical designs of many organisms suggest that repurposing could be a good strategy. We could maximize reuse of the components for different purposes while keeping smaller inner parts isolated. Relentless refactoring, removal of duplication and creating abstractions are the main methods. The ultimate condition is good understanding and mental control over the system.

No refactoring (or elimination of unused traits)
Lack of refactoring seems as a serious flaw of biological evolution. New organisms repurpose what exists and build new structures on top of old without cleaning and eliminating unnecessary stuff. DNA has a lot of junk. These structures are not needed

• wings of ostriches
• appendix
  
• male nipples

Software team can use refactoring as an essential part of effective and pragmatic implementation and improvement of evolutionary approach.

5. Making impossible possible with small continuous changes.

Small changes accumulated over time cause substantial changes in organisms. Continuous improvement of the genotype and small frequent changes work for all living organisms in this unpredictable and very complex world. They lead to highly fit, adapted and efficient solutions.

By weeding out harmful mutations and assembling beneficial ones, natural selection acts like an “improbability drive” that can, given enough time, produce results that appear completely impossible at first glance.

One of the most important lessons for Software Development is possibility to arrive with great solutions without planning and predicting them in advance. It is one of the best strategies in the complex, uncertain and changing environment.

Agile approach is successful because it strikes the balance between adaptation to changing environment and predictability required for business.

Principles:

• Be open and ready to changes
• Frequently get feedback, change and adjust in small steps
• Continuously learn and reflect what works and what doesn’t
• Often evaluate if your project and systems are fit for their purpose and environment.

Summary

Evolutionary approach works if software team

• can generate many good ideas
  
• can select most useful ideas how to build the system
• works under healthy pressure to meet fitness criteria
• pragmatic in implementation
• learn from results, continuously adapt to needs and environment
  

Resources:
Comparing Intelligent Software Evolution to Chaotic Biological Evolution
Evolution: 24 myths and misconceptions, Newscientist (requires subscription)
Evolution, Wikipedia

Posted by Andriy Solovey | Permalink | Trackback