Software Evolution is similar to a Bible Story: we, software creators, have a plan and goals and pursue them to create the perfect solution. We carefully select and add new features according to our grand design. We try to keep design clean and optimal. Things sometimes go wrong and the software system goes berserk, but we intentionally intervene and fix the problems to continue fruitful functioning of the system.

Biological Evolution is a different story: chaotic raise and fall of different species, without any plan and goals. New traits and variations appear all the time and many are useless or even harmful. Small modifications accumulate over time, cause significant changes and emergence of new species, better adapted for the evolving world. Survival of individual organisms and species is the matter of chance, pressure from environment and competition with other organisms.

Intentional growth of software versus chaotic evolution of organisms… Do they have anything in common?

Biological Evolution

In biology, evolution is the process of change in the inherited traits of a population of organisms from one generation to the next.

Species – a group of organisms that can reproduce with one another and produce fertile offspring.
Population – a localized group of individuals belonging to the same species.
Traits – particular characteristics of an organism.
Genes -portions of DNA molecule, which control inherited traits.
Genotype – complete set of genes within an organism’s genome.
Phenotype – the complete set of observable traits that make up the structure and behavior of an organism. These traits come from the interaction of its genotype with the environment.
Variations – the variation in phenotypes in a population reflects the variation in these organisms’ genotypes.

Most of the genome of a species is identical in all individuals of that species. However, even relatively small changes in genotype can lead to dramatic changes in phenotype: chimpanzees and humans differ in only about 5% of their genomes.

Mechanisms of increasing gene variations

• Mutations – mistakes that occur during DNA replication, which alternate version of a gene (alleles). Mutations give the new characteristics to the organism and can have positive and negative effects.
• Reshuffling of genes through sexual reproduction with random mixtures of their parents’ chromosomes.
• Gene flow – genes from one population enter the gene pool of another, e.g. migration between distant populations.

Mechanisms of decreasing gene variations

• Natural selection – a process causing heritable traits that are helpful for survival and reproduction to become more common in a population, and harmful traits to become more rare. Natural selection is the result of pressure from environment and competition between organisms.
  

• Genetic drift – produces random changes in the frequency of traits in a population, which results from the role chance plays in whether a given individual will survive and reproduce.
  

Though the changes produced in any one generation by drift and selection are small, differences accumulate with each subsequent generation and can, over time, cause substantial changes in the organisms.

Natural selection usually predominates in large populations, while genetic drift dominates in small populations. Some theories suggest that genetic drift and mutation play much larger role than natural selection in evolution.

Fitness is the central concept of natural selection. Fit does not necessarily mean biggest, fastest, strongest or having more children. Instead fitness measures the organism’s genetic contribution to the next generations – the proportion of subsequent generations that carry an organism’s genes.

Now we can provide another definition of evolution: the change in the gene pool of a population over time.

Outcomes

• Adaptation – structures or behaviors that enhance a specific function, causing organisms to become better at surviving and reproducing

• Co-evolution – the evolution of one species causes adaptations in a second species, such as a conflict between predator and its pray.

• Co-operation – mutually beneficial interactions. Can be found in social insects, such as bees, termites and ants, where sterile insects feed and guard the small number of organisms in a colony that are able to reproduce.
  

• Speciation – the process where a species diverges into two or more descendant species, mostly occurs in populations isolated geographically. It caused by gradual accumulation of small genetic changes.
• Extinction – the disappearance of an entire species. Indeed, virtually all animal and plant species that have lived on earth are now extinct.
  

Software Development Ecosystem

The first question: what is the object of evolution in Software Development?

People?
I don’t have guts to suggest interbreeding most successful programmers and receive new generations of the best programmers. Very impractical, too long to wait and modern people are not an easy target for experiments…

Code?
Code is a good candidate. We constantly evolve and change code. But individual code doesn’t have comparable set of genes and variations (unless it is a genetic algorithm or you create mutants with intensive copy and paste) and don’t compete with other code for survival (only at level of products where code is one factor from many). We could apply only limited evolutionary principles to code and they are mostly related to ideas behind the software systems.

Can we find a better candidate?

I believe that genome of software development is the knowledge about how to build software systems.

Knowledge, DNA of Software Development, has two components

• process: principles, activities, roles and flow (Agile, Extreme programming, Scrum, CMM, RUP, Waterfall, Cleanroom, Crystal Clear, SWE-BOK, Context Driven Testing, etc.)
  
• expertise: design, programming and domain (Algorithms, Object Oriented Design and Programming, Design and Enterprise Architecture Patterns, Refactoring, Domain Driven Design, Behavior Driven Design, Relational Databases, Web Design, Information Architecture, Usability, etc.)
  

In essence, this is the knowledge about what software team members do at any given time to create the system: understanding requirements, design and build, verification. Humans are the best carriers of this knowledge, but it is possible to transfer it with less effective documents, books and other media. People carry huge amount of intrinsic knowledge that comes with experience, which is difficult to extract, describe and transfer to other people. Our methodologies are created to represent software development knowledge (genome) in the formal way convinient to carry between people heads and projects. However, this genome immediately meets the harsh reality of environment pressure.

Software Development Environment:

• people: experience, capabilities, skills, motivation, personalities
• requirements: software needs, expectations, constraints, regulations
  
• customer: knowledgeability, relations and attitude
• market: trends, competition, standards
  
• technology: development platforms and tools, technical infrastructure
• organization: goals, plans, management, culture, leadership style, resources, finances
  

Software Development Ecosystem Characteristics:

• Populations – there are different kinds of software development (web, desktop, embedded devices; system, productivity, games; business, military, science; commercial, in-house, open source; for fun, business-critical or life-critical, etc.)
• Traits – characteristics of approaches used in software development
  
• Genotype – software knowledge carried by individual team and used in the project to build software system
  • Encoding of information – contains enough information to build an individual software system
    
  • Inheritance – knowledge can be passed to the next project iterations, other projects or to the whole development community
    
• Phenotype – all the activities in software project that are outcome of interaction between this knowledge and project environment.
• Co-evolution – ideas compete (Waterfall vs. Agile, Procedural vs. Object-Oriented vs. Functional Programming), cooperate and enhance each other (Extreme Programming and Scrum, Refactoring and Design Patterns)
  
• Mechanisms:
  • Mutations – continious invention and modification of the new approaches
  • Reshuffling – frequent recombination of various ideas
  • Gene flow – exchange of ideas with other populations and industries (e.g. NASA, Lean Manufacturing)
  • Natural Selection – software development knowledge is often under the pressure to achieve success, better meet customers needs and be effecient. Better methods have more chances to survive and become more common (e.g. adoption of Iterative approach that successfully solves challenge of changing requirements)
    
  • Genetic drift – some ideas and approaches become popular by chance, some great ideas disappear without wide acceptance. Management and leadership change, politics, better sponsors, strong opinion of less qualified people, lack of knowledge about alternative approaches could play significant role in establishing far from optimal development knowledge for the project.
    

Software Development Evolution is change over time in the essential knowledge about how to build software systems.

Examples of software development evolutionary genes (views, ideas and activities):

There is no best set of ideas. Success of the every software creation gene depends on interaction with a project environment. Biological evolution can teach us few lessons of selecting and evolving most advantageous genes for our projects. The next post will explore these lessons.

Resources:
Evolution, Wikipedia
Introduction to Evolutionary Biology

Posted by Andriy Solovey | Permalink | Trackback