From the first sight biological evolution looks too unpredictable to have any value for the constrained software development. We just don’t have time, money and resources for these wild experiment, unlimited trials and errors. It really seems that Nature could learn from us how to make things fast and effectively.

However, there are some principles that helped evolution to come up with amazing and efficient designs that made life flourishing on the Earth. This post will explore what software development could learn from biological evolution. See my previous post for the review of evolution concepts and mechanisms and how they could be applied to software development.

1. Evolution works

The evidence is richness of our flora and fauna which adapted to almost all possible conditions on our planet. Organisms live in polar ice, in desert rocks, in the bottom of Pacific ocean Mariana Trench (11 km deep) and even on the Moon.

Evolutionary approach is one of the most powerful and proven methods known to us for designing effective systems under complex uncertain conditions. There are no reasons why software teams should ignore this approach – they indeed often deal with complex uncertain requirements and environment.

There are 2 areas where evolutionary approach could be used:

1. project management – enhance the way people work and interact for building a system
2. design and build – the growth of this software system itself.
   

Simple Steps:

1. Generate new and modify existing ideas (increase variations).
2. Select potentially best ideas for building software better in a specific environment (form project genotype)
   
3. Make this generation of ideas work for some time (interact with environment).
   
4. Learn from results – keep useful and dump harmful ideas (natural selection – improve fitness).
   
5. Go to the first step and produce the new generation of ideas
   

We can make this process more intelligent and less random and still learn a lot from biological evolution.

2. Increase beneficial variations or how to generate good ideas.

Organisms will not evolve without getting new variations of traits. Some new variations in a gene pool could lead to better survival and fitness, but most of them are harmful. Software development genotype is a pool of ideas applied for building a specific software system.

How could we generate many good ideas and avoid harmful?

Mechanisms of idea’s evolution

• Mutations - continuous invention and modification of the new approaches based on individual creative insights and team brainstorming
  
• Reshuffling - recombination of existing ideas
• Gene flow – outside expert opinions and borrowed ideas from other disciplines (engineering, art, architecture, manufacturing, etc.) and study of complex systems (physical, biological, markets, societies, etc.)
  

Prerequisites for generating many quality ideas:

Powerful software teams with diverse range of

• skills / specialization – they cover broad knowledge areas of software development: customer domain and needs, project management, UI design, development, testing
  
• expertise / experience – combination of different approaches, insights and methods of experienced professionals with fresh ideas of newcomers.
  
• personalities – various traits that complement the whole team: creativity, discipline, pragmatism, attention to details , imagination, focus and leadership.

Decentralized control / self-organization
Centralized control reduces ability to come up with new ideas, which are not compliant with imposed top views. People from bottom, who implement the system, could generate better ideas, because they know environment deeply, experience it closely and receive immediate feedback from it. Independent people, who have control over project decisions and responsible for results will strive to make the best decisions.

Idea incubator

• Free flow of ideas, sharing of knowledge and expertise.
  
• Encouragement for innovative and creative thoughts and tolerance to failures.
• Opportunity for every team member to express ideas, be heard and respected.
  

Good ideas in a software project are necessary foundation of success, lack of them is disaster. Do whatever is possible to generate many good ideas.

3. Improve Fitness – keep useful and dump harmful ideas

Genes passed to the next generation are result of two factors: Fitness (natural selection) and Chance (genetic drift).

How could we improve blind process of selecting best ideas and avoid high failure rate?

Establish clear fitness criteria
Every organism have clear short-term goals – survive and reproduce to accomplish the long term goal – passing and increasing share of their genes in the next generations.
Software projects should have clear fitness criteria, which aligns team members, guide their decisions and actions. Criteria should be relevant to the most important goals of the project. It could reflect the purpose of the system, time to market, quality or cost. It should be clear, short and really important for survival and success.

Maintain selection pressure and increase role of Fitness vs. Chance
Low natural selection leads to more complex solutions. Most mutations which don’t increase fitness will be eliminated in a large population with fierce competition and strong selection pressure. This process could lead to a greater simplicity keeping only useful for fitness traits. In populations without much selection pressure, genetic drift increase chances of new genes survival (even disadvantageous). So population could have increased genes complexity without much advantage for fitness.

Avoid genetic drift
Genetic drift produces random changes in the frequency of traits in a population, which results from the role chance plays in whether a given individual will survive and reproduce. Chance play enormous role in biological evolution (maybe even more important than natural selection) and a great source of inefficiency. We can do better in software development. Transparent and independent decision making, self organization and power of diverse opinions will substantially decrease the role of the Blind Chance.

Therefore, a team with many good ideas shaped by selection pressure and fitness criteria will produce the best results.

4. Pragmatic implementation

Good enough working solutions
Animals shouldn’t be perfectly adapted, they just need to be in par with their competitors. Natural selection requires only requires something to work, not perfectly but work. Evolution is more likely reshape existing structures than invent new ones.
The classic example is the panda’s thumb, which it uses to grasp bamboo. The panda’s true thumb is committed to another role. The lobed fins of early fish have turned into structures as diverse as wings, fins, hoofs and hands.

In software we could be satisfied with solutions which are not perfect at the beginning. We can live with them knowing that evolution forces and pressure will shape ideas behind and we will make these solutions better with more experience.

Practical and immediate purpose

Every new feature in the system should have some immediate purpose – evolution invented only something that has purpose.

Exaptation – structures and behaviors that evolved for one purpose but take on a wholly new one, while remaining useful at every intermediate stage. It means that some features cannot evolve if they are not completely useful immediately. Two way radio could be useful for animal communication, but it didn’t evolve, because half-way radio is useless – it doesn’t tell anything useful for animals about their environment. Visible light is different – any small improvement in vision can provide big advantage in many environments.
There is an interesting question: could we miss great design ideas if we only develop for immediate purpose? There are many useful inventions that came as a result of fundamental science curiosity without much practical need at the beginning. However, I’m inclined to believe that software development is empirical discipline and more effective with developing for immediate purpose only.

Repurposing
Repurposing a structure does not have to involve the loss of the original structure. Reptilian jaw bones turned into mammalian ear bones, without the loss of the jaw. The neural circuitry that allows us to make fine limb movements may have been adapted to produce speech as well.

Sometimes just one aspect of a feature can be co-opted for another use. The first hard mineralised structures to evolve in our ancestors were the teeth of early fishes known as conodonts. Once the ability to form hard hydroxyapatite had evolved, it could be exploited elsewhere in the body and may have been the basis of the bony skeletons of all vertebrates.

Repurposing is an interesting lesson for the software development – successful reuse of existing structures for the new functions. At the first glance, it goes against main principles of good software design – single responsibility, isolation and decoupling of components. Biological organisms don’t have much choice with mutations- which could add only small changes. But efficient and economical designs of many organisms suggest that repurposing could be a good strategy. We could maximize reuse of the components for different purposes while keeping smaller inner parts isolated. Relentless refactoring, removal of duplication and creating abstractions are the main methods. The ultimate condition is good understanding and mental control over the system.

No refactoring (or elimination of unused traits)
Lack of refactoring seems as a serious flaw of biological evolution. New organisms repurpose what exists and build new structures on top of old without cleaning and eliminating unnecessary stuff. DNA has a lot of junk. These structures are not needed

• wings of ostriches
• appendix
  
• male nipples

Software team can use refactoring as an essential part of effective and pragmatic implementation and improvement of evolutionary approach.

5. Making impossible possible with small continuous changes.

Small changes accumulated over time cause substantial changes in organisms. Continuous improvement of the genotype and small frequent changes work for all living organisms in this unpredictable and very complex world. They lead to highly fit, adapted and efficient solutions.

By weeding out harmful mutations and assembling beneficial ones, natural selection acts like an “improbability drive” that can, given enough time, produce results that appear completely impossible at first glance.

One of the most important lessons for Software Development is possibility to arrive with great solutions without planning and predicting them in advance. It is one of the best strategies in the complex, uncertain and changing environment.

Agile approach is successful because it strikes the balance between adaptation to changing environment and predictability required for business.

Principles:

• Be open and ready to changes
• Frequently get feedback, change and adjust in small steps
• Continuously learn and reflect what works and what doesn’t
• Often evaluate if your project and systems are fit for their purpose and environment.

Summary

Evolutionary approach works if software team

• can generate many good ideas
  
• can select most useful ideas how to build the system
• works under healthy pressure to meet fitness criteria
• pragmatic in implementation
• learn from results, continuously adapt to needs and environment
  

Resources:
Comparing Intelligent Software Evolution to Chaotic Biological Evolution
Evolution: 24 myths and misconceptions, Newscientist (requires subscription)
Evolution, Wikipedia

Software Evolution is similar to a Bible Story: we, software creators, have a plan and goals and pursue them to create the perfect solution. We carefully select and add new features according to our grand design. We try to keep design clean and optimal. Things sometimes go wrong and the software system goes berserk, but we intentionally intervene and fix the problems to continue fruitful functioning of the system.

Biological Evolution is a different story: chaotic raise and fall of different species, without any plan and goals. New traits and variations appear all the time and many are useless or even harmful. Small modifications accumulate over time, cause significant changes and emergence of new species, better adapted for the evolving world. Survival of individual organisms and species is the matter of chance, pressure from environment and competition with other organisms.

Intentional growth of software versus chaotic evolution of organisms… Do they have anything in common?

Biological Evolution

In biology, evolution is the process of change in the inherited traits of a population of organisms from one generation to the next.

Species - a group of organisms that can reproduce with one another and produce fertile offspring.
Population – a localized group of individuals belonging to the same species.
Traits – particular characteristics of an organism.
Genes -portions of DNA molecule, which control inherited traits.
Genotype - complete set of genes within an organism’s genome.
Phenotype - the complete set of observable traits that make up the structure and behavior of an organism. These traits come from the interaction of its genotype with the environment.
Variations – the variation in phenotypes in a population reflects the variation in these organisms’ genotypes.

Most of the genome of a species is identical in all individuals of that species. However, even relatively small changes in genotype can lead to dramatic changes in phenotype: chimpanzees and humans differ in only about 5% of their genomes.

Mechanisms of increasing gene variations

• Mutations – mistakes that occur during DNA replication, which alternate version of a gene (alleles). Mutations give the new characteristics to the organism and can have positive and negative effects.
• Reshuffling of genes through sexual reproduction with random mixtures of their parents’ chromosomes.
• Gene flow – genes from one population enter the gene pool of another, e.g. migration between distant populations.

Mechanisms of decreasing gene variations

• Natural selection - a process causing heritable traits that are helpful for survival and reproduction to become more common in a population, and harmful traits to become more rare. Natural selection is the result of pressure from environment and competition between organisms.
  

• Genetic drift – produces random changes in the frequency of traits in a population, which results from the role chance plays in whether a given individual will survive and reproduce.
  

Though the changes produced in any one generation by drift and selection are small, differences accumulate with each subsequent generation and can, over time, cause substantial changes in the organisms.

Natural selection usually predominates in large populations, while genetic drift dominates in small populations. Some theories suggest that genetic drift and mutation play much larger role than natural selection in evolution.

Fitness is the central concept of natural selection. Fit does not necessarily mean biggest, fastest, strongest or having more children. Instead fitness measures the organism’s genetic contribution to the next generations – the proportion of subsequent generations that carry an organism’s genes.

Now we can provide another definition of evolution: the change in the gene pool of a population over time.

Outcomes

• Adaptation – structures or behaviors that enhance a specific function, causing organisms to become better at surviving and reproducing

• Co-evolution – the evolution of one species causes adaptations in a second species, such as a conflict between predator and its pray.

• Co-operation – mutually beneficial interactions. Can be found in social insects, such as bees, termites and ants, where sterile insects feed and guard the small number of organisms in a colony that are able to reproduce.
  

• Speciation – the process where a species diverges into two or more descendant species, mostly occurs in populations isolated geographically. It caused by gradual accumulation of small genetic changes.
• Extinction – the disappearance of an entire species. Indeed, virtually all animal and plant species that have lived on earth are now extinct.
  

Software Development Ecosystem

The first question: what is the object of evolution in Software Development?

People?
I don’t have guts to suggest interbreeding most successful programmers and receive new generations of the best programmers. Very impractical, too long to wait and modern people are not an easy target for experiments…

Code?
Code is a good candidate. We constantly evolve and change code. But individual code doesn’t have comparable set of genes and variations (unless it is a genetic algorithm or you create mutants with intensive copy and paste) and don’t compete with other code for survival (only at level of products where code is one factor from many). We could apply only limited evolutionary principles to code and they are mostly related to ideas behind the software systems.

Can we find a better candidate?

I believe that genome of software development is the knowledge about how to build software systems.

Knowledge, DNA of Software Development, has two components

• process: principles, activities, roles and flow (Agile, Extreme programming, Scrum, CMM, RUP, Waterfall, Cleanroom, Crystal Clear, SWE-BOK, Context Driven Testing, etc.)
  
• expertise: design, programming and domain (Algorithms, Object Oriented Design and Programming, Design and Enterprise Architecture Patterns, Refactoring, Domain Driven Design, Behavior Driven Design, Relational Databases, Web Design, Information Architecture, Usability, etc.)
  

In essence, this is the knowledge about what software team members do at any given time to create the system: understanding requirements, design and build, verification. Humans are the best carriers of this knowledge, but it is possible to transfer it with less effective documents, books and other media. People carry huge amount of intrinsic knowledge that comes with experience, which is difficult to extract, describe and transfer to other people. Our methodologies are created to represent software development knowledge (genome) in the formal way convinient to carry between people heads and projects. However, this genome immediately meets the harsh reality of environment pressure.

Software Development Environment:

• people: experience, capabilities, skills, motivation, personalities
• requirements: software needs, expectations, constraints, regulations
  
• customer: knowledgeability, relations and attitude
• market: trends, competition, standards
  
• technology: development platforms and tools, technical infrastructure
• organization: goals, plans, management, culture, leadership style, resources, finances
  

Software Development Ecosystem Characteristics:

• Populations - there are different kinds of software development (web, desktop, embedded devices; system, productivity, games; business, military, science; commercial, in-house, open source; for fun, business-critical or life-critical, etc.)
• Traits – characteristics of approaches used in software development
  
• Genotype - software knowledge carried by individual team and used in the project to build software system
  • Encoding of information – contains enough information to build an individual software system
    
  • Inheritance - knowledge can be passed to the next project iterations, other projects or to the whole development community
    
• Phenotype – all the activities in software project that are outcome of interaction between this knowledge and project environment.
• Co-evolution – ideas compete (Waterfall vs. Agile, Procedural vs. Object-Oriented vs. Functional Programming), cooperate and enhance each other (Extreme Programming and Scrum, Refactoring and Design Patterns)
  
• Mechanisms:
  • Mutations - continious invention and modification of the new approaches
  • Reshuffling - frequent recombination of various ideas
  • Gene flow – exchange of ideas with other populations and industries (e.g. NASA, Lean Manufacturing)
  • Natural Selection – software development knowledge is often under the pressure to achieve success, better meet customers needs and be effecient. Better methods have more chances to survive and become more common (e.g. adoption of Iterative approach that successfully solves challenge of changing requirements)
    
  • Genetic drift – some ideas and approaches become popular by chance, some great ideas disappear without wide acceptance. Management and leadership change, politics, better sponsors, strong opinion of less qualified people, lack of knowledge about alternative approaches could play significant role in establishing far from optimal development knowledge for the project.
    

Software Development Evolution is change over time in the essential knowledge about how to build software systems.

Examples of software development evolutionary genes (views, ideas and activities):

There is no best set of ideas. Success of the every software creation gene depends on interaction with a project environment. Biological evolution can teach us few lessons of selecting and evolving most advantageous genes for our projects. The next post will explore these lessons.

Resources:
Evolution, Wikipedia
Introduction to Evolutionary Biology

I can’t wait to share this simple secret with you right now.

The Secret: Effective Software Systems are the systems that easy to understand and operate with human brains.

Programmers are more productive with effective software systems. Programmers can better learn and grow these system. Programmers have less problems, work faster and make better decision with them.

Now, you can avoid spending time reading this post if you already know this secret and you know how to avoid building the software system that:

• almost impossible to understand in reasonable time
  
• has confusing and convoluted swamp of logic and structure
• scary to change as nobody has any clue what will be broken, but sure that it will be broken
  

If you are still interested, lets find out what makes software systems effective.

Software Development is a pure mental endeavor (except typing on keyboard) that includes 3 main activities:

• Understand - learn and know system concepts and implementation
• Evolve - build, modify and support growth of the system ideas in the code
  
• Share - communicate and exchange ideas about the system

Programmers should care about 7 areas to make the system better suited for our brains:

1. Knowledge Creation and Retention – parsing, memorization and comprehension of the system ideas
   
2. System Organization – elements, relations and structure in the system
   
3. Sustaining Emerging Order – support evolution of the system and gain control over chaos
   
4. Minimize Noise and Purify – avoid adding unnecessary stuff to the system
   
5. System Discovery and Learning – making sense of the system
   
6. Mental Models – our internal explanations for how things are working in the real system
   
7. Shared Knowledge – ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge.

1. Knowledge Creation and Retention

Information is not knowledge. – Albert Einstein

Programmers should keep as less as possible things in the mind when work with the system. Less information, simpler and more logical ideas behind the code cause less brain damage and more understanding.
Main principles:

• Chunking – combining many units of information into a limited number of units and chunks, so that information is easier to process and remember. Logical groups (subsystems -> modules -> classes) should contain only few elements. These groups should emerge with growth of the system functionality and complexity. They should replace large clusters of difficult to remember similar and unstructured elements.
  

• Abstraction - generalize, find common meaning and remove redundancy in the similar concepts across the system. You can substantially reduce number of ideas required for understanding the system.
• Simplicity - simpler ideas make understanding (and programmers life) much easier.
• Isolation - reduce relations and dependencies between elements. These relations add significant complexity to the system and therefore effort to understand and remember them. The volume of knowledge about relations could easily exceed the knowledge about individual elements.

Examples of useful tools for supporting knowledge formation:

• Unit tests – capture knowledge about the system correct behavior. They evolve with the system and become a parallel verification system. Good unit tests reduce our effort and limit necessary for understanding scope as they focus on the system behavior in isolated components and functions.
  
• Visuals - take advantage of diagrams, mind maps and pictures to simplify, distill and integrate important knowledge and ideas about the system
  
• Storytelling - create imagery, emotions and understanding of the system behavior through stories (our episodic memory is very powerful). For example, create narratives how users accomplish tasks with your system. Or share dramatic stories how programmers doomed the system with ignoring important practices.
  

2. System Organization

Be regular and orderly in your system organization, so that you may be violent and original in your software solutions. (paraphrasing Flaubert)

Human programmers shouldn’t have hard time to memorize and comprehend a software system. Computers don’t care much about the organization as long as the logic is correct, but we, human programmers, do. Clear, logical and consistent organization make us much more productive.
Main principles:

• Modularity - managing system complexity by dividing large subsystems into multiple, smaller self-contained systems
• Layering - process of organizing information into related groupings in order to manage complexity and reinforce relationships in the information
• Hierarchy - hierarchical organization is the simplest structure for visualizing and understanding complexity
  

3. Sustaining Emerging Order

Evolution is not a force but a process. Not a cause but a law. – John Morley

The main challenge for effective system organization is the fact that a system is constantly changing during active development. Good organization today doesn’t mean good organization tomorrow with many new added features. New customer needs, contribution of fellow developers and growing complexity push a software system to chaos.
Therefore, we have to keep organization optimal for today needs and still ready for tomorrow changes with

• Refactoring – constant effort to improve the system internal structure and making it effective for recent changes to prevent system degradation and rule of chaos
• Simple principles and rules (that everybody follows) help to keep consistency and integrity of the system in time of intensive growth and modifications.
• Small independent, single-purpose components – enable rich behavior with minimum number of elements, which could participate in various and often unforeseen scenarios. Bigger components are less reusable, more specialized and lead to more code, complexity and duplication.
• Building adaptive system using agile approach

4. Minimize Noise and Purify

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery

Complexity is one of the biggest problem for the programmers brains. While good organization and representation of the system ideas help to fight it, the ultimate challenge is to avoid adding unnecessary stuff to the system.

Principals for Complexity Reduction:

• Increase Signal-to-Noise Ratio – ratio of relevant to irrelevant information in the system. Minimize noise by removing unnecessary elements and minimize the expression of necessary elements.

• Follow 80/20 Rule (Pareto’s Principle ) – a high percentage of effects in any large system are caused by a low percentage of variables. In other words, 80% of requested functionality will be used rarely or not used at all, while substantially increasing system complexity.
  
• Consider Good Enough Alternatives – don’t dash against the rock implementing exactly what customer asks. Slight modifications and alternatives could often cover need with less effort and complexity.
  

5. System Discovery and Learning

The real voyage of discovery consists not in seeking new lands, but in seeing with new eyes. – Marcel Proust

Learning and navigating the complex software system is a difficult task. Usually developers read code, documentation and debug the system. These methods take time and often are not efficient.

A better way to discover the system and make it discoverable with:

• Entry Points – points of attentional entry into the system. There are points in the system that doesn’t require much knowledge about the rest of the system. They could be starting points to understand other elements. For a example, UI screens or the system input / output. Other parts could start making sense and uncover their purpose in relation to these entry points. Discover these points and help newcomers to find them.
• Progressive Disclosure – a strategy for managing information complexity in which only necessary information is visible at any given time. Good design and coding style allow understanding of intent and logic of the studied system area without deeply diving into boundless waters of implementation details.

Code example:
Criteria criteria = Criteria.AddPriceRange(10, 100).AddDateRange(Period.LastMonth);
IList orders = Database.Orders.LoadBy(criteria);
decimal totalTaxes = orders.Sum(order => order.Tax)

• Recognition over Recall – memory for recognizing things is better than memory for recalling things. (e.g. multiple choice questions with possible answers vs. fill-in-the-blank questions). Minimize the need to recall information from memory whenever possible. Use tools to make available options clearly visible. For example, Visual Studio Intellisense and Resharper remove need in memorizing members, purpose and syntax of .Net library classes and methods. Build tools or extend a development environment to construct code without memorizing all details.
  
• Inverted pyramid – method of presentation in which information is presented in descending order of importance. Discover first what is most important and make it easily accessible following by less and less important information.
• Performance load – the greater the effort to accomplish a task, the less likely the task will be accomplished successfully. Most important tasks should be easy to accomplish and most valuable information should be easiest to access (including direct access to the people who has the best knowledge).
  
• Picture superiority effect – pictures are remembered better than words. Use pictures and words together to reinforce the learning.
• Associations with familiar – link new information to already familiar concepts. Build on this knowledge by comparing and contrasting to reach deep understanding.
  
• Examples – learning examples is always one of the best method to understand how to use the system (including best practices, how-to, and tutorials).
  

6. Mental Models

A mental model is an explanation in someone’s thought process for how something works in the real world. – Wikipedia

People understand systems and environments and interact with them by comparing the outcomes of their mental models with the system and real-world domain concepts. Good mental models bring better understanding and optimal decision making.

Schemata – networks of connected ideas or relationships that help to organize experience and information into a meaningful system. Conversation between individuals is most effective if both share common schemata.
Examples:

• Design and architecture patterns – general reusable solutions to a commonly occurring problems in software design.
• System metaphors – a simple story of how the system works. A system metaphor provides common vision and shared vocabulary.
  
• Conventions – standard and well known approaches in architecture, design, code and user interface. They make intuitive and common sense and save mental energy for present and future developers. Custom and novel approaches should be used only if they have serious advantage.
  

7. Shared Knowledge

I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant. – Robert McCloskey

The system is rarely envisioned, built and used by the same person. A successful system is a result of people interactions – intensive ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge. It is not easy for people to properly translate, communicate and understand each other ideas. Complex knowledge, different background and experience increase the gap that people should jump to completely understand each other.
Main principles:

• Ubiquitous language – the shared language that people with different perspectives use to exchange ideas about the system. The same meaning of the words is very important for understanding, but very difficult to achieve.
  
• Clear goals and intention – help to align people thinking and actions, and avoid wasteful effort caused by misunderstanding or confusion.
  
• Express business domain concepts in the code – represent business concepts in the code. Developers will better understand domain, synchronize implementation with customer ideas and align the system with relevant business concepts. As a result, the system can effectively grow with the refinement and expansion of business needs.
  
• Code readability – good naming and readable code enables better understanding by fellow programmers, reduce chances of errors and lead to correct future modifications.
  
• Shared mental models – frequent face-to-face discussions, pair programming and direct conversations with customers bring shared understanding, effective exchange of knowledge and most optimal solutions.
  

Evolution didn’t consider adaptation of human brains for software development. But our brains is the most important tool for programming. You cannot change brains (and human nature), so make software development compatible with them. Build effective software systems.

Resources:
The Programmer’s Brains At Work: Understanding The Software System
Universal Principles of Design, William Lidwell , Kritina Holden, Jill Butler

“Ruin in community is not caused by witches with broomstick but it’s something that starts in people’s heads.” – Prof. Preobrazhensky (Michail Bulgakov’s “The heart of a dog”)

The newborn software system looks beautiful inside after the few weeks of development. Several talented programmers put their souls together into this amazing software idea. But week after week the system become older and uglier. It could still look exciting for users, but the system rots inside. It is much more difficult to modify, it breaks often, development time and cost soars. Programmers become scared to work with it. The system becomes Frankenstein. Why could it happen?

Three reasons

There are hundred objective reasons why it could happen – complex technology, changing customer needs, management pressure, time to market and many more. But some teams still deliver a good system under these conditions. And some teams cannot deliver even in the most favorable conditions. Why?

Degradation of the system starts in the people heads and there are three main reasons:

1. Incomprehension – developers don’t understand the purpose, ideas, design or technology behind the system
2. Inarticulateness – developers unable to express ideas through clear and effective architecture, design and code
3. Inconsistency – developers cannot act consistently or don’t care about the system

Incomprehension – don’t understand

“I am so clever that sometimes I don’t understand a single word of what I am saying.” – Oscar Wilde
Every person has limits for understanding:

• We have limited memory and perception, which are far from perfection and distort reality.
• Team members as any individuals have different background, mindset and expertise. It is not a surprise that they could easily misunderstand each other.
• It is even more difficult to understand outsiders as clients coming from completely another world.
• Separated teams, make situation more complicated. Conway’s Law says
  • Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.
  • Corollary: If you have four groups working on a compiler, you’ll get a 4-pass compiler. It is easier to implement this way than to understand what other groups are doing.

It is hard job to reach good understanding. But if you and your team don’t understand the purpose, ideas, design and technology behind the system – you cannot effectively build the system.

Inarticulateness – cannot express

“I didn’t have time to write a short letter, so I wrote a long one instead.” – Mark Twain
Understanding is not enough. The crucial step is to actually write a program. And programming requires special style of thinking: logical, abstract and creative in the same time. Ability to clearly and effectively express oneself is a precious skill for any speaker and writer. Programming requires a similar talent multiplied on complexity of technological means for expression – software platforms, languages and tools. Every programmer is unique – our innate and learned skills determine our potential, our experience shapes our capabilities, and our tools frame our mind. After all, every mortal programmer has limits for the problem he could solve. Many programmers could write programs only for straightforward, defined and standard problems (and some have unique skills to create chaos even with simple programs). Few programmers could tackle complex, wicked and open-ended problems in the elegant, clear and effective way.

The system is doomed if you or your team cannot express ideas through clear and effective code, design and architecture.

Inconsistency – cannot act consistently or don’t care

“Of course, if people could just act consistently, they could keep their desks clean, avoid cavities, lose weight, give up smoking, play a musical instrument, and possibly even produce software on a regular and timely basis.” – Alistair Cockburn
Having full understanding and expertise to create the software system is still not enough. People are not completely rational beings and there are many difficult to explain factors that affect productivity, quality of work and the level of responsibility:

• how happy, motivated and satisfied they are – unmotivated, uncaring or stressed programmers will usually produce bad software.
• peer pressure, conformity and groupthink could substantially alter our behavior and thinking. The group could detach from reality and start believing in wrong ideas.
• environment, structure of work and ability to focus are important. Distractions, inconvenient environment and confusing tasks will have negative influence.
• team dynamics, leadership style and established processes affect team efficiency

We cannot simply tell people: “please, care and be responsible” and they will immediately become responsible, concerned and motivated to keep the system in the good shape. It is very difficult and often against our nature. People should make effort and need right conditions to behave rationally, consistently and meet expectations (which are imposed by other imperfect people :)).

Further complications

Technology Tower of Babel
“Go to, let us go down, and there confound their language, that they may not understand one another’s speech.” – Genesis 11:6-7

Complex technology, levels of abstractions and various components of the system require different perspectives, layers and approaches even for simple applications. Nonetheless, all these different representations create seams between subsystems and components. Mess and rot are often starting to appear around these places without proper care and mastery.

Consider, for example, representation levels and languages in the traditional web application:

• HTML – markup language for layout of web pages
• CSS – stylesheet rule-based language for describing presentation of web pages
• JavaScript – dynamically-typed language for web page functionality on browser
• ASP.NET / JSP / PHP – server-side declarative languages for web pages generation on the web server
• C#, Java – object-oriented languages for domain and middle-tier logic
• SQL – declarative query and data manipulation language

Few web developers master all these levels, because of time, experience and volume of knowledge involved. Web and graphic design, information architecture, usability, database programming, networking, server and infrastructure administration add additional perspectives to the system. This mixture significantly contribute to the complexity and effort to understand , evolve and maintain the system.

Increase in complexity and entropy
“The law that entropy always increases – the second law of thermodynamics – holds I think, the supreme position among the laws of Nature.” – Sir Arthur Stanley Eddington

Entropy affects all the complex systems in the Universe. Amount of disorder in these systems tends toward maximum. Any system degrades and collapses unless external or internal forces keep integrity of structure and adapt it to changing conditions.
The development team is this external force that evolves and maintains the software system. It is tough job, considering limited human ability for mental control over vast and complex information. In addition, more complex systems are less predictable and more difficult to change. There is essential complexity related to complexity of the problem and wasteful caused by inability to deeply understand the problem, find optimal solution or over-engineering. The effective development team should control essential complexity and avoid wasteful.

How to save the software system?

Any complex problem has the perfect software solution that meets two criteria:

• customer is completely satisfied, the system is simple and user friendly
• programmers can easily understand, support and modify the system for the future changes

Unfortunately, we don’t have now any formal methods how to reach such solution. Software development is empirical process and the best we can do is to assemble good team, equip and prepare ourselves for the journey. However, if the team rigorously tries to meet these two criteria from the first day of development – the chances are high that the system will be good at the finish. If the team temporarily abandon these criteria in hope to achieve them on later stages – the chances are high that the system will have serious problems at the finish.

Before start

1. Find right and diverse people – they are the most important factor of success. Experience and skills should match the problem, but there is more. Team should be diverse and members should supplement each other:
   • diverse people will better handle different system representations and knowledge: system architecture, domain expertise, web design, object oriented and database programming.
   • less risk for conformity, groupthink and distortion of reality by the whole group.
   • they could select from wider range of strategies for achieving results and be effective in execution of all tasks, from routine to creative.
   • finally, ideas, brainstorming and solutions will be more balanced, broad and have higher quality
2. Match people, problems and compensation – people will be happy, motivated and dedicated if they have interesting work – not overwhelmingly complex and not too simple. It will boost growth, motivation and productivity. Compensation is another important factor – it should be fair and linked to the end results of the whole project.
3. Give control to the team, define clear goals and set criteria of success – software team members are the best people to make project related decisions. They will make right decision if they understand project vision and what is important for success. The empowered team will take full responsibility, self-organize and really care.

After start

1. Establish excellent information flow – it is the basis of the development team understanding, judgment and decisions. A team will be the most effective in processing project related information if the team is
   • diverse – people will understand information differently and reconciliation of this knowledge will create much better understanding for the whole team
   • small – people will avoid communication overhead and high number of interactions
   • self-organized – people will know what is important and ask what is needed. They will strive for the better information instead of relying on external help and guidance.
   • facing customers directly, understand their ideas and feel their pain. It helps to avoid incorrect information caused by misinterpretation and distortion of information through people chains.
   • understanding and solving problems in the same time with all necessary information available
2. Fight complexity, which is the main reason for the project degradation. Too complex solutions cause poor understanding, fear of change and inefficiency in the development. The software code is the key indicator for the amount of entropy and health of the system. The best systems have easy to understand, clear and well-structured code even for the complex problems. Code is good when
   • it is minimal, readable and relevant
     • no duplication and dead code
     • no over-engineering
     • developers understand each other ideas behind the code
   • achieve optimal representation with
     • constant refactoring to improve design and understanding
     • balance between expansion and consolidation phases, which allow rapid evolution of the system while keeping structural integrity
     • frequent synchronization with design ideas: architecture, design and code evolve and reconcile against reality together. It is two way – architecture and design provide meaning, direction and high level explanation, while code is ultimate and true description of the system behavior. Good code will enable, enhance and provide feedback for the future architecture and design ideas. Bad code will kill even genius ideas.
   • support abstractions and decoupling between subsystems and layers
     • big picture is easy to keep in mind
     • clear boundaries, isolated behavior and minimal dependencies reduce complexity of each subsystem. Poor implementation in some places will have minimal impact on other components.
     • solution overall complexity grows while allowing human brains relatively easy understanding of the individual pieces.
3. Adopt and learn – every project has unique requirements, people and context. It is almost impossible to make right decisions from the beginning. Therefore, the process should have built-in adaptation and correction mechanism as one of the most important components.:
   • Different opinions and alternatives provide more chances for selecting the best strategy
   • Short problem solving cycles (a.k.a iterations) allow fast sync with reality, reconcile mistakes and boost learning
   • People are encouraged to learn, listen to others and change

It is a titanic work to keep the system away from Frankenstanization, disorganization and degradation. But it is possible if we realize – our success depends on struggle with complexity and… ourselves, human programmers.

[UPDATE: As few readers correctly pointed out, the post compares software systems to Frankenstein's monster. The confusion with Frankenstein is a common mistake. Anyway, we are still a bit far from creation of creators of the monsters as Dr. Frankenstein :). ]

Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Is it easy to build useful programs for humans? Failure rates and dissatisfaction with the software projects (more than 50% still fails or challenged) show that it is not quite easy task. Can AI help to build more successful projects, compete and eventually replace human programmers?

From ideas to specifications

Human programmers face objective challenges in building software systems, which AI will face in the future:

1. It is difficult to understand what people need.
2. Customer’s ideas are shifting, once they start using the real system and experience all the consequences and effects for interacting with it in the context of their problems.
3. Customer’s needs are changing constantly reflecting outside business and company trends, situation and problems.

There are 4 stages to build a software system:

1. Understand unclear human needs and overcome communication barriers
2. Translate these needs into specification or the system model
3. Implement these specifications within constraints and required system qualities
4. Verify that the system meets expectations.

Two first stages need strong AI with human like intelligence and comprehension, two last stages could be achieved with weaker AI. The key is translation of human needs into specifications that machine could understand. Stronger AI will need less defined specification and could handle ambiguous messages and human language, weaker AI will need more details, which at the extreme means writing a program as almost we do now.

As a part of the system specifications we should provide to AI expected input, output and required system qualities (performance, reliability, security, etc.). Even today some trends and approaches reduce gap in translation customer needs to computer specifications:

• Domain Specific Languages – expressing programming logic in language, better suited for specific domains and problems; domain experts can use this language and define the logic. Intentional Software could offer even more interesting approach in the near future.
• Automated Acceptance and Unit Tests – xUnit, Fit and similar frameworks allow to write specifications as tests in declarative or programmatic style.
• Test Driven Development, Behaviour Driven Development and Domain Driven Design provide approaches for evolving and programming a software system primarily from customer perspective as oppose to traditional views for building the system with distinct technological perspective, often disconnected from customer needs.

These approaches put more stress and effort to specify WHAT is expected from the system and less stress on HOW these result could be achieved. It is similar to declarative paradigm of programming (HTML, SQL), where a programmer specify what he expects and the underlying system comes up with optimal algorithms to satisfy this request.

Going forward application programmers and domain experts will focus on describing needs and specifications for the system and AI will increase contribution to implementation of these requests. As AI becomes stronger it will need less and less formalized description and could start comprehend human language directly.

Implementation and Delivery

Baron Munchhausen escaping from a swamp by pulling himself up by his own hair

AI could deliver a system in several ways depending on our level of trust:

1. Transparent System – computer generates code and human software professionals review, modify, integrate and accept it. It is similar to vendor model to speed up the main development.
2. Black Box – human provide specification, integrate and test the system without much understanding what AI has built inside.
3. Key-turn System – we trust computer to specify, build and integrate the system without human involvement and verification.

The first option seems safer and easier, but it requires a computer to generate software code in a human-readable format, which adds complexity and double translation: customer -> computer -> programmer. The Black Box and Key-turn System options doesn’t require this complex translation and could be easier to implement.

Based on today approaches AI could build system in few ways:

• Brute force: genetic algorithms or swarm intelligence, where computer try to find the fittest solution evolving behavior of the system based on selections, mutations, recombinations or emergent behavior of individual agents, competing and cooperating with each other.
• Trained neural networks: interconnected groups of artificial neurons, imitating how our brains are working.
• Symbol processor: similar to the way how a human operates with knowledge based on classic theory of mind: semantic, logic, reasoning, abstractions, etc.

We have been already successfully using AI software solutions in many areas, e.g. manufacturing, finances, medicine, science, air traffic control. Computers even analyze volatile political and military situations and predict unpredictable. It is matter of time when these approaches will be applied to software development and computers will be able to program themselves.

Silver Bullet?

Steffe
AI indeed could solve problems that Frederick Brooks indicated as essential complexity in software development (No Silver Bullet):

• Complexity – computers could handle much more information then our individual brains, carry knowledge about all possible states of the system and effectively manipulate it.
• Conformity - computers could seamlessly conform to interfaces of the other system and support rules of interactions with brothers computers.
• Changeability - computers will have full intellectual control over the system code and can quickly understand what should be modified and effectively support change requests.
• Invisibility - our brains are specialized on particular processing of information influenced by our biological origin. Computer don’t have this problem and could naturally manipulate information without need to visualize and limit scope for better comprehension.

Can we assume now that a computer-programmer could be superior to a human-programmer and become a silver bullet?

Competing with human programmers

AI definitely will have some advantage over humans. The computer-programmer will show higher productivity, consistency, reliability and will not be affected by mood swings, personal problems or lack of recognition (unless computer intelligence will make twist in their evolution). However there are several human characteristics that will be difficult to beat.

Creativity and Innovation
Creativity is a mental process involving the generation of new ideas or concepts, or new associations between existing ideas or concepts. – Wikipedia
Gödel incompleteness theorem shows that a computer should go beyond rules and algorithms embedded by humans to become creative and innovative. It is still not clear if computer could go over initial set of rules, generate new ideas, extend and connect existing.

Adaptability and Learning
Humans have unique ability to adapt to changing environment and quickly learn new things. It doesn’t take much time to adjust thinking, learn new rules and concepts and start programming effectively. Human programmers can successfully shift to the new domains, still keeping their mastery, apply previous experience enhanced by new knowledge. Could computers become so flexible in emerging new domains, shift in knowledge, perspectives and needs?

Aesthetics and Usability
Human designers can naturally grasp and implement subtle aesthetic and usability needs enhancing user experience for dealing with the systems. This knowledge is ingrained in our psychology and it will be difficult to formalize it enough for computer to allow building easy-to-use and pleasant programs.

Problem solving and Diversity
Human rational mind combined with experience and intuition effectively deals with wicked problems, which involve uncertainty, trade-offs and multiple options. Our brain machine is specialized on solving this kind of problems and it will be challenge for computer to match these capabilities. AI could use brute force at some extent as with chess, but it has limits when number of variables and things to consider increases rapidly.
Diverse talents, broad range of views and different opinions often lead to surprisingly effective solutions, sometimes impossible to predict beforehand.

Generalization and Theorizing
Human ability to generalize, create abstractions and theories based on experience, experiments, many observations and individual cases are very valuable in the software development.
We can substantially reduce complexity of the system logic and even anticipate future needs by creating abstract models and solutions, which could be applied for wide range of different situations and encapsulate theories behind our knowledge and understanding of the problem domain.

Understanding and Shared Experience
Humans will understand other humans much better than computer unless computers tap into our minds and start reading our thoughts. However, it could be scary and still not clear if AI could filter productive thought from the non sense our brains produce.
Human build together shared experience – less and less information is required to understand each other and people become more effective as a team – at the end, ideas and their flow is the most important part in software development.

Seeing The Big Picture and Thirst for Knowledge and Achievements
Many programmers have diverse background, education and knowledge – far more than required to build programs. Many solutions require to go outside of the problem context and understand broader picture. Some solutions could emerge from different problem domains and unexpected areas of human knowledge.
Human strength is irrational curiosity, thirst for knowledge and irresistible drive for achievements, which move our progress forward, open new areas of knowledge and make impossible possible. Can computers inherit this drive and motivation?

Can even super intelligent and powerful computers replicate human strengths that make us so effective programmers? There are definitely some areas that human programmers will rule for a long time. Does it mean that we could relax? I doubt it – computers are taking over more and more human programmer responsibilities and soon they will affect our jobs and replace us in some programming areas. What should modern developer do to stay competitive with gaining strength and power computer AI? This is the topic of the next post.

Intelligence is what you use when you don’t know what to do. – Jean Piaget

Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Computers blindly follow our instructions. They are much faster than humans, but still computers are stupid things dependent on our algorithms and knowledge how to solve problems.
Even huge processing power is not enough to start programming. Non-trivial solutions require understanding of ideas, problem solving, learning from experience and much more – everything what we can define as intelligence. Can computers become smarter than human programmers?

What intelligence is required for building programs?

What is human intelligence?

We have pre-wired capabilities: large and effective brains, genetic development of specialized functional areas (vision, speech, reasoning, etc.). However, we gain our intelligence mostly through learning and experience.
Jean Piaget defined four stages for human intellectual development:
1. Sensorimotor stage: from birth to age 2 years (children experience the world through movement and senses and learn object permanence)
2. Preoperational stage: from ages 2 to 7 (acquisition of motor skills)
3. Concrete operational stage: from ages 7 to 11 (children begin to think logically about concrete events)
4. Formal operational stage: after age 11 (development of abstract reasoning).

Nobody directly puts intellect in our heads. We construct our own intellect through complex interactions with the world and other people all our life long.

Highlights:

• Memory is the basis for all intellectual activities. Memory mostly encodes the relationships between things than the details of the things themselves. It maybe exists only to make good predictions about future
• We are not only using senses and react on external stimulus (keeping car on the road), but constantly constructing an internal model of the outside world and predicting how things behave (our and other cars dynamic).
• Our emotions are important as they are brain states that quickly assign value to outcomes and provide a simple plan of action. Therefore, emotion can be viewed as a type of computation, a rapid, automatic summary that initiates appropriate actions.
• Other species are hardwired to solve particular problems, while our ability to abstract allows us to solve an open-ended series of problems.

• Interaction between senses and memory allows us to construct a qualia (properties of sensory experiences). We could create a scene in our mind and make connections with past scenes.
• With ability to narrate and long-term memory we connect conceptual systems and develop semantics (meaning) and true language. Now we can become conscious of being conscious.

Building intelligent computers.

Computers could become intelligent in several distinct ways:

1. Simulation – scan and copy brain structure and emulate with computer how brain functions.
2. Singularity – in a system complex enough, consciousness and intelligence could simply pop into existence without our participation. Star systems emerged from simple particles, biological life emerged from simple chemical elements, and new intelligence could emerge from huge number of computers and their networks.
3. Symbiosis – extension of human capabilities with machine intellect (transhuman and posthuman) using nano-, bio- and information technology advances.
4. Artificial Intelligence built by humans.

AI is the most realistic way. History of AI had few winters, one of each still continues. Despite failed grand expectations, AI brought many practical results and applications:

• Postal Services use AI to help deliver the millions of packages that pass through its transportation network every day in the most efficient way possible.
• Telecoms operators use AI to establish the quickest connections for phone calls through their networks or to retrieve web pages speedily from the Internet.
• Manufacturers and retailers use AI for optimization of their supply chains.
• Financial organization use AI to organize operations, invest in stocks, prevent fraud, and manage properties. Algorithms carry 70 % of foreign currency trades.
• A medical clinic can use AI systems to organize bed schedules, make a staff rotation, and provide medical information.
• Neural networks are used in homeland security, speech and text recognition, medical diagnosis, data mining, and e-mail spam filtering.
• Robots have become common in many industries. They are often given jobs that are considered dangerous and exhausting to humans. General Motors uses around 16,000 robots for tasks such as painting, welding, and assembly.

There are 2 main school of thoughts for building AI: Classical Theory of Mind and Connectionism.

• The classicist believes that mind is a symbolic processor, where strings are produced in sequence according to the instructions of a (symbolic) program. The mind operates by performing purely formal operations on symbols.
• The connectionist views mental processing as the dynamic, bottom up and graded evolution of activity in a neural net. It models mind as the emergent processes of interconnected networks of simple units, usually neural networks.

Conventional AI (based on classic theory)

• Expert systems (knowledge based): apply reasoning capabilities to reach a conclusion. Expert system contains some of the subject-specific knowledge, and contains the knowledge and analytical skills of one or more human experts.
• Case based reasoning: stores a set of problems and answers in an organized data structure called cases. A case based reasoning system upon being presented with a problem finds a case in its knowledge base that is most closely related to the new problem and presents its solutions as an output with suitable modifications. It is used in Recommendation and Decision Support Systems, Help Desk, medicine.
• Bayesian networks: used for modelling knowledge in bioinformatics, medicine, engineering, document classification, image processing, data fusion, and decision support systems.
• Behavior based AI: a modular method of building AI systems by hand.

Computational AI (based on connectionism)

• Neural networks: trainable systems with very strong pattern recognition capabilities. Neural net contains simple units, each unit’s activation depending on the connection strengths and activity of its neighbors, according to the activation function.
• Fuzzy systems: techniques for reasoning under uncertainty, have been widely used in modern industrial and consumer product control systems; capable of working with concepts such as ‘hot’, ‘cold’, ‘warm’ and ‘boiling’.
• Evolutionary computation: applies biologically inspired concepts such as populations, mutation and survival of the fittest to generate increasingly better solutions to the problem. These methods include evolutionary algorithms (e.g., genetic algorithms) and swarm intelligence (e.g., ant algorithms).

Learning
Computers can quickly replicate and copy their intelligence as oppose to human individual learning. However, unique intelligence of each human is a strength as it adds diversity, different strategies and wide range of solutions if applied properly (Wisdom of Crowds).

The Grand Challenge: Can a computer understand?

The Computational Theory of Mind claims that the brain is a kind of computer and that mental processes are computations. If it is true we can build AI, which can match human mind (strong AI), have intelligence and can understand meaning. Otherwise, we will struggle to build better and smarter algorithms (weak AI), which will never achieve true intelligence in the human sense.

There are three serious objections to possibility of building strong AI.

• Gödel Incompleteness Theorem
• Searle’s Chinese Room Argument
• Wittgenstein’s Beetle and The Box

1. Gödel incompleteness theorem
Any effectively generated theory capable of expressing elementary arithmetic cannot be both consistent and complete. (For more explanation see here and here.)
Any mathematical or logical theory should have effective ways to validate proofs. However, each of them has some initial propositions (axioms), which assumed to be true and cannot be proved within the system. To prove them you can go outside the system and come up with new rules and axioms, but by doing so you’ll only create a larger system with its own unprovable statements. The implication is that all logical systems of any complexity are incomplete, i.e. have true statements without prove.
Modern computer is a logical system. Therefore, computer cannot have true human intelligence as it will be always limited by fixed axioms and initial rules programmed by humans, while humans can discover new axioms and rules. In other words, a computer can play chess, regulate air traffic, trade stocks based on initial programmed rules, but it cannot define new rules which are not derived from initial set.
Another interesting outcome is that we will never be able to understand ourselves, since our mind is closed system, which can only be sure of what it knows about itself by relying on what it knows about itself.

2. Chinese room argument
In his argument Searle argued against a position that computer can have strong AI, think and understand.

Suppose a human is locked in the room. He takes Chinese symbols as input, follows thick rule book on English and returns Chinese symbols as output. But the person doesn’t understand even a word of Chinese! If you replace human in Chinese room with a computer following these instructions, it can easily pass the Turing test and convince a human Chinese speaker that the program is itself a human Chinese speaker.
But computer is the mindless operator of symbols with zero understanding of language and meaning (as your dishwasher doesn’t understand what it is washing). Therefore, we can simulate mind in computer, but we cannot create computer mind that understands meaning. Computers are missing intentionality (relationship between mental acts and the external world). A computer points to other data while human intentionality points outside the brain (e.g. to real flowers).

There are interesting counter-arguments like the System Reply, which debates that in Searle’s thought experiment a human is a processor and don’t understand Chinese, but the whole system can understand. However, the Chinese room argument still holds position against strong AI from 1980.

3. Wittgenstein’s Beetle and The Box (Private Language Argument)
Wittgenstein shows that the idea of a language understandable by only a single individual is incoherent. Imagine, he says, that everyone has a small box in which they keep a beetle. However, no one is allowed to look in anyone else box, only in their own. Over time, people talk about what is in their boxes and the word “beetle” comes to stand for what is in everyone box.
Wittgenstein is trying to point out that the beetle is an analogy to an individual mind. No one can know exactly what it is like to be another person or experience things from another perspective (look in someone else box), but it is generally assumed that the mental workings of other people’s mind are very similar to our own (everyone has a beetle which is more or less similar to everyone else).
Therefore, computers should be part of our life, social interactions and cultural context to really understand us and construct similar meaning.

Summary

Deep Thought

Human intellectual capabilities are amazing and unique and human programmer will be superior comparing to AI for a long time. However, our modern civilization cannot progress without computers – we continue building better computers and require more and more from them. It is inevitable that computers will become intelligent whether it happens with our help or not. There are serious objections to possibility of building human-like “strong AI”; however, “weak AI” even today stepped in the territory previously believed to be human only. A computer needs programs and it is matter of time before they learn to program themselves and who knows what happened after this. But if computers continue to serve humans, even intelligent computer should effectively interact and understand us to write useful programs. This is a topic for the next post.

Useful Resources:
Standford Encyclopedia of Philosophy
Business By Numbers, The Economist
Great Ideas of Psychology, Teaching Company, Professor Daniel N. Robinson
Computing machinery and intelligence, Turing, A.M. (1950).
Mind, Brains, and Programs, by John R. Searle
10 Unsolved Mysteries Of The Brain, Discovery Magazine
Evolution in Your Brain, Discovery Magazine

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations - optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions - the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation - a software project consumes energy to keep going, mostly in the form of money :)
• Instability - software development deals with constantly changing situations
• Multiple equilibria – there are many possible satisfying solutions
• Complexity - any non-trivial software project has complexity
• Hierarchies - there are many perspectives and levels – organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Why self-organized systems are better?

1. Distributed control – “control” of the system organization is typically distributed over the whole of the system. All parts contribute evenly to the resulting arrangement. The free market is a good example. There are many independent market agents – companies and individuals. Problems with individual company or even market segment doesn’t effect economy much as there are always players who will use this opportunity and take over the vacant niche. The brain is another example. Its organization is distributed over a network of interacting neurons. Although different brain regions are specialized for different tasks, no neuron or group of neurons has overall control. This is shown by the fact that minor brain lesions because of accidents, surgery or tumors normally do not disturb overall functioning, whatever the region that is destroyed.

   Therefore, in effective software projects any skilled and experienced developer with tacit and specialized knowledge should have control over architecture decisions in his area as oppose to giving this control to few architects (or what even worse managers). Global order should emerge from these local development decisions.

2. Robustness – self-organized systems are relatively insensitive to perturbations or errors, and can restore themselves, unlike most human designed systems. One reason for this fault-tolerance is the redundant, distributed organization. Another reason for this intrinsic robustness is that self-organization thrives on randomness, fluctuations or “noise”, which could bring unexpected and better solutions. And third reason is stabilizing effect of feedback loops.
3. Feedback - the dynamics of a self-organizing system is typically non-linear, because of circular or feedback relations between the components. Positive feedback leads to an explosive growth, which ends when all components have been absorbed into the new configuration, leaving the system in a stable, negative feedback state.

   Effective feedback loops could be established only with open, adaptive and distributed approach contrary to closed, rigid and centralized traditional approach to software development.

4. Emergence - property or feature not previously observed as a functional characteristic of the system. Self organization turns a collection of interacting elements into an individual, coherent whole. Elements produce their own emergent properties and form the next level of structure in the system, which in turn form the building blocks for the next higher level of organization, with different emergent properties, and so on.

   Therefore, it is almost impossible to come up with these properties and decisions on the preliminary software design stage. The key for building great software systems is to learn how to tame and use emergence effectively instead of fighting against it.

5. Bifurcations - which of the possible configurations the system will settle in depends on a chance. A system may settle in a range of stable configurations. Since small fluctuations are amplified by positive feedback, this means that even small and impossible to observe fluctuation can lead to the to completely different project outcome.

   Therefore, even if we know business needs, environment and system specifications at the beginning of the process, the outcome is still unpredictable. It is waste of effort trying to predict and plan it – it is better to prepare for changes and adaptation.

6. Adaptation - a configuration of a system may be called “fit” if it is able to maintain or grow given the specific configuration of its environment. An “unfit” configuration is one that will disintegrate the system under the given boundary conditions. Different configurations can be compared as to their degree of fitness, or probability to survive under the given conditions imposed by the environment. Thus, adaptation can be defined as achieving a fit between the system and environment. Systems may be called adaptive if they can adjust to such changes while keeping their organization as much as possible intact.

   Considering above points, adaptation is very important for survival and success of the software project.

7. Self Regulation - adaptation can be considered as a problem of regulation or control: minimizing deviations from a goal configuration by counteracting perturbations before they become large enough to endanger the system.

   Therefore, the software development process should be able to produce variety of actions to cope with each of the possible perturbations and select the best action for the given problem. How it is possible without developers, creators of the software system? They should be empowered, trained and encouraged to do this.

How to control a self-organized system?

1. Establish attractors (state of equilibrium, a preferred and stable position for the system). It is better to establish vision, goals for expected system, good motivation and compensation for the team instead of telling what to do to and how to achieve this state.
2. Establish principles (or fitness criteria) for choosing the best action for the given circumstances.
   • The most straightforward method is to let the environment itself determine what is fit by trial and errors. This can be dangerous for the company since it may lead to the failure of the software project.
   • Another approach is to create internal models of the environment. This allows to try out a potential action “virtually”, in the model, and use the model to decide on its fitness. Theoretically, this is the most effective approach and many architect are trying to follow it, but it is very difficult in reality as this approach should take in consideration huge number of variables and create credible simulation.
   • Use simple rules, empirical patterns and best practices – considering the current state of the software development it is the best approach.
3. Create effective self-organized teams.

Developers are not janitors and prima donnas, because they are the most important people to maintain healthy self organization in the development of the software system, avoid system degradation and lead to the project success. Architects should help developers to establish better attractors, fitness criteria and internal models (or patterns) for selection of the optimal solutions, but they are not the best people to make decisions about these solutions. Certainly it is valid statement, if we agree that evolution, adaptivity and self-organization are important (bottom up approach). Please notice, in this post I even didn’t touch people, idea flow and psychology factors – I considered software development only from perspective of the theory of complex systems and self-organization.

Useful Resources:
Self-Organizing Systems FAQ
The Science of Self-Organization and Adaptivity, Francis Heylighen
Order Out of Chaos, Ilya Prigogine

Recently I was reviewing requirements for Microsoft Architect Certification and had found description for architect roles. After reading I had a question: do we need this kind of architects in a software team? Can they really help with building successful software?

The summary of architect responsibilities (as per Microsoft):

• Enterprise architects: set the overall vision and framework for the IT environment from a strategic business perspective.
• Solutions architects: design the solution to take advantage of the existing assets, integrate them into the existing environment, follow the enterprise architecture, and solve the business problems of the business owner or unit.
• Infrastructure architects: responsible for creating an architecture that meets the business and service level agreement requirements of the business owners and supports the applications and solutions that are required to operate their day-to-day businesses.

The bottom line is that company hires expensive and canny software architects before start of the project. They talk with the business, come up with solutions and make bright key decisions about architecture. But they don’t program. During the course of the project they ensure that developers don’t spoil this great architecture.

Now business owners can hire not so bright programmers, who should just follow directives and implement this great architecture. There is even correspondence with theories of some management consultants that programmers are just house painters.

Completely irrelevant questions: Can you trust advise of your friend about Paris, if he saw the city only on TV? Can you trust battle plans if battle didn’t begin and your commander knows little about enemy? How many chess moves ahead can you predict with high certainty before start of the game?

I don’t reject a need in architecture, but I don’t agree who, when and how should make architecture decisions in a software project.

Top 10 reasons why you don’t need Software Architect

The Architect (dedicated non-programming technical decision maker and problem solver for business):

1. Has outdated programming knowledge and experience, loss of touch with modern development approaches and practices.
2. Don’t program and don’t know much about evolving system internals, but makes key technical decisions. Often has completely irrelevant and unreal picture what is happening with the system.
3. Tends to complex, premature and generic solutions when the system is still in infancy and nothing is clear. Applies latest modern buzzword technologies as SOA, MDA, SaaS, Software Factories, etc. which look so beautiful in technical magazines, conferences and CV, but cause unnecessary headache for developers.
4. Plays role of the middleman introducing complexity in coordination and project responsibilities. Represents software team in interactions with business customers reducing communication value for the rest of the team and impacting idea flow.
5. Reduces quality of decisions, which become limited to one perspective; decision making starts lacking diversity, independence and decentralization, which are essential attributes of collective intelligence.
6. Creates tension with developers who experience mismatch between grand design and reality. Often continues pushing design decisions until the system becomes overly complex, difficult to change and becomes completely unusable.
7. Secures job and justifies high salary – becomes authoritative center for solving business problems without much input from the team.
8. Causes loss of sense of ownership, motivation and accountability in developers by detaching them from the key architecture decisions.
9. Concentrates project knowledge and the big picture in one head, limiting (and sometimes preventing) complete understanding for others.
10. Contributes to creation of specialized IT verticals that hurt relations with the business.

I’m sure that some architects bring strong technical leadership, excellent solutions and overall project success. However, the architect role itself, as described by Microsoft and employed by other companies, doesn’t encourage productive development and effective solutions.

What is effective architecture? Who is effective architect?

Emerging and evolutionary architecture is a core for a successful agile (and not only) software system, emerging from the implementation of business needs, learning from working code, problems and interactions with people, other systems and operational environment.

Effective architecture:

1. Provides stable foundation and integrity for the growing software system, enabling desired system qualities for the business solutions.
2. Ensures seamless and consistent integration, well defined interfaces and interaction between subsystems, external systems and operational environment.
3. Supports emerging abstractions, key system elements and frameworks for conceptual integrity, efficiency and reuse.

Effective architect:

1. Guards a system against failure, sensing worrying changes in the project dynamic, system code and business requests.
2. Keeps the trinity of system qualities (known from time of Roman empire) in a balance and prevent degradation and entropy:
   • Firmitas (Strength) – the system is reliable, secure, has good performance and easy to support
   • Utilitas (Utility) – the system is usable, meets business business needs and add business value every day instead of drifting into technology trenches
   • Venustas (Beauty) – code and system structure are clean, easy to understand, minimal
3. Encourages constant improving of the code design, enhancing system abstractions and structure, removing duplication, defining boundaries and interfaces of the subsystems.
4. Keeps solutions as simple as possible, maintains intellectual control over system and avoids over-engineering.
5. Most important – grows and coaches other developers to become architects

If your company can afford dedicated non-programming technical person, hire an expert for complex technical areas, a coach to share best practices and experience or a technical supporter to coordinate with different external groups and help to resolve problems and make them all part of the team.

But don’t take technical decisions and concern about architecture from the development team.

The ultimate solution – every developer covers these architect responsibilities. Hire few experienced, motivated and capable developers and start project with them. Developers will constantly evolve the system based on real needs and shape effective architecture. You can trust that system will be healthy, meet business needs and bring success. Often only few important architecture decisions should be made before start of the project – trust them to the developers, who will implement the system.

So, here is the answer regarding who, when and how should make architecture decisions:

Who – every developer timely makes architecture decisions and constantly improves evolving architecture
When – mostly after start of implementation, when system becomes more complex and need design improvements to maintain system qualities, intellectual control and conceptual integrity.
How – preventing degradation and entropy by making design changes and refactoring; learning from the project dynamic, working code and business requests and adjusting a way how the development team works.

Do you still think that software projects need The Architect?

Useful resources:

Microsoft Architect Certification

Is Design Dead?, Martin Fowler

Big Ball of Mud, Brian Foote and Joseph Yoder

“I will work harder” – the horse Boxer (from George Orwell’s Animal Farm)

bichuas

The System Thinking Laws from Peter Senge’s book “The Fifth Discipline” applied to Software Development.

1. Today’s problems come from yesterday’s solutions.

We, humans, are happy when we solve problems. We often don’t think much about consequences. Surprisingly, our solutions could strike back and create new problems.

• A company decides to reward few key members of the very successful team with bonuses and promotions. The rest of the team feel unfairness and loss of motivation. Eventually tension between members is increased. The following projects are no longer successful.
• A project manager frequently asks developers to fix a new bug or work on urgent requests from customers. Developers do their best to fulfil these requests. Frequent distractions prevent them from finishing their main tasks for the iterations. Project shows only little progress.

2. The harder you push, the harder the system pushes back.

We have this stubborn reaction to push our way through when things are not working out as we want. We charge without time to stop, think and find better alternatives. Sometimes we solve problems, but often we find ourselves up to ears in the swamp of other problems.

• Managers keep pushing people to work overtime and meet deadline when a system is far from completion. The number of bugs is increasing and overall quality is rapidly dropping causing more delays. More and more effort is required to launch the software system.
• Developers heroically stretch the same architecture for the new system requirements, which don’t fit into the old rigid way. They are so busy doing it that don’t have time to stop, analyze and change approach. The system degrades.

3. Behavior grows better before it grows worse.

Short-term solutions give us a short break and temporary improvement, but don’t eliminate fundamental problems. These problems will make situation worse in the long run.

• A company gives customers hefty discounts and run expensive advertisement – many people buy the software. Customers are unhappy after purchase, because software is unusable and unreliable.

• Management promises development team big bonuses if they finish system in time. A team work hard, but soon realize that it is impossible. Developers becomes cynical and unmotivated.

4. The easy way out usually leads back in.

We learn few solutions in our life, which brought easy success earlier. We try to vigorously apply them in any situation disregarding particular context and people.

• Agile coach is forcing full Extreme Programming implementation when developers are not ready to accept some practices as pair programming or TDD. It creates stress, conflicts and allergy to any Agile approach.
• Developers apply design patterns everywhere unnecessarily complicating the system.

5. The cure can be worse than the disease.

Some familiar solutions could be even dangerous like drinking beer while programming to reduce stress for unreal deadlines.

• A company hires various contractors to work on core features, because doesn’t trust full-time developers. As a result, the system doesn’t have conceptual integrity, in-house developers don’t understand and cannot change it. Domain knowledge, interpretation and concepts are missing from the brains of company employees.
• Developers make many shortcuts, copy and paste code for similar functionality to please management and deliver first version fast. They do quick progress at the beginning, but code eventually becomes Big Ball of Mud.

6. Faster is slower.

When we feel smell of success we start advance at the full speed without much caution. However, the optimal rate of growth usually is much slower than the fastest growth possible.

• Managers add many people to already successful project. The overall progress becomes slower, because of communication overhead and loss of team coherence.
• Developers quickly add new features to the system without proper refactoring and improving existing code. System becomes difficult to understand and modify.

7. Cause and effect are not closely related in time and space.

We are good at finding causes to our problems, even if they are just symptoms and far from real root causes.

• Development team stops accepting requirement changes from customers to finish a system in time. Customers are unhappy with delivered software.
• After few incidents with a live system, management compel developers to get approval and write detailed technical specification before implementing any change in the system. Developers lose motivation for any improvements in the system and start procrastinating.

8. Small changes can produce big results-but the areas of highest leverage are often the least obvious.

Most obvious grand solutions like changing company policy, vision or tag line often don’t work. Small ordinary, but consistent changes could make a huge difference.

• Developers have everyday interactions with customer and make most decisions. As a result, customer needs are well understood, decisions are better and solutions are optimal.
• Developers build automated unit tests for each function in the system. As a result, design is flexible, people are confident, the system is fully tested after each change.

9. You can have your cake and eat it too – but not at once.

We often face rigid “either-or” choices. Sometimes they are not dilemmas if we change our perspective and rules of the system.

• Experience managers know that we cannot increase the number of features and reduce time and cost simultaneously. However, it could be possible to achieve if we just improve flow of ideas, find right people and avoid over-engineering.
• Developers believe that they should follow either Transaction Script or Domain Model architecture patterns. However high performance solution in the complex domain can combine both for the best effect.

10. Dividing an elephant in half does not produce two small elephants.

Inability to see the system as a whole could often lead to suboptimal decisions.

• A manger evaluates developers based on the number of lines of codes they produce or points they implemented during iteration. Developers produce a lot of useless code.
• Management promises testers $5 dollars for any found bug in the system. Testers are no longer interested in cooperating with developers and prevent root causes of the bugs. Good and productive relations between teams disappear.

11. There is no blame.

We like to blame and point fingers to other people or circumstances, sometimes we even believe in this. But we and cause of our problems are part of the System.

• It is Joe fault that the team didn’t release system this morning. He didn’t fix all the bugs overnight even though PM kindly gave him free pack of beer, T-shirt and pizza.
• People don’t use a company’s excellent Web 2.0 social application. Users are simply stupid and don’t appreciate hard work.

So what?

These 11 laws of The System Thinking show that all our solutions have consequences, sometimes bad and unexpected. Systems around us are what they are, and we shouldn’t blame, but learn them. To master The System Thinking and control these systems we should

1. understand what are the systems we are dealing with, either human or software.
2. consciously learn relations, cause and effect chains
3. perceive the systems as a whole and as the part of other systems

There are many challenges to the system thinking. Many we can defeat with gaining and using knowledge how systems work. But the most serious challenge is our own contradictory human nature. Our passions, emotions and instincts could easily defy the rational and systematic way of thinking. First step to master the system thinking is to learn how to cooperate with ourselves.

Question: What is your experience with the system thinking (or absence) in software development?

Only entropy comes easy. – Anton Chekhov

What are the systems in software creation?

System is a group of interacting, interrelated, or interdependent elements forming a complex whole. – The American Heritage Dictionary

There are some powerful forces in our Universe that lead systems to more complex organization – from elementary particles to molecules, from space dust to stars and planets, from chemical elements to biological organisms and from early tribes to advanced civilization. Even more powerful forces lead systems to destruction or degradation back to primitive forms or complete mess.

System evolution is the constant fight between order (or more complex organization) and entropy (or degradation) of the systems.

System forces are working on 2 levels of the software creation:

• People, how they organized and the way they build software
• The software system itself

Both these levels intervene and affect each other to deliver desired result – a working software system. These levels are different – people systems are the product of biological, social and economic forces. Software systems are the product of technology, mental concepts and theories. But both of them share common challenges related to the system forces.

What are the system forces and challenges?

Structural

• Centralization vs. Self-Organization - which systems are more effective – directed (e.g. traditional corporation) or self-organized (e.g. classical market). Compare large centralized development of Windows with the effort of independent developers to build open-source Linux.
• Complexity vs. Simplicity – what should be a degree of internal complexity which enables complex behavior while maintaining consistency and integrity in the system. Compare collaboration software: simplicity and elegance of Basecamp and complexity and power of Sharepoint.
• Abstraction vs. Concrete - what should be the levels of abstraction in the system to allow most effective perspectives on structure, interactions and relations. Compare multi-layer perspective on enterprise Java system with the flat PHP solution containing all the logic in template files.
• Diversity vs. Similarity - how different should be elements to allow rich behavior while avoiding conflicts and maintaining the same direction and progress. Compare diverse team with different skills, backgrounds and opinions with single-minded and compliant team.

Behavioral

• Adaptation vs. Stability – how to balance adaptation to the new unexpected changes and stability to sustain disturbances and keep working what is good. Compare changing existing system for the new requirements with the need to stabilize it for reliability and safety.
• Planning vs. Emergence – what strategy provides better results – relying on planned system changes or anticipating new properties and behavior emerged from unpredicted interactions within the system and with external environment. Compare Waterfall and Agile processes.
• Feedback vs. Isolation - what should be a level of feedback to enable system self-correction and adaptation and avoid information overflow, noise and destabilization. Compare hour release cycles of Flickr with years of Windows Vista development.
• Expansion vs. Consolidation – what is the best way to combine two main types of the system evolution – adding new functions and improving internal organization. Compare adding new features to refactoring.

And the main fight is between Order vs. Entropy - how to avoid system degradation and maintain order. Compare well organized and messy systems (Ball of Mud).

How can we resolve and tame System Forces?

1. Favor what supports business needs and goals:
   • for life critical systems (medical, space shuttle): Stability, Planning, Centralization
   • for competitive and rapidly changing systems (most businesses and web) : Adaptation, Self-Organization, Emergence.
2. Favor what is most effective for the human way of thinking and working: Simplicity, Self-Organization, Abstraction (for many people Concrete approach is more effective).
3. Favor what keeps more options open for evolution and better fit for changes in environment: Adaptation, Feedback, Diversity.
4. Consider positioning and life stage of the system
   • expanding into the new markets: Adaptation, Expansion, Feedback
   • retaining and satisfying large user base: Stability, Consolidation, Centralization

Taming complex systems is difficult – it needs constant skillful effort to avoid Entropy and keep Order. Economic and Human forces add more challenges and nuances.

The art and mystery of the software creation is how the system of interacting imperfect people can build useful, reliable and complex software systems in time and within budget.

Question: Is it possible that computer systems will evolve in the future without human intervention?