A really great talent finds its happiness in execution. – Johann Wolfgang von Goethe

source

Qualities of well composed code:

1. Quick discovery and understanding of programming logic and components
2. Clear organization (for human brains)
3. Ease of reuse, modification and evolution
4. Close connection between customer ideas and system implementation

Style Components:

• Intention - understand your task and how to get it done
• Approach - basic principles of writing code
• Composition - organization of code
• Expression - expressing ideas in code
• Object Oriented Pragmatic Style

1. Design for customer problems
Make the customer problem space a core for the software system design. Make every line of code accountable for solving customer needs. It is true that any software system requires a lot of plumbing beyond pure implementation of customer needs. However, technology concerns shouldn’t prevail. High-level languages and modern programming platforms give us a power to concentrate on customer problems more than on solving technical problems.

Code could be divided into two categories:

• customer oriented – solves customer problems directly, e.g. implementing domain logic, UI interactions and display of information;
• system oriented – solves technical and application specific problems – data access, event handling, interfaces between subsystem, utilities and other code required for using libraries, systems and platforms.

Customer-oriented code brings most value and should be primary design concern. System-oriented code puts in place infrastructure and support for running customer oriented code.

Therefore, start with design of customer-oriented code and support it with minimal system-oriented code. This doesn’t mean that system-oriented code is not important. Complex system environments, sophisticated user interfaces and challenging non-functional requirements (performance, availability, reliability, etc.) could demand major system-oriented development effort. But still customers needs should pull and direct this effort and the whole implementation.
There are many ways to make technical problems dominant over customer problems:

1. Making predetermined technical decisions (databases, programming platforms, messaging systems, expensive middle tier etc.) without considering simpler alternatives.
2. Coming up with complex up-front design instead of evolving a simple design into optimal design.
3. Starting programming around interesting technical problems forgetting what is important for a customer.
4. Building frameworks and system-oriented code before writing code for immediate customer problems.

For a example, some teams could start with this picture in mind stretching customer requirements to fit it (even for relatively simple applications).

source
2. Organize and evolve code around domain concepts and customer ideas
Make system design reflecting customer domain and problem space. Keep customer ideas and system implementation connected and synchronized all the time. This is as important as refactoring practice for effective creation of the software system.
Domain-driven design is an excellent way to build software systems. It puts emphasis on ubiquitous language, distilled domain knowledge and shared domain models that drive software development.

3. Think as a customer
What we see depends mainly on what we look for – John Lubbock
A programmer often considers programming as an opportunity for solving interesting technical problems. Often a programmer even doesn’t understand what customers really want. There are 2 consequences. First, the programmer is not interested in simpler solutions focused on the customer problem. Second, the programmer is missing coherent view on the customer problems and purpose of the software system. Over-engineered, complex and misaligned code is one of the most serious problems in software development. An army of business analysts, stressed project managers and smart customer proxies will not force indifferent and ignorant programmers to write code in the best interest of customer.
The programmer should constantly ask himself: “Will this solution work for my customer?”. To give correct answer on this simple question the programmer should understand a lot – customer language, needs, problems, a big picture and business purpose.

Certainly, it is difficult to think as a customer if you are at the bottom of software creation chain:

• separated from a customer by project and product managers, architects, designers, consultants, experts and panels of stakeholders
• pressed by the heavy process, rigid organization structure and inability to make decisions
• fed with over-processed, distorted and disjointed doses of information coming down through long chain

Beautiful software ideas can turn into monster applications – unusable and disconnected from real needs. Did you feel (as a user) frustrated by complex confusing and irrational program screens and logic? Yes, we, programmers, can easily produce bad stuff if we cannot think as people who need our systems.

4. Sketch programming ideas with unit tests.
Write unit tests first. Any programmer should understand what he is trying to solve with the new code and how it will be used – possible input, expected output and public interface. A programmer should think about behavior in isolation, under border conditions or within context of existing code. An unit test is an excellent place to sketch nonexistent code, play with it without compilation and see if it is convenient and makes sense. Once the programmer likes sketched ideas in unit tests, he just need to build simple implementation to satisfy these unit tests. The process of writing unit tests focuses a programmer on solving customer problems, understanding code intent and making optimal design decisions. As an additional side effect, the programmer receives comprehensive suite of automated regression tests and executable specifications.

For a example,

CustomerDB.Save(TEST_USER, TEST_PWD)
Customer customer = CustomerDB.Login(TEST_USER, TEST_PWD);
Assert.IsNotNull(customer, "customer should be logged in");
customer.Purchase(TEST_PRODUCT);
List<Order> orders = OrderDB.LoadFor(customer) or maybe... customer.Orders //what is better?
Assert.AreEqual(1, orders.Count, "count");

There are many design decision are made in this unit test without writing any production code. And now programmer can concentrate on implemention without throes of creation.

5. Eliminate duplication
Simplification is ultimate sophistication – da Vinci
Elimination of duplication is the moving force of evolutionary design. This is straightforward and powerful method to grow an optimal system. Duplication in the code tells that you need better design. Many interesting design ideas are born from the simple need to eliminate duplication in the growing system. These ideas are based on deeper understanding and gained experience as oppose to speculative up-front design ideas.
The main goals of removing duplication are

• reduce code
• simplify solution
• make the system more manageable for programmers’ minds.

Some sources of duplication:

• copy and paste – acceptable only temporary
• similar logic different in details – generalization is required
• another implementation for the same problem – better team communication and reuse are required

Master the core skill of good evolutionary designers – how to detect and eliminate duplication.
The knowledge of design patterns is the most useful on this step. The initial simple solution rarely requires design patterns. But as a system grows, design patterns can effectively and elegantly solve new complex design problems.
Example of elimination of high level duplication with Template method design pattern
This code was used in many data access classes almost the same way, but with different inner steps

        using (SqlConnection conn = NewSqlConnection())

        {

            SqlCommand cmd = conn.CreateCommand();

            cmd.CommandType = CommandType.StoredProcedure;

            cmd.CommandText = "SearchCustomers";

            cmd.Parameters.Add(new SqlParameter("@city", city));

            ... //long list of parameters

            conn.Open();

            try

            {

                using (SqlDataReader reader = cmd.ExecuteReader())

                {

                    foreach (DbDataRecord record in reader)

                        customers.Add(new Customer(record["id"], record["name"]))

                }

            }

            catch (Exception e)

            {

                //error handling logic here

            }

        }

The common logic could be moved into one template method with placeholders for different steps:

List<Customer> customers =  LoadRecordsFromSP<Customer>("SearchCustomers",

    (SqlCommand cmd) => {

         cmd.Parameters.Add(new SqlParameter("@city", city));

         ...     //long list of parameters

    },

    (DbDataRecord record) => { new Customer(record["id"], record["name"]); }

);

6. Reduce code smells
Write code that doesn’t smell. Train your sense of smell for code and design problems. They will provide hints where code is probably bad.
These are common bad smells from a long list

• Duplicated code
• Large method
• Large class
• Long parameter list
• Dead Code
• Over-generalized code
• Lazy class

7. Use design patterns and abstractions to make code simpler
Learn design patterns. At some level of complexity they will provide excellent solutions and simplify design.
However, wrong abstractions and over used design patterns will make code more complex and less manageable. So, apply them carefully.

8. Decouple and isolate components
Only talk to your immediate friends; Don’t talk to strangers – Law of Demeter.
Reduce connections between programming units – subsystems, modules and objects – as much as possible. More relations between components make software system more complex and rigid. Decoupled and isolated system components have significant benefits:

• better understanding – easier to keep in memory and make sense for less states and relations
• increased reuse – components with minimal well-defined public interface and relations less dependent on context and can be used in many scenarios
• decreased breakdowns – caused by unpredictable run-time combination of relations
• effective testing – highly connected system require much more testing beyond isolated testing of components

9. Keep related code together
Put related code to the same package. It will enhance

• discovery - easier to find and reuse
• design - allows isolation, minimal interactions and consistent structure for related components
• maintenance and testing - reduce effect of new changes to fewer packages and system components.

There are two package approaches:

• Feature – code related to the same problem placed together
• Application layers – code for similar programming concepts

Benefits of packaging:

• Feature: related code in one place, changes together and form self-sufficient granule for reuse and release
• Application: separation of concerns, logical layering and reduced dependencies,  consistent global structure

Composing the program – Putting Things Together

1. Understand what customer wants and form intent.
2. Translate customer ideas into programming concepts (technology, platform and language, UI, libraries, components)
3. Make them simple, minimal and clear with pragmatic approach.
4. Organize code around domain concepts and customer needs; use domain-driven design.
5. Describe UI ideas with paper sketches
6. Specify program ideas with automated unit tests.
7. Merge UI and program ideas
8. Package related ideas together, isolate subsystems and components
9. Express ideas in code and UI screens
10. Remove duplication and bad code smells; improve design; consider using design patterns
11. Evolve and refactor common code into framework; emerge layers and abstractions for similar concepts

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. – Antoine De Saint Exupery

The approach to programming is concerned with finding the best ways to translate programmer’s intention into the good system design and code.

The programming is communication. The programmer continuously add, change and refine ideas in the code. Source code has two important goals: tell a computer what to do and tell people what the computer should do. The program code is the only true medium for storing and communicating ideas about the software system behavior. Quality of the ideas expression in the code directly affects overall quality of the system.

So, what are characteristics of the good code?

1. clear - easier to work with ideas;
2. minimal - less effort to understand and change ideas;
3. testable - easier to validate ideas.

These are 6 top reasons for bad design and code:

1. lack of expertise
2. unrestrained technical curiosity and creativity
3. missing big picture: system purpose and customer goals
4. blindly following popular methods and over-using technology
5. sloppiness; lack of attention to details
6. over-complicating design to have more work or increase job security

The programmer can write better code (and avoid most of these problems) by improving programming style and approach.

Style Components:

• Intention - understand your task and how to get it done
• Approach - basic principles of writing code
• Composition - organization of code
• Expression - expressing ideas in code
• Object Oriented Pragmatic Style

Approach

Two most important rules are

• work from the simplest design
• revise and refactor after each task

1. Work from the simplest design
The simplest design that solves business needs is the best design. The simplest design is the product of discipline and mastery. The simplest design requires creative thinking – it is easier to follow traditional ways than to come up the simple, but clear and powerful solution.

Agile development has famous principle DTSTTCPW (Do The Simplest Thing That Could Possibly Work) best explained by Ron Jeffries:

We’re saying consider all solutions to your task that could possibly work. Implement the simplest solution. Refactor from there if and when needed…  And we’re saying that when you do enhance the simple solution (and you generally will), you will always wind up with a system that is just right for what it does so far. And we’re saying that that is just where you want to be. Everything just right, nothing added that isn’t needed.

For example, if you love design patterns, MVC and object oriented programming, you may end up with design below for finding orders by customer’s postal code.

I love these patterns too, but I would still recommend to start with the simpler design.

The simplest design is the most effective way to deliver business value. You spend less time, build more reliable, easier to understand and change solution. You save energy for other tasks and reduce load on your hard-working brains.

Why do people not go for the simplest solution all the time? There are three main reasons:

• unintentional - the programmer didn’t find simpler solution (lack of experience, pressure and no time to think, relying on traditional ways, compliance with architect’s grand design and others)
• intentional - the programmer wants to over-complicate solution for personal benefits (play with interesting technology, build up resume or increase scope of paid work)
• foretelling - the programmer bets on the future advantage of more complex design

The last reason is the most compelling for many programmers. What if your experience and knowledge tells you that you will need this more advanced design and additional code?
Lets do cost-benefit analysis

What are the chances of mismatch with our initial expectations for a non-trivial project? Big. Mismatch with expectations happens when we learn more from technology, domain and customer feedback while programming. Mismatch happens when we understand the problem deeper and come up with the better solutions. Mismatch happens when the new requirements are coming, priorities are altered and the whole world around is changing (e.g. new company opportunities or credit crunch and economic downturn).

As we can see the best approach is the simplest design that requires little refactoring in the future and doesn’t prohibit anticipated design evolution. Therefore, use you experience and intuition to predict direction of the design evolution. Use this knowledge not for making design more complex to accommodate this prediction, but for ensuring that the simplest solution is not stupid, prohibitive for evolution and will survive coming changes.

However, the level of expertise has significant impact on the choice. The novice programmer should use standard well-described solutions and learn how to simplify them later. The expert programmer should be more comfortable to design simple, sound and easy to evolve solutions from the start.

2. Revise and refactor
Focus on the simplest local design and code for individual features without revisions and design improvements could sometimes create global mess and painful complexity .

For example, it is bad, if after adding several features, your OrderDB contains many methods like LoadForPostalCode, LoadForPostalCodeAndStatus, LoadShippedOrders and web pages are overloaded with domain logic. This kind of code calls for refactoring.

Revise and refactor after completing any programmer tasks.
You should ask yourself 4 questions:

• Can I reduce code?
• Can I make code clearer?
• Can I make code more testable?
• Can I make the whole system simpler?

Refactor, if you answer yes to any of these questions. Learn code smells to identify concrete places that need refactoring. Learn refactoring techniques to improve code.
Main methods:

• remove duplication and increase code reuse

• create abstractions to generalize individual cases
• use design patterns to simplify design

Refactoring is one the most important practice for productive software creation. Refactoring allows to keep the system in the good shape all the time – simple, well designed, ready to change. Implementing the simplest solutions without refactoring and evolutionary approach is risky.

Other rules should enhance your approach further.

3. Discover Form and Function in parallel.
Form follows Function is a traditional principle in design. In other words, customers requirements dictate user interface. However, discovery of complex interactions could affect what the customer wants and change understanding of needs. Sometimes starting with UI paper sketches and discussing them with customers could expose missing details, gaps in understanding and suggest better ways to satisfy customer needs.  Therefore, work on discovery of Form and Function of the system in parallel.

4. Avoid using new tools
New Systems Mean New Problems – The Fundamental Theorem of Systemantics
Do not use new languages, tools and libraries unless existing are bad and you can tolerate high risk and delays.
Every new programming tool claim to make you more productive and powerful. Few of them really do this. But , it is guaranteed, that every new tool takes time for learning and brings new problems.
I know that it is hard to continue using old tools for a long time and stay competitive in fast moving software development world. But try to limit impact on customer. It is not fair to make them victims of your learning and experiments. Create separate experimental prototypes and greenfield projects to evaluate new tools and approaches. Prefer to be honest and transparent with your client about risks of new technologies, tools and approaches.

5. Use creative ideas sparingly
Creative ideas can bring excellent solutions. However, any creative idea adds element of unexpectedness and departure from established ways to solve similar problems. Programmers will spend more time to understand creative ideas and will have challenges to support them. Therefore, use creative ideas when they bring significant advantages (especially to simplify solution) and avoid them in trivial situations. Don’t worry, you will always find where to apply your creative energy if you are productive, build good solutions and have more and more customers.

6. Prefer standard solutions to the non-standard
This rule compliments the previous rule. Standard solutions are based on experience of many people. Maybe they are not the best in your particular case, but they often are safe bet if these problems are new for you. You can find many examples, quickly learn how to apply them and understand consequences beforehand. New creative solutions will make life more interesting in expense of productivity and certainty in the final results. Therefore, prefer standard solutions to reinventing the wheel again. Don’t forget to learn how to make them simpler.

7. Separate production code and experiments
Restrain technical curiosity when writing production code. Many programmers find the joy of programming in solving complex problems. It is so tempting to solve difficult technical puzzles and try out new cool techniques without need. However, the pragmatic programmer solves complex customer problems with simple solutions, instead of creating complex solutions for the simple problems.
Think first about customer requirements and simple design. Experiment and look for the new solutions only if you cannot find simple solutions with familiar methods. Keep your production and experimental code separately. Adopt ideas from experiments without unnecessary complication of the solution.

8. Do not overwrite (or over-eingeneer)
Write minimal code relevant to present customer requests. Surprisingly, complex over-engineered solutions are easier to create – just follow recent trends and try to anticipate everything that came to mind. The simplest and clear solutions require extra effort.
Over-engineering harms the software system, takes away precious time and project money. Avoid over-engineering if you are motivated to build good system and satisfy your customer.

9. Write own code for the core and use external components for supporting functionality.
Concentrate your effort on writing code for the core functions specific to customer domain or code that is bringing competitive advantage. Don’t waste too much effort on secondary supportive functions. For example, there are many good enough libraries, for solving standard dynamic web user interface needs – prototype, jQuery, Microsoft AJAX, Yahoo user interface library and others – but many programmers are still building their own solutions. I like to have full control over my software systems and write as much as possible components, but one cannot embrace unembraceable (Kozma Prutkov). You have to trust some parts of the system to other libraries that will save time and do better job (but bring new problems).

10. Do not write code for future.
The dirty trick of predicting the future is that future is unpredictable. Everything else is fine. Written for the future code adds unnecessary complexity. This code is useless for the present problems. Avoid writing code for the future and strive for the simplest solutions for the present needs.

11. Write code for other programmers
Can other programmers understand your code? Do you have clear names, consistent formatting and self-explaining code? Writing code for others will make code better for you.

12. Switch between big picture and details
Good programmers can focus on the big picture and still pay attention to details. Clear intention and forward thinking are as important as reliable implementation.

13. Pair program for complex problems
Pair programming brings significant benefits:

• better ideas – two brains solving the problems
• better quality – two pair of eyes validating code
• better knowledge – two programmers understand implementation and ideas behind
• better productivity – problems are solved faster
• more enjoyment – people like to communicate and work together

I’m still not sure about serious benefits of pair programming for simple and routine programming tasks. But many Agile teams pair program for all production code and like results.

Putting It All Together

Inspiring reference: The Elements of Style, W. Strunk Jr. and E.B. White

From the first sight biological evolution looks too unpredictable to have any value for the constrained software development. We just don’t have time, money and resources for these wild experiment, unlimited trials and errors. It really seems that Nature could learn from us how to make things fast and effectively.

However, there are some principles that helped evolution to come up with amazing and efficient designs that made life flourishing on the Earth. This post will explore what software development could learn from biological evolution. See my previous post for the review of evolution concepts and mechanisms and how they could be applied to software development.

1. Evolution works

The evidence is richness of our flora and fauna which adapted to almost all possible conditions on our planet. Organisms live in polar ice, in desert rocks, in the bottom of Pacific ocean Mariana Trench (11 km deep) and even on the Moon.

Evolutionary approach is one of the most powerful and proven methods known to us for designing effective systems under complex uncertain conditions. There are no reasons why software teams should ignore this approach – they indeed often deal with complex uncertain requirements and environment.

There are 2 areas where evolutionary approach could be used:

1. project management – enhance the way people work and interact for building a system
2. design and build – the growth of this software system itself.
   

Simple Steps:

1. Generate new and modify existing ideas (increase variations).
2. Select potentially best ideas for building software better in a specific environment (form project genotype)
   
3. Make this generation of ideas work for some time (interact with environment).
   
4. Learn from results – keep useful and dump harmful ideas (natural selection – improve fitness).
   
5. Go to the first step and produce the new generation of ideas
   

We can make this process more intelligent and less random and still learn a lot from biological evolution.

2. Increase beneficial variations or how to generate good ideas.

Organisms will not evolve without getting new variations of traits. Some new variations in a gene pool could lead to better survival and fitness, but most of them are harmful. Software development genotype is a pool of ideas applied for building a specific software system.

How could we generate many good ideas and avoid harmful?

Mechanisms of idea’s evolution

• Mutations - continuous invention and modification of the new approaches based on individual creative insights and team brainstorming
  
• Reshuffling - recombination of existing ideas
• Gene flow – outside expert opinions and borrowed ideas from other disciplines (engineering, art, architecture, manufacturing, etc.) and study of complex systems (physical, biological, markets, societies, etc.)
  

Prerequisites for generating many quality ideas:

Powerful software teams with diverse range of

• skills / specialization – they cover broad knowledge areas of software development: customer domain and needs, project management, UI design, development, testing
  
• expertise / experience – combination of different approaches, insights and methods of experienced professionals with fresh ideas of newcomers.
  
• personalities – various traits that complement the whole team: creativity, discipline, pragmatism, attention to details , imagination, focus and leadership.

Decentralized control / self-organization
Centralized control reduces ability to come up with new ideas, which are not compliant with imposed top views. People from bottom, who implement the system, could generate better ideas, because they know environment deeply, experience it closely and receive immediate feedback from it. Independent people, who have control over project decisions and responsible for results will strive to make the best decisions.

Idea incubator

• Free flow of ideas, sharing of knowledge and expertise.
  
• Encouragement for innovative and creative thoughts and tolerance to failures.
• Opportunity for every team member to express ideas, be heard and respected.
  

Good ideas in a software project are necessary foundation of success, lack of them is disaster. Do whatever is possible to generate many good ideas.

3. Improve Fitness – keep useful and dump harmful ideas

Genes passed to the next generation are result of two factors: Fitness (natural selection) and Chance (genetic drift).

How could we improve blind process of selecting best ideas and avoid high failure rate?

Establish clear fitness criteria
Every organism have clear short-term goals – survive and reproduce to accomplish the long term goal – passing and increasing share of their genes in the next generations.
Software projects should have clear fitness criteria, which aligns team members, guide their decisions and actions. Criteria should be relevant to the most important goals of the project. It could reflect the purpose of the system, time to market, quality or cost. It should be clear, short and really important for survival and success.

Maintain selection pressure and increase role of Fitness vs. Chance
Low natural selection leads to more complex solutions. Most mutations which don’t increase fitness will be eliminated in a large population with fierce competition and strong selection pressure. This process could lead to a greater simplicity keeping only useful for fitness traits. In populations without much selection pressure, genetic drift increase chances of new genes survival (even disadvantageous). So population could have increased genes complexity without much advantage for fitness.

Avoid genetic drift
Genetic drift produces random changes in the frequency of traits in a population, which results from the role chance plays in whether a given individual will survive and reproduce. Chance play enormous role in biological evolution (maybe even more important than natural selection) and a great source of inefficiency. We can do better in software development. Transparent and independent decision making, self organization and power of diverse opinions will substantially decrease the role of the Blind Chance.

Therefore, a team with many good ideas shaped by selection pressure and fitness criteria will produce the best results.

4. Pragmatic implementation

Good enough working solutions
Animals shouldn’t be perfectly adapted, they just need to be in par with their competitors. Natural selection requires only requires something to work, not perfectly but work. Evolution is more likely reshape existing structures than invent new ones.
The classic example is the panda’s thumb, which it uses to grasp bamboo. The panda’s true thumb is committed to another role. The lobed fins of early fish have turned into structures as diverse as wings, fins, hoofs and hands.

In software we could be satisfied with solutions which are not perfect at the beginning. We can live with them knowing that evolution forces and pressure will shape ideas behind and we will make these solutions better with more experience.

Practical and immediate purpose

Every new feature in the system should have some immediate purpose – evolution invented only something that has purpose.

Exaptation – structures and behaviors that evolved for one purpose but take on a wholly new one, while remaining useful at every intermediate stage. It means that some features cannot evolve if they are not completely useful immediately. Two way radio could be useful for animal communication, but it didn’t evolve, because half-way radio is useless – it doesn’t tell anything useful for animals about their environment. Visible light is different – any small improvement in vision can provide big advantage in many environments.
There is an interesting question: could we miss great design ideas if we only develop for immediate purpose? There are many useful inventions that came as a result of fundamental science curiosity without much practical need at the beginning. However, I’m inclined to believe that software development is empirical discipline and more effective with developing for immediate purpose only.

Repurposing
Repurposing a structure does not have to involve the loss of the original structure. Reptilian jaw bones turned into mammalian ear bones, without the loss of the jaw. The neural circuitry that allows us to make fine limb movements may have been adapted to produce speech as well.

Sometimes just one aspect of a feature can be co-opted for another use. The first hard mineralised structures to evolve in our ancestors were the teeth of early fishes known as conodonts. Once the ability to form hard hydroxyapatite had evolved, it could be exploited elsewhere in the body and may have been the basis of the bony skeletons of all vertebrates.

Repurposing is an interesting lesson for the software development – successful reuse of existing structures for the new functions. At the first glance, it goes against main principles of good software design – single responsibility, isolation and decoupling of components. Biological organisms don’t have much choice with mutations- which could add only small changes. But efficient and economical designs of many organisms suggest that repurposing could be a good strategy. We could maximize reuse of the components for different purposes while keeping smaller inner parts isolated. Relentless refactoring, removal of duplication and creating abstractions are the main methods. The ultimate condition is good understanding and mental control over the system.

No refactoring (or elimination of unused traits)
Lack of refactoring seems as a serious flaw of biological evolution. New organisms repurpose what exists and build new structures on top of old without cleaning and eliminating unnecessary stuff. DNA has a lot of junk. These structures are not needed

• wings of ostriches
• appendix
  
• male nipples

Software team can use refactoring as an essential part of effective and pragmatic implementation and improvement of evolutionary approach.

5. Making impossible possible with small continuous changes.

Small changes accumulated over time cause substantial changes in organisms. Continuous improvement of the genotype and small frequent changes work for all living organisms in this unpredictable and very complex world. They lead to highly fit, adapted and efficient solutions.

By weeding out harmful mutations and assembling beneficial ones, natural selection acts like an “improbability drive” that can, given enough time, produce results that appear completely impossible at first glance.

One of the most important lessons for Software Development is possibility to arrive with great solutions without planning and predicting them in advance. It is one of the best strategies in the complex, uncertain and changing environment.

Agile approach is successful because it strikes the balance between adaptation to changing environment and predictability required for business.

Principles:

• Be open and ready to changes
• Frequently get feedback, change and adjust in small steps
• Continuously learn and reflect what works and what doesn’t
• Often evaluate if your project and systems are fit for their purpose and environment.

Summary

Evolutionary approach works if software team

• can generate many good ideas
  
• can select most useful ideas how to build the system
• works under healthy pressure to meet fitness criteria
• pragmatic in implementation
• learn from results, continuously adapt to needs and environment
  

Resources:
Comparing Intelligent Software Evolution to Chaotic Biological Evolution
Evolution: 24 myths and misconceptions, Newscientist (requires subscription)
Evolution, Wikipedia

I can’t wait to share this simple secret with you right now.

The Secret: Effective Software Systems are the systems that easy to understand and operate with human brains.

Programmers are more productive with effective software systems. Programmers can better learn and grow these system. Programmers have less problems, work faster and make better decision with them.

Now, you can avoid spending time reading this post if you already know this secret and you know how to avoid building the software system that:

• almost impossible to understand in reasonable time
  
• has confusing and convoluted swamp of logic and structure
• scary to change as nobody has any clue what will be broken, but sure that it will be broken
  

If you are still interested, lets find out what makes software systems effective.

Software Development is a pure mental endeavor (except typing on keyboard) that includes 3 main activities:

• Understand - learn and know system concepts and implementation
• Evolve - build, modify and support growth of the system ideas in the code
  
• Share - communicate and exchange ideas about the system

Programmers should care about 7 areas to make the system better suited for our brains:

1. Knowledge Creation and Retention – parsing, memorization and comprehension of the system ideas
   
2. System Organization – elements, relations and structure in the system
   
3. Sustaining Emerging Order – support evolution of the system and gain control over chaos
   
4. Minimize Noise and Purify – avoid adding unnecessary stuff to the system
   
5. System Discovery and Learning – making sense of the system
   
6. Mental Models – our internal explanations for how things are working in the real system
   
7. Shared Knowledge – ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge.

1. Knowledge Creation and Retention

Information is not knowledge. – Albert Einstein

Programmers should keep as less as possible things in the mind when work with the system. Less information, simpler and more logical ideas behind the code cause less brain damage and more understanding.
Main principles:

• Chunking – combining many units of information into a limited number of units and chunks, so that information is easier to process and remember. Logical groups (subsystems -> modules -> classes) should contain only few elements. These groups should emerge with growth of the system functionality and complexity. They should replace large clusters of difficult to remember similar and unstructured elements.
  

• Abstraction - generalize, find common meaning and remove redundancy in the similar concepts across the system. You can substantially reduce number of ideas required for understanding the system.
• Simplicity - simpler ideas make understanding (and programmers life) much easier.
• Isolation - reduce relations and dependencies between elements. These relations add significant complexity to the system and therefore effort to understand and remember them. The volume of knowledge about relations could easily exceed the knowledge about individual elements.

Examples of useful tools for supporting knowledge formation:

• Unit tests – capture knowledge about the system correct behavior. They evolve with the system and become a parallel verification system. Good unit tests reduce our effort and limit necessary for understanding scope as they focus on the system behavior in isolated components and functions.
  
• Visuals - take advantage of diagrams, mind maps and pictures to simplify, distill and integrate important knowledge and ideas about the system
  
• Storytelling - create imagery, emotions and understanding of the system behavior through stories (our episodic memory is very powerful). For example, create narratives how users accomplish tasks with your system. Or share dramatic stories how programmers doomed the system with ignoring important practices.
  

2. System Organization

Be regular and orderly in your system organization, so that you may be violent and original in your software solutions. (paraphrasing Flaubert)

Human programmers shouldn’t have hard time to memorize and comprehend a software system. Computers don’t care much about the organization as long as the logic is correct, but we, human programmers, do. Clear, logical and consistent organization make us much more productive.
Main principles:

• Modularity - managing system complexity by dividing large subsystems into multiple, smaller self-contained systems
• Layering - process of organizing information into related groupings in order to manage complexity and reinforce relationships in the information
• Hierarchy - hierarchical organization is the simplest structure for visualizing and understanding complexity
  

3. Sustaining Emerging Order

Evolution is not a force but a process. Not a cause but a law. – John Morley

The main challenge for effective system organization is the fact that a system is constantly changing during active development. Good organization today doesn’t mean good organization tomorrow with many new added features. New customer needs, contribution of fellow developers and growing complexity push a software system to chaos.
Therefore, we have to keep organization optimal for today needs and still ready for tomorrow changes with

• Refactoring – constant effort to improve the system internal structure and making it effective for recent changes to prevent system degradation and rule of chaos
• Simple principles and rules (that everybody follows) help to keep consistency and integrity of the system in time of intensive growth and modifications.
• Small independent, single-purpose components – enable rich behavior with minimum number of elements, which could participate in various and often unforeseen scenarios. Bigger components are less reusable, more specialized and lead to more code, complexity and duplication.
• Building adaptive system using agile approach

4. Minimize Noise and Purify

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery

Complexity is one of the biggest problem for the programmers brains. While good organization and representation of the system ideas help to fight it, the ultimate challenge is to avoid adding unnecessary stuff to the system.

Principals for Complexity Reduction:

• Increase Signal-to-Noise Ratio – ratio of relevant to irrelevant information in the system. Minimize noise by removing unnecessary elements and minimize the expression of necessary elements.

• Follow 80/20 Rule (Pareto’s Principle ) – a high percentage of effects in any large system are caused by a low percentage of variables. In other words, 80% of requested functionality will be used rarely or not used at all, while substantially increasing system complexity.
  
• Consider Good Enough Alternatives – don’t dash against the rock implementing exactly what customer asks. Slight modifications and alternatives could often cover need with less effort and complexity.
  

5. System Discovery and Learning

The real voyage of discovery consists not in seeking new lands, but in seeing with new eyes. – Marcel Proust

Learning and navigating the complex software system is a difficult task. Usually developers read code, documentation and debug the system. These methods take time and often are not efficient.

A better way to discover the system and make it discoverable with:

• Entry Points – points of attentional entry into the system. There are points in the system that doesn’t require much knowledge about the rest of the system. They could be starting points to understand other elements. For a example, UI screens or the system input / output. Other parts could start making sense and uncover their purpose in relation to these entry points. Discover these points and help newcomers to find them.
• Progressive Disclosure – a strategy for managing information complexity in which only necessary information is visible at any given time. Good design and coding style allow understanding of intent and logic of the studied system area without deeply diving into boundless waters of implementation details.

Code example:
Criteria criteria = Criteria.AddPriceRange(10, 100).AddDateRange(Period.LastMonth);
IList orders = Database.Orders.LoadBy(criteria);
decimal totalTaxes = orders.Sum(order => order.Tax)

• Recognition over Recall – memory for recognizing things is better than memory for recalling things. (e.g. multiple choice questions with possible answers vs. fill-in-the-blank questions). Minimize the need to recall information from memory whenever possible. Use tools to make available options clearly visible. For example, Visual Studio Intellisense and Resharper remove need in memorizing members, purpose and syntax of .Net library classes and methods. Build tools or extend a development environment to construct code without memorizing all details.
  
• Inverted pyramid – method of presentation in which information is presented in descending order of importance. Discover first what is most important and make it easily accessible following by less and less important information.
• Performance load – the greater the effort to accomplish a task, the less likely the task will be accomplished successfully. Most important tasks should be easy to accomplish and most valuable information should be easiest to access (including direct access to the people who has the best knowledge).
  
• Picture superiority effect – pictures are remembered better than words. Use pictures and words together to reinforce the learning.
• Associations with familiar – link new information to already familiar concepts. Build on this knowledge by comparing and contrasting to reach deep understanding.
  
• Examples – learning examples is always one of the best method to understand how to use the system (including best practices, how-to, and tutorials).
  

6. Mental Models

A mental model is an explanation in someone’s thought process for how something works in the real world. – Wikipedia

People understand systems and environments and interact with them by comparing the outcomes of their mental models with the system and real-world domain concepts. Good mental models bring better understanding and optimal decision making.

Schemata – networks of connected ideas or relationships that help to organize experience and information into a meaningful system. Conversation between individuals is most effective if both share common schemata.
Examples:

• Design and architecture patterns – general reusable solutions to a commonly occurring problems in software design.
• System metaphors – a simple story of how the system works. A system metaphor provides common vision and shared vocabulary.
  
• Conventions – standard and well known approaches in architecture, design, code and user interface. They make intuitive and common sense and save mental energy for present and future developers. Custom and novel approaches should be used only if they have serious advantage.
  

7. Shared Knowledge

I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant. – Robert McCloskey

The system is rarely envisioned, built and used by the same person. A successful system is a result of people interactions – intensive ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge. It is not easy for people to properly translate, communicate and understand each other ideas. Complex knowledge, different background and experience increase the gap that people should jump to completely understand each other.
Main principles:

• Ubiquitous language – the shared language that people with different perspectives use to exchange ideas about the system. The same meaning of the words is very important for understanding, but very difficult to achieve.
  
• Clear goals and intention – help to align people thinking and actions, and avoid wasteful effort caused by misunderstanding or confusion.
  
• Express business domain concepts in the code – represent business concepts in the code. Developers will better understand domain, synchronize implementation with customer ideas and align the system with relevant business concepts. As a result, the system can effectively grow with the refinement and expansion of business needs.
  
• Code readability – good naming and readable code enables better understanding by fellow programmers, reduce chances of errors and lead to correct future modifications.
  
• Shared mental models – frequent face-to-face discussions, pair programming and direct conversations with customers bring shared understanding, effective exchange of knowledge and most optimal solutions.
  

Evolution didn’t consider adaptation of human brains for software development. But our brains is the most important tool for programming. You cannot change brains (and human nature), so make software development compatible with them. Build effective software systems.

Resources:
The Programmer’s Brains At Work: Understanding The Software System
Universal Principles of Design, William Lidwell , Kritina Holden, Jill Butler

Can we say that clearing house from rubble and debris after home repair is a necessary waste for construction workers? Can we say that making design of the car that could be easily supported and repaired by mechanics is a necessary waste for engineers? Can we say that making ideas clear, easy to follow and understand is a necessary waste for the influential writer?

Amr Elssamadisy posted an interesting opinion on InfoQ about refactoring:

… there are two types of waste in Lean: pure waste, and necessary waste. Pure waste is one that has no value to either the team building the software or the customer using the software. Necessary waste, on the other hand, is the best way we currently know how to do a job even if it does not have customer value. Refactoring is clearly an instance of the latter.

Amr tries to make an important point – minimize refactoring that doesn’t contribute directly to customer requirements under development. It is a reasonable point, but I don’t like word the ‘waste’ and belittling of one of the most important practice of the modern developer. This post sparked a very interesting discussion around value of refactoring with many insightful comments (including Kent Beck and Tom Poppendieck).

We, software professionals, sometimes forget that source code is the most important representation of our ideas about how software should work and what it actually does. In the essence, software development is communication, processing and coding of people ideas. We substantially reduce value of our software if we leave messy, convoluted and difficult to understand software ideas, though correctly implemented. This code will waste time and effort of people who evolves and supports our software (including ourselves). And therefore, it will reduce value of the software for the customers who expects that it will be well supported, easy to change and become better over time. Refactoring is the practice that forces us to improve code and represent software ideas better.

Certainly, as with any complex human activity, we should learn when, how and why we should do refactoring. We should learn how to balance refactoring and development of the new features – software consolidation and expansion phases (read an excellent article from Brian Foote and William F. Opdyke). Refactoring should be not only targeted to reduce technical debt, but also to reduce comprehension debt – understanding, refinement and enhancement of implemented ideas. Finally, refactoring is the essential part of our professional mastery, which distinguish ace programmers from amateurs or indifferent hirelings.

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations - optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions - the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation - a software project consumes energy to keep going, mostly in the form of money :)
• Instability - software development deals with constantly changing situations
• Multiple equilibria – there are many possible satisfying solutions
• Complexity - any non-trivial software project has complexity
• Hierarchies - there are many perspectives and levels – organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Why self-organized systems are better?

1. Distributed control – “control” of the system organization is typically distributed over the whole of the system. All parts contribute evenly to the resulting arrangement. The free market is a good example. There are many independent market agents – companies and individuals. Problems with individual company or even market segment doesn’t effect economy much as there are always players who will use this opportunity and take over the vacant niche. The brain is another example. Its organization is distributed over a network of interacting neurons. Although different brain regions are specialized for different tasks, no neuron or group of neurons has overall control. This is shown by the fact that minor brain lesions because of accidents, surgery or tumors normally do not disturb overall functioning, whatever the region that is destroyed.

   Therefore, in effective software projects any skilled and experienced developer with tacit and specialized knowledge should have control over architecture decisions in his area as oppose to giving this control to few architects (or what even worse managers). Global order should emerge from these local development decisions.

2. Robustness – self-organized systems are relatively insensitive to perturbations or errors, and can restore themselves, unlike most human designed systems. One reason for this fault-tolerance is the redundant, distributed organization. Another reason for this intrinsic robustness is that self-organization thrives on randomness, fluctuations or “noise”, which could bring unexpected and better solutions. And third reason is stabilizing effect of feedback loops.
3. Feedback - the dynamics of a self-organizing system is typically non-linear, because of circular or feedback relations between the components. Positive feedback leads to an explosive growth, which ends when all components have been absorbed into the new configuration, leaving the system in a stable, negative feedback state.

   Effective feedback loops could be established only with open, adaptive and distributed approach contrary to closed, rigid and centralized traditional approach to software development.

4. Emergence - property or feature not previously observed as a functional characteristic of the system. Self organization turns a collection of interacting elements into an individual, coherent whole. Elements produce their own emergent properties and form the next level of structure in the system, which in turn form the building blocks for the next higher level of organization, with different emergent properties, and so on.

   Therefore, it is almost impossible to come up with these properties and decisions on the preliminary software design stage. The key for building great software systems is to learn how to tame and use emergence effectively instead of fighting against it.

5. Bifurcations - which of the possible configurations the system will settle in depends on a chance. A system may settle in a range of stable configurations. Since small fluctuations are amplified by positive feedback, this means that even small and impossible to observe fluctuation can lead to the to completely different project outcome.

   Therefore, even if we know business needs, environment and system specifications at the beginning of the process, the outcome is still unpredictable. It is waste of effort trying to predict and plan it – it is better to prepare for changes and adaptation.

6. Adaptation - a configuration of a system may be called “fit” if it is able to maintain or grow given the specific configuration of its environment. An “unfit” configuration is one that will disintegrate the system under the given boundary conditions. Different configurations can be compared as to their degree of fitness, or probability to survive under the given conditions imposed by the environment. Thus, adaptation can be defined as achieving a fit between the system and environment. Systems may be called adaptive if they can adjust to such changes while keeping their organization as much as possible intact.

   Considering above points, adaptation is very important for survival and success of the software project.

7. Self Regulation - adaptation can be considered as a problem of regulation or control: minimizing deviations from a goal configuration by counteracting perturbations before they become large enough to endanger the system.

   Therefore, the software development process should be able to produce variety of actions to cope with each of the possible perturbations and select the best action for the given problem. How it is possible without developers, creators of the software system? They should be empowered, trained and encouraged to do this.

How to control a self-organized system?

1. Establish attractors (state of equilibrium, a preferred and stable position for the system). It is better to establish vision, goals for expected system, good motivation and compensation for the team instead of telling what to do to and how to achieve this state.
2. Establish principles (or fitness criteria) for choosing the best action for the given circumstances.
   • The most straightforward method is to let the environment itself determine what is fit by trial and errors. This can be dangerous for the company since it may lead to the failure of the software project.
   • Another approach is to create internal models of the environment. This allows to try out a potential action “virtually”, in the model, and use the model to decide on its fitness. Theoretically, this is the most effective approach and many architect are trying to follow it, but it is very difficult in reality as this approach should take in consideration huge number of variables and create credible simulation.
   • Use simple rules, empirical patterns and best practices – considering the current state of the software development it is the best approach.
3. Create effective self-organized teams.

Developers are not janitors and prima donnas, because they are the most important people to maintain healthy self organization in the development of the software system, avoid system degradation and lead to the project success. Architects should help developers to establish better attractors, fitness criteria and internal models (or patterns) for selection of the optimal solutions, but they are not the best people to make decisions about these solutions. Certainly it is valid statement, if we agree that evolution, adaptivity and self-organization are important (bottom up approach). Please notice, in this post I even didn’t touch people, idea flow and psychology factors – I considered software development only from perspective of the theory of complex systems and self-organization.

Useful Resources:
Self-Organizing Systems FAQ
The Science of Self-Organization and Adaptivity, Francis Heylighen
Order Out of Chaos, Ilya Prigogine

Recently I was reviewing requirements for Microsoft Architect Certification and had found description for architect roles. After reading I had a question: do we need this kind of architects in a software team? Can they really help with building successful software?

The summary of architect responsibilities (as per Microsoft):

• Enterprise architects: set the overall vision and framework for the IT environment from a strategic business perspective.
• Solutions architects: design the solution to take advantage of the existing assets, integrate them into the existing environment, follow the enterprise architecture, and solve the business problems of the business owner or unit.
• Infrastructure architects: responsible for creating an architecture that meets the business and service level agreement requirements of the business owners and supports the applications and solutions that are required to operate their day-to-day businesses.

The bottom line is that company hires expensive and canny software architects before start of the project. They talk with the business, come up with solutions and make bright key decisions about architecture. But they don’t program. During the course of the project they ensure that developers don’t spoil this great architecture.

Now business owners can hire not so bright programmers, who should just follow directives and implement this great architecture. There is even correspondence with theories of some management consultants that programmers are just house painters.

Completely irrelevant questions: Can you trust advise of your friend about Paris, if he saw the city only on TV? Can you trust battle plans if battle didn’t begin and your commander knows little about enemy? How many chess moves ahead can you predict with high certainty before start of the game?

I don’t reject a need in architecture, but I don’t agree who, when and how should make architecture decisions in a software project.

Top 10 reasons why you don’t need Software Architect

The Architect (dedicated non-programming technical decision maker and problem solver for business):

1. Has outdated programming knowledge and experience, loss of touch with modern development approaches and practices.
2. Don’t program and don’t know much about evolving system internals, but makes key technical decisions. Often has completely irrelevant and unreal picture what is happening with the system.
3. Tends to complex, premature and generic solutions when the system is still in infancy and nothing is clear. Applies latest modern buzzword technologies as SOA, MDA, SaaS, Software Factories, etc. which look so beautiful in technical magazines, conferences and CV, but cause unnecessary headache for developers.
4. Plays role of the middleman introducing complexity in coordination and project responsibilities. Represents software team in interactions with business customers reducing communication value for the rest of the team and impacting idea flow.
5. Reduces quality of decisions, which become limited to one perspective; decision making starts lacking diversity, independence and decentralization, which are essential attributes of collective intelligence.
6. Creates tension with developers who experience mismatch between grand design and reality. Often continues pushing design decisions until the system becomes overly complex, difficult to change and becomes completely unusable.
7. Secures job and justifies high salary – becomes authoritative center for solving business problems without much input from the team.
8. Causes loss of sense of ownership, motivation and accountability in developers by detaching them from the key architecture decisions.
9. Concentrates project knowledge and the big picture in one head, limiting (and sometimes preventing) complete understanding for others.
10. Contributes to creation of specialized IT verticals that hurt relations with the business.

I’m sure that some architects bring strong technical leadership, excellent solutions and overall project success. However, the architect role itself, as described by Microsoft and employed by other companies, doesn’t encourage productive development and effective solutions.

What is effective architecture? Who is effective architect?

Emerging and evolutionary architecture is a core for a successful agile (and not only) software system, emerging from the implementation of business needs, learning from working code, problems and interactions with people, other systems and operational environment.

Effective architecture:

1. Provides stable foundation and integrity for the growing software system, enabling desired system qualities for the business solutions.
2. Ensures seamless and consistent integration, well defined interfaces and interaction between subsystems, external systems and operational environment.
3. Supports emerging abstractions, key system elements and frameworks for conceptual integrity, efficiency and reuse.

Effective architect:

1. Guards a system against failure, sensing worrying changes in the project dynamic, system code and business requests.
2. Keeps the trinity of system qualities (known from time of Roman empire) in a balance and prevent degradation and entropy:
   • Firmitas (Strength) – the system is reliable, secure, has good performance and easy to support
   • Utilitas (Utility) – the system is usable, meets business business needs and add business value every day instead of drifting into technology trenches
   • Venustas (Beauty) – code and system structure are clean, easy to understand, minimal
3. Encourages constant improving of the code design, enhancing system abstractions and structure, removing duplication, defining boundaries and interfaces of the subsystems.
4. Keeps solutions as simple as possible, maintains intellectual control over system and avoids over-engineering.
5. Most important – grows and coaches other developers to become architects

If your company can afford dedicated non-programming technical person, hire an expert for complex technical areas, a coach to share best practices and experience or a technical supporter to coordinate with different external groups and help to resolve problems and make them all part of the team.

But don’t take technical decisions and concern about architecture from the development team.

The ultimate solution – every developer covers these architect responsibilities. Hire few experienced, motivated and capable developers and start project with them. Developers will constantly evolve the system based on real needs and shape effective architecture. You can trust that system will be healthy, meet business needs and bring success. Often only few important architecture decisions should be made before start of the project – trust them to the developers, who will implement the system.

So, here is the answer regarding who, when and how should make architecture decisions:

Who – every developer timely makes architecture decisions and constantly improves evolving architecture
When – mostly after start of implementation, when system becomes more complex and need design improvements to maintain system qualities, intellectual control and conceptual integrity.
How – preventing degradation and entropy by making design changes and refactoring; learning from the project dynamic, working code and business requests and adjusting a way how the development team works.

Do you still think that software projects need The Architect?

Useful resources:

Microsoft Architect Certification

Is Design Dead?, Martin Fowler

Big Ball of Mud, Brian Foote and Joseph Yoder