in·no·va·tion – introduction of new things or methods
im·i·ta·tion – the copying of patterns of activity and thought of other groups or individuals
in·te·gra·tion – an act of combining into an integral whole.

What is the best strategy for an effective developer – innovation, imitation or integration? Should you introduce new creative solutions, adapt other people ideas or just integrate existing components?

Software Development is an exciting intellectual endeavor without physical barriers. It is easy to start innovating – come up with new ideas and quickly submerge into their implementation. And I don’t mean here fundamental breakthroughs. I consider as innovation building of any non-trivial solution that is not directly stemmed from Google search results, development resources or available examples. And certainly, I pose the dilemma – innovate or not innovate – to skillful developers who are quite capable to innovate and who enjoy meaningful creative work.

Discount Machine

Lets start from a tournament organized by Kevin Laland of the University of St Andrews to find out what strategy works best to gain maximum pay-off:

• innovation - a new behaviour randomly acquired by individual learning;
• observation - a new behaviour acquired by learning from others or imitation;
• exploitation - using a previously learned behaviour to gain pay-off.

The participant had to build a strategy that their virtual agents would use to decide between these options in a computer-generated world. The challenge was to create the strategy that generated the most successful agents.

Therefore, a tournament showed that the best strategy is keeping up-to-date by learning what others are doing and using their successful solutions most of the time.

Strategies

You have three main strategies for approaching a new problem in software development

1. Integrate into the system existing software product, component or service – commercial or open source (for example, Payment Gateway as PayPal, Blog Engine as WordPress, CMS as Drupal, UI Components as Telerik and so on)
2. Imitate good enough solutions and adapt to your problem (Architecture Patterns as MVC, available code examples and guidelines as MSDN, borrow ideas from blogs, open source projects, Starter Kits, SDK and so on)
3. Innovate and create new solutions or make significant improvements to existing approaches

	Integrate	Imitate	Innovate
Time to market	Fast, if effort to integrate with other system components is low	Slow, but predictable, if not many hidden pitfalls or adaptation problems are encountered	Unpredictable as any innovative work
Cost	Low, if components are reasonably priced and not much integration work needed	More expensive and depends on complexity and adaptation effort	Unpredictable
System integrity (with system architecture and environment)	Acceptable if new components don’t screw and over-complicate core architecture	Good, if developers adapt ideas to existing architecture	Solution is built to match core architecture and customer needs
Required Expertise	Not much specialized expertise is required, usually external support is available for integration	Good developers can effectively adopt good ideas that are explained well	High level expertise, creativity and specialized knowledge are required for good innovative solution
Control over code and future development	Little control and you are on mercy of external developers	Good control if ideas are applied well and not over-engineered	Full control
Competitive advantage and uniqueness	Not much for the standard solution that many can use	Depends on quality and creativity in adaptation	Innovation is an excellent opportunity to gain advantage
Maintenance, support and improving capabilities	Work is outsourced to dedicated external developers who fix, support and improve the product	Your effort is supported in original source of ideas if you are lucky	Completely your own effort
Learning curve, tacit knowledge, help	Usually supported by help, tutorials, training and community involvement	Partially supported by original source, however can drift far as the result of internal implementation	Should be covered by you to enable effective support and future development by existing and new developers

(*) Disclosure: I should confess that the table above has some assumptions. I assume that external components for integration have good quality, work as advertised and are backed by solid support and team . Also, I assume that internal developers involved in implementation have good skills and experience. They follow good practices and are motivated to do great job and know what they are doing. I fully realize that life is not simple, and my assumptions could be completely wrong and this would change the table and the whole game :)

• Improvement and simplification of the system design to make it easier to evolve and support
• Removing technical constraints and solving technical challenges to make the system faster, more responsive and reliable
• Introduction of important business features where no standard solutions exists
• Significant improvement of users experience
• Reducing cost of development and support

• Developing system features or properties that are not required
• Building alternatives for good available solutions (reinventing the wheel)
• Playing with interesting ideas without customer awareness

The Effective Way To Build Software System

1. Understand purpose of the system, essence of customer needs and desired outcome
2. Break down the system into components and research if
   1. standard solutions exist (for integration)
   2. implementation ideas exist (for imitation)
   3. innovation is required
3. Can you change needs and requirements? Transform customer needs and architecture ideas to minimize development effort
   1. by moving from innovation to imitation strategy
   2. by moving from imitation to integration strategy
4. Evaluate each component from the system and business perspective
   1. Should you avoid integration and use imitation if:
      • System integrity under the threat and the component is part of the system core
      • Control over code and future development is required
      • Competitive advantage and uniqueness are important
   2. Should you still go with innovation because of unresolved contradictions, challenging and unmet needs?
5. Build prototypes clearly separated from mainstream development to confirm selected strategy

Becoming an effective software developer

1. Systematically study other solutions in your area of specialization (at least a couple in a month – understand strengths, weaknesses, high-level architecture and interesting tricks)
2. Learn concepts and language of your business domain to be able to understand customers and shape their needs together
3. Enhance abilities to find and brainstorm alternatives (improve techniques, make them essential part of your process)
4. Become an expert in Google search and fast evaluation (no kidding, these skills become very important for any modern developer)
5. Master rapid prototyping, apply solutions in practice and seek for rapid feedback (answer in short time if proposed solution is good, learn and correct if you made a mistake)
6. Develop a holistic view and knowledge of the system, infrastructure and environment (understand subsystems, connections, integration options and trade-offs)
7. Keep up with latest software development trends, technologies and approaches (subscribe to blogs, magazines and other sources)

1. Achieve deep specialization in your core technical area and business domain (extensive experience and deep knowledge are great assets for innovator)
2. Continuously develop creativity and problem solving (the post about creative problem solving)
3. Enhance architecture and fundamental programming expertise based on own practice and ideas from others
4. Master Evolutionary and Domain Driven Design

Jason Cohen posted an interesting and provocative argument against Release Early, Release Often principle followed by many agile teams.

His main points:

• Ideas. The best ideas are not coming from users and they are bad in providing feedback (iPod). So, there is no point to release early to get their opinion and ideas.
• Features. Minimal early set of features could be unattractive for majority of users and will turn them down for future use (Apple Newton)
• Quality. A buggy and unpolished product could ruin your reputations
• Architecture. An incorrect initial architecture creates waste and serious problems down the road (Netscape, Twitter)

Therefore, Jason against releasing early and often. I don’t agree.

My answer: it depends!

Evolution is the process of small frequent changes to improve and adapt to environment.

One of my posts reviewed how software team should select the best strategy: evolution, revolution or retreat. Another post compared evolution to revolution. Evolution (and aligned core Agile principles) requires early and frequent releases. Revolution pushes innovative product that should disrupt market. Requirements for a revolutionary product ideas are much higher, because it should overcome resistance, create new niche and gain acceptance of the new paradigm. Therefore, revolutionary product should be well thought and prepared to hit the mark. I should note that revolution often becomes evolution after initial release.

Decision factors for selecting strategy Release Early, Release Often (evolutionary):

1. People. Are they highly talented and capable to come up with great ideas from the beginning ? Or should they learn and understand better the problem space, release incrementally and get more feedback from the users for initial assumptions?
2. The Game. Is the problem (software requirements and business domain) unclear, complex, challenging and require a lot of trials and feedback? Or is the problem space well known and team already have good experience with it and can release a great product from the first attempt?
3. The Dynamic. Is the team under a pressure to release a product early, catch at the market opportunity or help a company to survive? Or do they have luxury to take a time for designing properly and release a polished product?

There are some cases where long release cycles make sense for the company:

1. A company has deep expertise and hired highly talented and experienced people who know how to build successful product.
2. A company have enough funds and time to sustain long development and tolerant to inefficiency because incorrect assumptions or lack of feedback.
3. Customers have low tolerance for risk and the software is mission- or life-critical.

However, I expect that for many software teams the most optimal strategy will be evolutionary - Release Early, Release Often.

I partially agree with Jason that most users will not help with ideas or provide meaningful feedback. Also it is a bad idea to ship buggy, unfinished and useless product. But a team could get more than user feedback from early and frequent releases:

• release! This is a huge and most important criteria of development success – the product is releasable and working
• integrate and put the product pieces together, which is almost impossible to replicate in the test environment
• practically experience work of the live product - infrastructure and production problems, performance, scalability and user interactions in their environment. These problems can radically change the view on architecture
• reality check - learn from how people use the product: validate initial assumptions, collect praises and complains, real usage patterns, statistics and maybe even some ideas from users :)
• emergence of the new ideas and process improvement after experiencing product in the wild live environment

A software company should avoid treating users as guinea pigs for their early experiments, but nothing can beat practical benefits of releasing code and learning from it.

“In theory there is no difference between theory and practice. In practice there is.” — Yogi Berra

Referenced post: Releasing Early Is Not Always Good? Heresy! by Jason Cohen @onstartups.com

The manager slams a door and tells us that we are in a big trouble. Our old customers complain about many bugs and bad performance, new customers complain about delays and lack of dedication. And, top management considers our department financially unsustainable and wants to deeply cut expenses.
The manager tells that we are brilliant programmers, work very hard and create cool software solutions. But there is something wrong and we cannot work this way anymore.

Anxiety started to penetrate our souls. We know what is wrong: our team is short of people, we have too many commitments, our code is becoming a big mess, new technology and our new software version makes something bad with servers. A snowball of different problems makes us stressed, distracted and incapable of productive work.

What could our manager do next?

1. Distrust. Become a dictator, make own decisions including hiring external consultants to recommend what to do or even replace us. However,
   • we are good programmers and know our business well – the problem is not in lack of skill and knowledge
   • external people will take a lot of time to understand the system and they will have different motivation and won’t care about the long-term success
   • people will be demotivated and the manager cannot make effective decisions without active team involvement
2. Faith. Give to team the full power to fix a problems and make own decisions in hope that smart people, motivation and technical expertise will do magic. However,
   • fresh outlook and thinking out of box are hard when a team immersed for a long time into difficult situation
   • a team possibly doesn’t have understanding and control over external forces – management, customers, finances
   • changing of reality is tough (especially in people heads) and requires more than technical experience

There is a third way. Place improvement practices in the core of development process. Make self-improvement inevitable and required for any activity. Do it every day.

Toyota Way is the best example of large-scale reliable self-improvement process. It focuses on eliminating waste, solving problems at root cause and making right decisions. Toyota Way reduces problems, increases internal efficiency and makes a company successful. This is the best receipt for coming out of crisis.

Targets:

• Problems - emergencies, fires that require immediate fix: bugs, server crushes, deadline slips

• Waste - inefficient and non-value adding activities: waiting, misinformation, stress

• Challenges - adaptation to external forces (market, competitors, customers, society): new trends and technologies, changes in users expectations for user interface and functionality

Practices to see waste and stop to fix problems

1. Seeing waste

The team and managers should learn to see targets – real problems and waste. Otherwise improvements will be wild shots in the dark.

There are many targets in software development:

• stressed people – reduced energy, less productivity, more mistakes
• waiting – delays, tools / system problems and downtime, capacity bottlenecks (waiting for results of other people work)
• over-engineering – producing features and complicate design without real need
• unfinished work – functionality not used in a live system, probably still in design or under development or simply discarded (but not removed)
• defects – complete waste of time and money
• unused creativity – loosing time, ideas, skills, improvements and learning by not engaging or listening to your employees
• inadequate information – unclear, misleading or simply wrong information that causes useless activity and leads to rework at the end

• over-processing – taking unneeded steps to build software because of poor process and system design, overhead, bureaucracy, compliance, cumbersome tools
• motion – how much effort to get necessary information, access systems or use tools
• multi-tasking – losing time to switch between projects, tasks or different activities

2. Jidoka - stop to fix the problem to get quality right the first time

It is not enough to see your targets. The Team should carry commandment to shoot targets immediately. Otherwise the best intentions will be buried under growing avalanche of problems.

How to stop:

• quality for the customer drives all the processes – prevents temporary patches and bad for quality decisions
• low tolerance for quality problems and immediate detection are core work principle
• use Andon – a system to signal for help, notify about a problem and stop the process

• build in tools and process capabilities of detecting problems and stopping itself
• use all modern QA methods available
• managers encourage stops to fix problems and support implementation of counter measures

Making Decisions

“Make decisions slowly by consensus and thoroughly considering all options, implement rapidly.“ – Toyota Way

Even knowing problems and committed to solve them, the Team should make right decisions how to do it.

Make decisions slowly

1. Go and See (Genchi Genbutsu)
2. Understand underlying causes (Kaizen)
3. Broadly consider alternative solutions and develop a detailed rationale for preferred solution delaying certain decision as long as possible (Set-Based concurrent engineering)
4. Build consensus within a team and partners where group decision is preferred (while management can step in if consensus is not achieved)
5. Use very efficient communication devices – preferably one side of one sheet of paper

Implement rapidly

1. Put in place solutions and counter measures.
2. Evaluate the results
3. Standardize if solution is effective.

Practices to Eliminate Waste and Solve Problems

3. Genchi Genbutsu – go and see for yourself to thoroughly understand situation

How often do we jump to conclusion based on partial information, vague assumptions and what other people say? Information creates reality in your mind. This reality is a base for your decisions. So, you and the Team should get right information to make right decisions:

• observe situation with blank mind
• avoid assumptions and preconceptions
• use personally verified information

In short, base decisions on what really is going on

4. Kaizen (5 why’s) -continuous learning and improvement

“We view errors as opportunities for learning” – Toyota Way
The Team should find the root causes of the problems. Kaizen helps to find the root cause by repeatedly asking why the problem occurs.

Example of Kaizen
Problem: there are persistent javascript errors on a live site

1. Why? A developer didn’t build correct logic for complex web UI components interaction
2. Why? A developer built own solution without guidance and enough experience in this area
3. Why? A team expert didn’t tell about existing proven solutions, didn’t help and didn’t share knowledge
4. Why? The team is under stress, over-committed and don’t have time to communicate
5. Why? Managers accept too much work without consulting with development team
6. Why? you can continue…

Kaizen forces us to overcome desire to find a first convenient explanation and patch problems without fixing root causes. By ruthlessly applying this practice, we get deeper insight into reality and better learn our product, processes, people, environment and tools. Kaizen is a core practice to see waste, solve problems and improve process.

To avoid forgetting learning from Kaizen, it is important to standardize the improved process and make it a base for further improvements.

Practices to Support Flow

5. Standards – best you know today which is to be improved tomorrow

Standardized work is easier, cheaper and faster – stable repeatable methods can maintain predictability, high productivity and support quality.
Effective standards are not coming from theories, they come from

• best practices
• accumulated learnings and individual experience
• lessons from applying existing standards

The Team should try to use standards in many areas: project phases and activities; development practices; architecture and design approaches; code conventions; tools; programming techniques; libraries and third-party code; reuse of components and solutions; testing and so on.
Standardization in software development is a controversial topic – some theorists want to bring programming closer to standard-dominated engineering, practitioners are keen to reduce standardization to minimum promoting creativity and self-organization. In the rigid interpretation, standards are “must to follow” rules for any situation, in other interpretation standards are well defined steps and guidelines highly recommended for specific context. I support the latter definition. A productive team should have standards in place to focus on customer needs instead of fighting with the same puzzles and problems over and over again.

The system of standards shouldn’t be a heavy bureaucratic conduit, but a light and fluid book of knowledge. The book that contains most helpful and important rules and checklists. Standards will be effective if they are minimal, reviewed often (Kaizen) and followed by every team member.

6. Reliable thoroughly tested technology

The Team should be conservative with new technologies. Software development and IT thrive on change and innovation. However, Toyota Way suggests to be conservative in adapting technology and considers stability and reliability of operations as much more important goal than keeping on the cutting edge of technology.

Considerations for using technology

• primary goal is to improve flow and support people, process and values.
• process is driven by business, not technology concerns; software and tools do not eliminate themselves waste
• technology is visual and intuitive – people can use it correctly and effectively

• process manually before adapting technology to support the process – understand what problems it solves and how technology could help
• important: people / processes are easy to Kaizen, machines are difficult

Adopting new technology:

• new technology is unreliable and difficult to standardize, therefore it endangers flow
• proven process takes precedence over new and untested technology
• conduct actual tests before adapting new technology
• reject technology if it conflicts with culture or might disrupt stability, reliability and predictability

In the same time encourage people to consider new technologies while looking into new approaches. If technology improves process and flow – quickly implement after thorough testing.

7. Visual Controls

The Team should have clear status of information. Visual controls can convey complex information in fast and efficient for our brains way. We can use controls as a user story board; status of projects, servers or code build; burn down charts and others.
Simple visual indicators help people determine immediately whether they are deviating from the standards, provide quick gist of situation and direction for solving problems.

• use simple and most important indicators

• than provide clear picture for decisions and what to do next
• reduce reports to one screen / piece of paper even for the most important decisions

People, leaders and teams

8. People

People who build software are the people who should improve the process. They are directly involved and have first-hand experience of problems and waste.

Toyota Way expects that each team member is a problem solver and values job experience more than theoretical knowledge. The Team will beat any external consultants and find better way to work if people are open about problems and eager to find good solutions.

9. Leaders

The Team desperately needs strong leaders to build great products to overcome problems. Toyota grows leaders within who thoroughly understand the work, live the philosophy and must understand the daily work in great details

Chief Engineer is a key person in Toyota projects.:

• blessed by top management
• has control over project
• exceptional engineer
• critical link: between engineers and customer satisfaction
• coach for other engineers
• focus on concepts first, technicalities later

Chief Engineer concept is an excellent example for software technical leadership. A software team leader often lacks authority or makes too technical decisions without good understanding of customer needs.

10. Teams

The Team should be diverse and capable of solving wide range of problems. Toyota builds cross functional product teams, which

• use integrative decision making
• fast and accurate in implementation
• enhance process and flow by solving difficult technological problems

Software developers and their leaders are foundation of success in any project. Management, process and technology can only support them. And anyway, the process is as good as people follow it. Therefore, it is important to make software teams a key player in process improvements – they know problems, they understand work and they are capable to find good solutions.

Using Toyota Way

Can The Team revert a situation and win? Can it build the optimal process and expertise for fast development of high quality and low cost solutions?

This post shows the most effective option – build continuously improving process into the heart of development. The process that focuses on quality, eliminates waste and fixes problems at root cause. I believe this approach is a foundation of long term success. Your managers and company would love it!

A software project is a creative, unique and therefore unpredictable endeavor. We are not building the same thing over and over again, but solve new problems, address increasing demands and use perpetually changing technologies. Under these conditions, people – smart, creative and productive – are the most important factor of success . Software development process can only support and compliment these people, but it cannot guarantee success alone and make the factor of people negligible.

But, business wants predictable, reliable and successful results. I bet they don’t want to be at mercy how cards are shuffled in their talented development team. The answer is in establishing a process that increases chances of success and aligned with present nature of software development (unpredictable, empirical and heavily dependent on people).

The Toyota Way can be a great example that worth to learn. Toyota evolved from a small looming equipment shop to the largest car manufacturing company. The main foundation of successful growth is the system of few core principles that enables best quality, high productivity, lowest cost, shortest time and long-term success.

Agile and Lean Software Development adopted some practices from Toyota. Lean Software Development: An Agile Toolkit  (by Mary Poppendieck, Tom Poppendieck) is an excellent book that review Lean practices and principles in details. This post focuses more on philosophy and the system of practices as a whole that can turn software development team into a smart reliable machine.

1. Value

Why customer will pay money for your product? How does your company creates value?

All we are doing is looking at the time line from the moment the customer give us an order to the point when we collect the cash. And we are reducing that time line by removing the non-value-added wastes.

- Taiichi Ohno, founder of TPS

The only thing that adds value in software development is transformation of information and code into what customer wants. It is very important for software team to know how does it create value for customers. Or simply, decrypt the formula:
Customer (needs) -> Cash (delivered software)

Software Development Value Stream
Problems, needs and ideas are translated into units of development – functionality, user experience (interface, interactions, usability) and system qualities (performance, reliability, security, etc). A software development team transforms these pieces of requirements into a computer system using value-added and many not-so-value-added actions.

The ultimate goal of Toyota Way is to eliminate non-value-added activities while keeping the best quality of the product.

2. Flow

What is the process of creating value in your company?

Properties of the optimal flow for software development:

• cut back to zero the amount of time that any unit of work is sitting idle or waiting for somebody to work on it
• move design, code and information fast, link processes and people together that problems surface right away

One-piece flow
The most effective flow is one-piece flow – a customer need is immediately converted into a delivered software solution. In the same time this is the most demanding flow: a customer, designers, developers, testers and system administrators should be dedicated to the project, immediately available and work as one team together until the need is implemented in a live system.

One-piece flow has significant benefits (if it is implemented right):

• Quality - defects detected quickly; close proximity and connections ensure that necessary information is available immediately and important details are not missing; problems are solved quickly
• Flexibility - flexibility to respond and do what customer really wants in real time
• Productivity - the team engaged in high value added continuous work
• Morale - immediate results of work bring sense of accomplishment and job satisfaction
• Cost - lean production with minimal waste

However, in reality one-piece flow is difficult to achieve in many projects:

• a customer is not engaged all the time
• complex business domain requires analysis and preparation before development can start
• project activities involve people with different skills that are not available immediately
• deployment of mission-critical systems demands intensive testing and scheduling

• management, marketing, end-user expect specific time-lines, features and support

• and so on

Toyota uses two additional principles to overcome challenges of one-piece flow: pull systems and leveling out the workload.
However, Toyota Way says: “Flow where you can, pull where you must“. The big challenging goal is to turn eventually all the processes into one-piece flow.

Pull system (Kanban)
Kanban is a signaling system to trigger action. It adds to flow small buffers pulled by customer demand. Kanban allows optimal use of people and natural breaks in processes.

Agile iterations and Scrum sprints are close to Kanban ideas. A team pulls new user stories every time-boxed iteration from the backlog to design, implement, test and deploy within the same iteration.

However, it is not classical Kanban. An original idea would be to pull small batches of user stories for development after a team has finished a current batch (not at the end of fixed time iteration). In parallel, QA pulls implemented user stories for testing and so on. A customer can continue preparing next batches of work based on their priority of needs.

Main ideas

• provide dependent down stream teams with what they want, when they want and in amount they want
• just-in-time – started by high priority needs
• minimize work-in-progress and untested code
• frequently push what a customer can use right away
• be responsive to day-by-day shifts in demand instead of relaying on long-term schedules

Kanban is effective way to synchronize people and teams work without bringing them into continuous flow, where they should be always ready for the next unit of work.

Level Out the Workload (Heijunka)
The main idea of Heijunka is to level production by volume and activities mix.

• it is not built around actual flow of customer orders
• workload is leveled for the period considering previous history and scheduled projects

In software development workload could be leveled and scheduled when people could make reliable guess for what is expected. It is different from Kanban that works as just-in-time system adapting to the current workload.

Good candidates for Heijunka are

• standard procedures (automated testing, performance testing, deployment)
• recurring events (meetings, planning, demo, code reviews)

• scheduled tasks for people with limited availability (external consultants, customers, management)

• tasks with high confidence in estimation, usually done many times before (initial requirement exploration, design mockups, security audit, training new team members, reusing components and solutions)

Benefits:

• predictability and active planning

• balanced use of people, ability to schedule their involvement

• smoother demand on customers, shared resources and contractors

Toyota Way is focused on eliminating waste of non-value added work (Muda). Heijunka eliminates two additional types of waste:

• unevenness in project schedule (Mura)
• overburden of people and systems (Muri)

Building your own flow

These three approaches can help you to build most optimal flow. Consider cons and pros:

One-piece flow (push)
pros: minimal waste, fast implementation, low cost, quick problems resolution
cons: uneven workload, requires full availability and high level of engagement
when to use: for one collocated team work that is dedicated to one project

Kanban (pull)
pros: dynamic adaptation to load; synchronization of different teams and disconnected processes
cons: low predictability for completion; buffers hide waste
when to use: disconnected or involved in the multiple projects teams; activities that need people with limited availability

Heijunka (level)
pros: good predictability, ability to plan, balanced use of people: avoid overburden and uneven workload
cons: non-adaptive, inefficient for tasks with unpredictable effort (a lot in traditional software development)
when to use: activities that are standardized, highly predictable or have to be scheduled

You cannot build reliable and optimal flow from the scratch based on a theory only. The best process for your projects will emerge as result of evolution, problem solving and eliminating waste. This is a topic of the next post.

If some problem occurs in one-piece flow manufacturing then the whole production line stops. In this sense it is a very bad system of manufacturing. But when production stops everyone is forced to solve the problem immediately. So team members have to think, and through thinking team members grow and become better team members and people.

- Teruyuki Minoura

References:
The Toyota Way by Jeffrey Liker
Lean Software Development: An Agile Toolkit  by Mary Poppendieck, Tom Poppendieck

Computers are useless. They can only give you answers. – Picasso

Many people (including my mother-in-law) think that computers are becoming so smart that programmers will be no longer needed in the near future. Other people think that programmers are geniuses who constantly solve sophisticated math puzzles in front of their monitors. Even many programmers don’t have clear idea what they do.

In this post I want to provide some explanation to uninformed people what programmers really do:

Programmers are translators of human ideas into the language of computers.

They are a link between two worlds – human and computers. Do you think it is easy to maintain this link?

Human World Background

The problem with people is that they’re only human. -  Bill Watterson

Humans are product of biological evolution and have a unique and precious organ for producing ideas for programmers – the brain. The brain is a complex combination of neocortex (unique to humans) and older structures inherited from mammals and reptiles. Older brain structures are mostly responsible for reproduction (sex) and survival (finding food or escaping danger). Neocortex evolved to support these function better, however it started to cause strange side effects – consciousness, thinking and curiosity. Thanks to these effects humans invented civilization and thousands years later computers.

Somebody could believe that after thousands years of development humans should become completely dull, predictable and rational species, but it didn’t happen – their old brain structures, complex psychology and social behavior often make people irrational, unpredictable and deceiving. In addition, humans have poor memory, strong emotions and personal interests. However, programmers don’t have choice of working with more rational species and have to work with human beigns to translate their ideas into the language of computers.

Computer World Background

Part of the inhumanity of the computer is that, once it is competently programmed and working smoothly, it is completely honest. – Isaac Asimov

A computer is the best invention of human civilization. It consist of CPU, motherboard, memory, hard drive, monitor and some other parts. Computers moved our civilization to the new level, filled our life with meaning and entertainment and compensated weaknesses of our brains. There are good chances that computers will become more intelligent than humans. (However, I am a bit concerned if computers need human programmers after it happens.)

Modern computers are completely logical, straightforward and obedient. It is pleasure to work with a computer if you know what it should do and how to instruct it. The only problem is that computers will do exactly what you tell them to do. Therefore, you should have very clear ideas and instructions for a computer to avoid feeling miserable when you see your boss or customer.

Translation Between Humans and Computers

There are three main challenges in translation:

1. Language ambiguity. Human language is vague, complex and ambiguous – for example: “This program doesn’t provide good user experience”. Culture, background and context affect communication and meaning. On the contrary, any computer language is exact, straightforward and context-free.
2. Levels of details. Humans communicate often in general terms without many details – for example: “I want this f* program work right”. It allows them to save time and energy, but cause two big problems – misinterpretation and possibility that important details are missing. And a computer requires all details – everything should be spelled out.
3. Thinking style. Humans often think in terms of needs, outcomes and solutions – for example, “This report should run in 2 seconds instead of 2 hours”. However, computers need algorithms – sequence of steps how to achieve desired outcomes.

In order to write good software, programmers have to overcome these challenges, understand humans and translate their ideas into the computer language.

Skills of a Super Programmer

As we can see a super programmer should have two distinct sets of skills to deal with both worlds.

Understand humans and create solutions:

1. Communication -ability to establish contact with humans, talk with them without alienating and even heroic attempts to share own opinion.
2. Meaning - extract useful information from conversations with humans, decode and make sense from it
3. Logic - clearing, removing ambiguity and controversy from human ideas for uncompromising reality of computers
4. Creativity - dig, twist and play with human ideas to create good solutions
5. Design - wrap programming ideas with human friendly interfaces and convenient interactions
6. Big Picture – know how solutions fit into world of users, business and Universe to make your program useful.

Tell computers what to do and build solutions:

1. Logic (again) – organize programmer’s thoughts into cohesive software ideas and instructions for computers
2. Technology - uncovering and understanding the black box of technology (black box for 99% of population)
3. Programming Languages – learn the beautiful, logical and unambiguous languages for feeding computers with programmer’s ideas
4. Algorithms - master the most effective ways how a computer can accomplish a task
5. Modeling  - create abstractions and models for grasping and manipulating ideas in software code
6. Practices (as Refactoring, Unit Testing, Continuous Integration) – recurrent activities to keep system solid, healthy and possible to change

There is a big difference between a human-oriented and hardcore object / system – oriented programmer.
A programmer who is specialized to work with computers only is a half of the good programmer. Great solutions require skills for computers and human worlds.  Connect both worlds and become a super good programmer!

Reality is merely an illusion, albeit a very persistent one – Albert Einstein

An expert have much better models of reality and methods to build them than an ordinary specialist. The expert, armed with these models, can quickly put pieces of a problem puzzle together, find explanations and solve the problem.

Models can be related to anything – software systems, business domain or your personal relationships.

Predictions and Solutions!

A theory is a good theory if it satisfies two requirements: It must accurately describe a large class of observations on the basis of a model that contains only a few arbitrary elements, and it must make definite predictions about the results of future observations.

Stephen Hawking, A Brief History of Time

Any expert as good as he can effectively predict future outcomes and solve new unforeseen problems. Solutions and predictions are not coming out of blue. They stem from good understanding of reality in an expert’s head. Internal models of reality help to explain problems, find relations and play with future scenarios. These models establish the base for effective thinking and direct expert’s effort to solve problems. If you have poor models of reality, you will make bad decisions. Good models of reality allow reliable predictions and efficient solutions than vague guesses and ad-hoc spontaneous fixes.

Characteristics of a good model:

1. Explanation - provides understanding of elements, processes, events, root causes and effects
2. Testable predictions - a model that makes no predictions that can be observed is not a useful model.
3. Simple - you can remember and use a model without titanic effort
4. Frameworks - knowledge how to recognize common patterns and deal with range of similar problems. Frameworks allow to extend and apply existing models to new situations without building new models.

Brains – the main tool of an expert

I think, therefore I am – Rene Descartes

An expert’s brain carry and operates with models of reality. Unfortunately, our brains are not ideal for this task. Models they create are not identical to reality. Here is why.

limitations

• perception - our senses deliver to brain partial and incomplete information
• memory - our brains can process and memorize only fraction of available information; and even more – we quickly forget this knowledge without practice
• imagination - brains automatically filling gaps for missing information, substitute with interpretation based on internal representation of the outside world

cognitive biases (bugs in brain software)

• anchoring or priming – previous irrelevant experiences prime your consequent thinking and decisions
• need for closure – we are uncomfortable with doubt and uncertainty; we have urge to resolve and find convenient explanations quickly without much thinking
• confirmation bias -  from all facts we unconsciously pick facts that confirm our beliefs and predispositions
• symbolic reduction – we are anxious to reduce complexity for better understanding and tend to form simplified generalizations ignoring complex details and relations.
• and many others

social behavior

• subconscious models and believes – our minds carry beliefs and views imposed by our parents, teachers and culture.  These subconscious models deeply influence our models of reality without much awareness.
• peer pressure, conformity – our minds are wired to conform and agree with other people making us blind for inconvenient facts, ready to follow crowd and accept conventional views without critical revision

Objective Reality

There are no facts, only interpretations.  – Friedrich Nietzsche

Even smartest brains will have challenges to understand reality.

We deal with complex situations:

• incomplete information – we rarely have access to full information and even if it is available we don’t have capacity to collect and absorb all of it
• perpetual changes – reality is very fluid and changes every second
• complex relations and behavior – complex systems are difficult to understand: causes and effects, correlations, feedback loops, influence of processes in external systems
• difficult to test – we often cannot afford large number of experiments, trials and errors to come up with right models and explanations

We deal with people who make our reality very complicated:

• unpredictable - people are difficult to predict, they are often irrational and inconsistent
• confusing - people cannot clearly explain their thinking and feelings or even don’t understand themselves
• deceiving - sometimes people have difficulty to tell truth or simply lie for own advantage

Building Models

The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function. -  F. Scott Fitzgerald

We understand reality different ways – learning, investigating problems, finding new solutions and practicing. For years we can concentrate on solving immediate problems and do our work without attempts to understand the big picture. Many good specialists have incomplete understanding and fragmented knowledge that still allow to perform well. But it is impossible to become an expert without deep understanding and conscious effort to build good models of reality.

For example, we can stumble for a long time with challenges and failures in building software without understanding its essence and forces. (By the way, search for this understanding is the reason for this blog existence).

Potentially everything what we do in life could help to build and improve our models of reality. We just need to consciously relate our experience and learning to our understanding of reality. And we can do it on the go without special preparation, when we actively engage our minds – working, talking, reading, practicing, traveling, thinking and even sleeping. Just focus on reality modeling.

The models of reality could be explicit (facts and theories) and implicit (intuition and tacit knowledge).

You can use several strategies to build models in your whole mind.
A. Analytical

• find what is most important, map and connect main elements, players and subsystems; learn relations; draw Mind Maps
• understand influence of context and main forces
• create stories, metaphors and patterns – make your models better suited for your brain
• repeat the process for the most important subsystems

B. Empirical

• find most pressing problems, contradictions or real life facts
• understand how do they fit into your models of reality
• it is great if they don’t fit – you have opportunity to improve your models

C. Synthesis – reconcile your models with existing knowledge and leading models

• find what are dominating models, views and believes; what other experts think
• critically analyze this information and theories behind
• find the best opposing theories and try to combine them to form better model

D. Scientific method – use it for practical evaluation of models and confirmation of your theories.

• build hypothesis about reality and outcomes of your actions
• test it in practice
• learn from results and adjust your models

E. People interests and agenda – people are often most surprising and unpredictable element of reality. Your models will be much closer to reality if you understand motives, interests and agenda of involved people.
F. Change perspective – switch different perspectives to enrich models – economic, emotional, social, physical, etc. Try to view from opposite viewpoints and set of principles. Imagine yourself as user of your system, investor of your company, manager, any animal, computer part, etc. (Try to get back without loosing your identity :))  Web developers have interesting concepts of Personas – imaginary users with distinct behavior on website that help to design a system for various cohesive sets of needs.
G. Lateral thinking

• random association – associate models with recent books, distant knowledge fields, your favorite characters
• provocative thinking – make unacceptable, funny, stupid ways to explain problems and shock your models
• ask “why?” – continue asking ‘why’ until your reach questions that nobody could answer on this Earth (or the only answer is 42).
• expand concepts borrowed from other sources to have wider horizon for ideas

In overall, you will switch between two grand strategies – Evolution (discover reality and adopt to it) and Revolution (breakthrough and change reality). The choice depends on models maturity, certainty and your people capabilities.

Therefore

Few people have the imagination for reality.  – Johann Wolfgang von Goethe

You can be certain that reality in your head is different from objective reality, which is complex, changing and controversial. Effective experts don’t have choice and master reality by
1. Accepting own limitations in understanding of elusive reality
2. Building good models of reality applying different strategies
3. Using, testing and improving models in practice

Good models of reality will make your thinking, decisions and solutions much better. They will make you a master of reality instead of a victim of reality. Good luck with embracing reality!
References:
The Opposable Mind: How Successful Leaders Win Through Integrative Thinking , by Roger L. Martin
Pragmatic Thinking and Learning: Refactor Your Wetware, by Andy Hunt

A really great talent finds its happiness in execution. – Johann Wolfgang von Goethe

source

Qualities of well composed code:

1. Quick discovery and understanding of programming logic and components
2. Clear organization (for human brains)
3. Ease of reuse, modification and evolution
4. Close connection between customer ideas and system implementation

Style Components:

• Intention - understand your task and how to get it done
• Approach - basic principles of writing code
• Composition - organization of code
• Expression - expressing ideas in code
• Object Oriented Pragmatic Style

1. Design for customer problems
Make the customer problem space a core for the software system design. Make every line of code accountable for solving customer needs. It is true that any software system requires a lot of plumbing beyond pure implementation of customer needs. However, technology concerns shouldn’t prevail. High-level languages and modern programming platforms give us a power to concentrate on customer problems more than on solving technical problems.

Code could be divided into two categories:

• customer oriented – solves customer problems directly, e.g. implementing domain logic, UI interactions and display of information;
• system oriented – solves technical and application specific problems – data access, event handling, interfaces between subsystem, utilities and other code required for using libraries, systems and platforms.

Customer-oriented code brings most value and should be primary design concern. System-oriented code puts in place infrastructure and support for running customer oriented code.

Therefore, start with design of customer-oriented code and support it with minimal system-oriented code. This doesn’t mean that system-oriented code is not important. Complex system environments, sophisticated user interfaces and challenging non-functional requirements (performance, availability, reliability, etc.) could demand major system-oriented development effort. But still customers needs should pull and direct this effort and the whole implementation.
There are many ways to make technical problems dominant over customer problems:

1. Making predetermined technical decisions (databases, programming platforms, messaging systems, expensive middle tier etc.) without considering simpler alternatives.
2. Coming up with complex up-front design instead of evolving a simple design into optimal design.
3. Starting programming around interesting technical problems forgetting what is important for a customer.
4. Building frameworks and system-oriented code before writing code for immediate customer problems.

For a example, some teams could start with this picture in mind stretching customer requirements to fit it (even for relatively simple applications).

source
2. Organize and evolve code around domain concepts and customer ideas
Make system design reflecting customer domain and problem space. Keep customer ideas and system implementation connected and synchronized all the time. This is as important as refactoring practice for effective creation of the software system.
Domain-driven design is an excellent way to build software systems. It puts emphasis on ubiquitous language, distilled domain knowledge and shared domain models that drive software development.

3. Think as a customer
What we see depends mainly on what we look for – John Lubbock
A programmer often considers programming as an opportunity for solving interesting technical problems. Often a programmer even doesn’t understand what customers really want. There are 2 consequences. First, the programmer is not interested in simpler solutions focused on the customer problem. Second, the programmer is missing coherent view on the customer problems and purpose of the software system. Over-engineered, complex and misaligned code is one of the most serious problems in software development. An army of business analysts, stressed project managers and smart customer proxies will not force indifferent and ignorant programmers to write code in the best interest of customer.
The programmer should constantly ask himself: “Will this solution work for my customer?”. To give correct answer on this simple question the programmer should understand a lot – customer language, needs, problems, a big picture and business purpose.

Certainly, it is difficult to think as a customer if you are at the bottom of software creation chain:

• separated from a customer by project and product managers, architects, designers, consultants, experts and panels of stakeholders
• pressed by the heavy process, rigid organization structure and inability to make decisions
• fed with over-processed, distorted and disjointed doses of information coming down through long chain

Beautiful software ideas can turn into monster applications – unusable and disconnected from real needs. Did you feel (as a user) frustrated by complex confusing and irrational program screens and logic? Yes, we, programmers, can easily produce bad stuff if we cannot think as people who need our systems.

4. Sketch programming ideas with unit tests.
Write unit tests first. Any programmer should understand what he is trying to solve with the new code and how it will be used – possible input, expected output and public interface. A programmer should think about behavior in isolation, under border conditions or within context of existing code. An unit test is an excellent place to sketch nonexistent code, play with it without compilation and see if it is convenient and makes sense. Once the programmer likes sketched ideas in unit tests, he just need to build simple implementation to satisfy these unit tests. The process of writing unit tests focuses a programmer on solving customer problems, understanding code intent and making optimal design decisions. As an additional side effect, the programmer receives comprehensive suite of automated regression tests and executable specifications.

For a example,

CustomerDB.Save(TEST_USER, TEST_PWD)
Customer customer = CustomerDB.Login(TEST_USER, TEST_PWD);
Assert.IsNotNull(customer, "customer should be logged in");
customer.Purchase(TEST_PRODUCT);
List<Order> orders = OrderDB.LoadFor(customer) or maybe... customer.Orders //what is better?
Assert.AreEqual(1, orders.Count, "count");

There are many design decision are made in this unit test without writing any production code. And now programmer can concentrate on implemention without throes of creation.

5. Eliminate duplication
Simplification is ultimate sophistication – da Vinci
Elimination of duplication is the moving force of evolutionary design. This is straightforward and powerful method to grow an optimal system. Duplication in the code tells that you need better design. Many interesting design ideas are born from the simple need to eliminate duplication in the growing system. These ideas are based on deeper understanding and gained experience as oppose to speculative up-front design ideas.
The main goals of removing duplication are

• reduce code
• simplify solution
• make the system more manageable for programmers’ minds.

Some sources of duplication:

• copy and paste – acceptable only temporary
• similar logic different in details – generalization is required
• another implementation for the same problem – better team communication and reuse are required

Master the core skill of good evolutionary designers – how to detect and eliminate duplication.
The knowledge of design patterns is the most useful on this step. The initial simple solution rarely requires design patterns. But as a system grows, design patterns can effectively and elegantly solve new complex design problems.
Example of elimination of high level duplication with Template method design pattern
This code was used in many data access classes almost the same way, but with different inner steps

        using (SqlConnection conn = NewSqlConnection())

        {

            SqlCommand cmd = conn.CreateCommand();

            cmd.CommandType = CommandType.StoredProcedure;

            cmd.CommandText = "SearchCustomers";

            cmd.Parameters.Add(new SqlParameter("@city", city));

            ... //long list of parameters

            conn.Open();

            try

            {

                using (SqlDataReader reader = cmd.ExecuteReader())

                {

                    foreach (DbDataRecord record in reader)

                        customers.Add(new Customer(record["id"], record["name"]))

                }

            }

            catch (Exception e)

            {

                //error handling logic here

            }

        }

The common logic could be moved into one template method with placeholders for different steps:

List<Customer> customers =  LoadRecordsFromSP<Customer>("SearchCustomers",

    (SqlCommand cmd) => {

         cmd.Parameters.Add(new SqlParameter("@city", city));

         ...     //long list of parameters

    },

    (DbDataRecord record) => { new Customer(record["id"], record["name"]); }

);

6. Reduce code smells
Write code that doesn’t smell. Train your sense of smell for code and design problems. They will provide hints where code is probably bad.
These are common bad smells from a long list

• Duplicated code
• Large method
• Large class
• Long parameter list
• Dead Code
• Over-generalized code
• Lazy class

7. Use design patterns and abstractions to make code simpler
Learn design patterns. At some level of complexity they will provide excellent solutions and simplify design.
However, wrong abstractions and over used design patterns will make code more complex and less manageable. So, apply them carefully.

8. Decouple and isolate components
Only talk to your immediate friends; Don’t talk to strangers – Law of Demeter.
Reduce connections between programming units – subsystems, modules and objects – as much as possible. More relations between components make software system more complex and rigid. Decoupled and isolated system components have significant benefits:

• better understanding – easier to keep in memory and make sense for less states and relations
• increased reuse – components with minimal well-defined public interface and relations less dependent on context and can be used in many scenarios
• decreased breakdowns – caused by unpredictable run-time combination of relations
• effective testing – highly connected system require much more testing beyond isolated testing of components

9. Keep related code together
Put related code to the same package. It will enhance

• discovery - easier to find and reuse
• design - allows isolation, minimal interactions and consistent structure for related components
• maintenance and testing - reduce effect of new changes to fewer packages and system components.

There are two package approaches:

• Feature – code related to the same problem placed together
• Application layers – code for similar programming concepts

Benefits of packaging:

• Feature: related code in one place, changes together and form self-sufficient granule for reuse and release
• Application: separation of concerns, logical layering and reduced dependencies,  consistent global structure

Composing the program – Putting Things Together

1. Understand what customer wants and form intent.
2. Translate customer ideas into programming concepts (technology, platform and language, UI, libraries, components)
3. Make them simple, minimal and clear with pragmatic approach.
4. Organize code around domain concepts and customer needs; use domain-driven design.
5. Describe UI ideas with paper sketches
6. Specify program ideas with automated unit tests.
7. Merge UI and program ideas
8. Package related ideas together, isolate subsystems and components
9. Express ideas in code and UI screens
10. Remove duplication and bad code smells; improve design; consider using design patterns
11. Evolve and refactor common code into framework; emerge layers and abstractions for similar concepts

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. – Antoine De Saint Exupery

The approach to programming is concerned with finding the best ways to translate programmer’s intention into the good system design and code.

The programming is communication. The programmer continuously add, change and refine ideas in the code. Source code has two important goals: tell a computer what to do and tell people what the computer should do. The program code is the only true medium for storing and communicating ideas about the software system behavior. Quality of the ideas expression in the code directly affects overall quality of the system.

So, what are characteristics of the good code?

1. clear - easier to work with ideas;
2. minimal - less effort to understand and change ideas;
3. testable - easier to validate ideas.

These are 6 top reasons for bad design and code:

1. lack of expertise
2. unrestrained technical curiosity and creativity
3. missing big picture: system purpose and customer goals
4. blindly following popular methods and over-using technology
5. sloppiness; lack of attention to details
6. over-complicating design to have more work or increase job security

The programmer can write better code (and avoid most of these problems) by improving programming style and approach.

Style Components:

• Intention - understand your task and how to get it done
• Approach - basic principles of writing code
• Composition - organization of code
• Expression - expressing ideas in code
• Object Oriented Pragmatic Style

Approach

Two most important rules are

• work from the simplest design
• revise and refactor after each task

1. Work from the simplest design
The simplest design that solves business needs is the best design. The simplest design is the product of discipline and mastery. The simplest design requires creative thinking – it is easier to follow traditional ways than to come up the simple, but clear and powerful solution.

Agile development has famous principle DTSTTCPW (Do The Simplest Thing That Could Possibly Work) best explained by Ron Jeffries:

We’re saying consider all solutions to your task that could possibly work. Implement the simplest solution. Refactor from there if and when needed…  And we’re saying that when you do enhance the simple solution (and you generally will), you will always wind up with a system that is just right for what it does so far. And we’re saying that that is just where you want to be. Everything just right, nothing added that isn’t needed.

For example, if you love design patterns, MVC and object oriented programming, you may end up with design below for finding orders by customer’s postal code.

I love these patterns too, but I would still recommend to start with the simpler design.

The simplest design is the most effective way to deliver business value. You spend less time, build more reliable, easier to understand and change solution. You save energy for other tasks and reduce load on your hard-working brains.

Why do people not go for the simplest solution all the time? There are three main reasons:

• unintentional - the programmer didn’t find simpler solution (lack of experience, pressure and no time to think, relying on traditional ways, compliance with architect’s grand design and others)
• intentional - the programmer wants to over-complicate solution for personal benefits (play with interesting technology, build up resume or increase scope of paid work)
• foretelling - the programmer bets on the future advantage of more complex design

The last reason is the most compelling for many programmers. What if your experience and knowledge tells you that you will need this more advanced design and additional code?
Lets do cost-benefit analysis

What are the chances of mismatch with our initial expectations for a non-trivial project? Big. Mismatch with expectations happens when we learn more from technology, domain and customer feedback while programming. Mismatch happens when we understand the problem deeper and come up with the better solutions. Mismatch happens when the new requirements are coming, priorities are altered and the whole world around is changing (e.g. new company opportunities or credit crunch and economic downturn).

As we can see the best approach is the simplest design that requires little refactoring in the future and doesn’t prohibit anticipated design evolution. Therefore, use you experience and intuition to predict direction of the design evolution. Use this knowledge not for making design more complex to accommodate this prediction, but for ensuring that the simplest solution is not stupid, prohibitive for evolution and will survive coming changes.

However, the level of expertise has significant impact on the choice. The novice programmer should use standard well-described solutions and learn how to simplify them later. The expert programmer should be more comfortable to design simple, sound and easy to evolve solutions from the start.

2. Revise and refactor
Focus on the simplest local design and code for individual features without revisions and design improvements could sometimes create global mess and painful complexity .

For example, it is bad, if after adding several features, your OrderDB contains many methods like LoadForPostalCode, LoadForPostalCodeAndStatus, LoadShippedOrders and web pages are overloaded with domain logic. This kind of code calls for refactoring.

Revise and refactor after completing any programmer tasks.
You should ask yourself 4 questions:

• Can I reduce code?
• Can I make code clearer?
• Can I make code more testable?
• Can I make the whole system simpler?

Refactor, if you answer yes to any of these questions. Learn code smells to identify concrete places that need refactoring. Learn refactoring techniques to improve code.
Main methods:

• remove duplication and increase code reuse

• create abstractions to generalize individual cases
• use design patterns to simplify design

Refactoring is one the most important practice for productive software creation. Refactoring allows to keep the system in the good shape all the time – simple, well designed, ready to change. Implementing the simplest solutions without refactoring and evolutionary approach is risky.

Other rules should enhance your approach further.

3. Discover Form and Function in parallel.
Form follows Function is a traditional principle in design. In other words, customers requirements dictate user interface. However, discovery of complex interactions could affect what the customer wants and change understanding of needs. Sometimes starting with UI paper sketches and discussing them with customers could expose missing details, gaps in understanding and suggest better ways to satisfy customer needs.  Therefore, work on discovery of Form and Function of the system in parallel.

4. Avoid using new tools
New Systems Mean New Problems – The Fundamental Theorem of Systemantics
Do not use new languages, tools and libraries unless existing are bad and you can tolerate high risk and delays.
Every new programming tool claim to make you more productive and powerful. Few of them really do this. But , it is guaranteed, that every new tool takes time for learning and brings new problems.
I know that it is hard to continue using old tools for a long time and stay competitive in fast moving software development world. But try to limit impact on customer. It is not fair to make them victims of your learning and experiments. Create separate experimental prototypes and greenfield projects to evaluate new tools and approaches. Prefer to be honest and transparent with your client about risks of new technologies, tools and approaches.

5. Use creative ideas sparingly
Creative ideas can bring excellent solutions. However, any creative idea adds element of unexpectedness and departure from established ways to solve similar problems. Programmers will spend more time to understand creative ideas and will have challenges to support them. Therefore, use creative ideas when they bring significant advantages (especially to simplify solution) and avoid them in trivial situations. Don’t worry, you will always find where to apply your creative energy if you are productive, build good solutions and have more and more customers.

6. Prefer standard solutions to the non-standard
This rule compliments the previous rule. Standard solutions are based on experience of many people. Maybe they are not the best in your particular case, but they often are safe bet if these problems are new for you. You can find many examples, quickly learn how to apply them and understand consequences beforehand. New creative solutions will make life more interesting in expense of productivity and certainty in the final results. Therefore, prefer standard solutions to reinventing the wheel again. Don’t forget to learn how to make them simpler.

7. Separate production code and experiments
Restrain technical curiosity when writing production code. Many programmers find the joy of programming in solving complex problems. It is so tempting to solve difficult technical puzzles and try out new cool techniques without need. However, the pragmatic programmer solves complex customer problems with simple solutions, instead of creating complex solutions for the simple problems.
Think first about customer requirements and simple design. Experiment and look for the new solutions only if you cannot find simple solutions with familiar methods. Keep your production and experimental code separately. Adopt ideas from experiments without unnecessary complication of the solution.

8. Do not overwrite (or over-eingeneer)
Write minimal code relevant to present customer requests. Surprisingly, complex over-engineered solutions are easier to create – just follow recent trends and try to anticipate everything that came to mind. The simplest and clear solutions require extra effort.
Over-engineering harms the software system, takes away precious time and project money. Avoid over-engineering if you are motivated to build good system and satisfy your customer.

9. Write own code for the core and use external components for supporting functionality.
Concentrate your effort on writing code for the core functions specific to customer domain or code that is bringing competitive advantage. Don’t waste too much effort on secondary supportive functions. For example, there are many good enough libraries, for solving standard dynamic web user interface needs – prototype, jQuery, Microsoft AJAX, Yahoo user interface library and others – but many programmers are still building their own solutions. I like to have full control over my software systems and write as much as possible components, but one cannot embrace unembraceable (Kozma Prutkov). You have to trust some parts of the system to other libraries that will save time and do better job (but bring new problems).

10. Do not write code for future.
The dirty trick of predicting the future is that future is unpredictable. Everything else is fine. Written for the future code adds unnecessary complexity. This code is useless for the present problems. Avoid writing code for the future and strive for the simplest solutions for the present needs.

11. Write code for other programmers
Can other programmers understand your code? Do you have clear names, consistent formatting and self-explaining code? Writing code for others will make code better for you.

12. Switch between big picture and details
Good programmers can focus on the big picture and still pay attention to details. Clear intention and forward thinking are as important as reliable implementation.

13. Pair program for complex problems
Pair programming brings significant benefits:

• better ideas – two brains solving the problems
• better quality – two pair of eyes validating code
• better knowledge – two programmers understand implementation and ideas behind
• better productivity – problems are solved faster
• more enjoyment – people like to communicate and work together

I’m still not sure about serious benefits of pair programming for simple and routine programming tasks. But many Agile teams pair program for all production code and like results.

Putting It All Together

Inspiring reference: The Elements of Style, W. Strunk Jr. and E.B. White

“If you do not change direction, you may end up where you are heading.” – Lao Tzu

Software teams have three main strategies to achieve success: retreat, evolution or revolution.

• Retreat – refusal to act or the art of knowing when to say NO. 
• Evolution – continuous improvement and generation of ideas stemmed from existing set of ideas.
• Revolution – rapid advance with radical and disruptive ideas, overhaul of existing core ideas.

How can software teams choose the best strategy? They should consider three components:

1. The Players
2. The Game
3. The Dynamics

1. The Players
A software team includes programmers, designers, domain experts, system administrators, testers and other active contributors. Each players has 2 main characteristics

• motivation to achieve success
• capabilities (experience, knowledge, skills, ability to learn)

2. The Game
Each project could be considered as The Game where the main goal is successful delivery of software. The game represents various factors: client’s needs and goals, technology, project context, constraints and environment. The game has two main variables:

• novelty – how much players know and understand about The Game, and their ability to make intelligent moves
• complexity – how complex, predictable and controllable is The Game

3. Dynamics
Interaction of a software team with external and internal forces. Characteristics:

• pressure – customers, users, competitors, market, environment
• trend – positive or negative dynamic of the project and software product

Strategies overview

1. Retreat

“A good retreat is better than a bad stand” - Irish sayings

Pros

• saving energy, time and money
• maintain capacity to join the game later under more favourable conditions or move to another game

Cons

• no chances to succeed

How?

• Don’t do anything

When? 

• poor players – lack of expertise, skills or low morale
• too complex, uncertain and overwhelming game
• a project is collapsing or hopelessly loosing positions in a highly competitive environment

2. Evolution

“Evolution is not a force but a process. Not a cause but a law.” - John Morley

Pros

• Reliable mechanism
• Continuous improvement, adaptation to changes and complex environment
• Emergence of unforeseen advantageous traits
• Supports stability and full functioning of the project
• Enables productive focus, collaboration and harmony; keeps morale high

Cons

• Slow, requires many improvement cycles
• Reliance on existing fundamentals and core ideas
• Low chances of emergence of disruptive ideas and breakthroughs

How?
Select working existing ideas and improve them, dump harmful ideas

When? 

• interested and learning players
• complex, unpredictable, but promising game
• moderate pressure, positive dynamic, absence of serious fundamental problems

3. Revolution

 “It is impossible to predict the time and progress of revolution. It is governed by its own more or less mysterious laws.” - Vladimir Lenin 

Pros 

• Fast leap forward, rapid advance
• Quick removal of barriers, fundamental problems and weaknesses
• Overcoming people inertia and group thinking

Cons

• Sporadic, discontinuous, unreliable, risky
• Possible resistance and fear of change
• Provides little time for adaptation and ignores complex environment interactions and feedback loops
• Causes destabilization, temporary chaos in the project

How?
Come up with new crazy ideas, close your eyes and jump to implement them

When?

• the best and highly motivated players
• manageable and well understood game
• radical shift is required to sustain pressure and overcome limits to growth or fundamental problems

The Road to The Best Strategy

As you can see, there is no best strategy for every situation.

Evolution is the most preferable, comfortable and reliable strategy in the complex uncertain world. Few revolutionary ideas will be very successful, come at right time and have significant impact, but most will fail and fade. Most of our achievements are result of evolution than revolution. However, revolution could promote new disruptive ideas, solve fundamental problems and achieve radical shift and advantage. Retreat is a conservative and disaster preventing strategy. A software team can apply these strategies on different stages for different levels of the project.

Software team should think about two goals:

1. The best strategy for the present moment (criteria are discussed above)
2. Actions to enable better strategy in the future

What can a team do to prepare for the better strategy?

1. Grow - increase capabilities and motivation of players:
   • Experiment, make mistakes, acquire knowledge and shape skills. 
   • Consider personalities, group psychology, interests to make people energized and motivated to achieve success.
   • Create stimulus for growth
2. Learn - you cannot actively change the game, but you can learn the game and make it more predictable and manageable. 
   • Apply system thinking to understand core principles, relations and forces.
   • Become an expert, whose knowledge and intuition could suggest the best moves
3. Watch - you should watch the internal and external changes
   • Set up feedback loops (e.g. retrospectives) and intelligence that serve as early warning and prevention system
   • Honestly evaluate existing situation without self-deception and blaming others.
4. Adapt - constantly improve, adjust and respond to become fit for the changing environment, people minds and new opportunities
   • Be Agile
5. Innovate - strive for the new revolutionary ideas that could change the game or at least give you advantage (within tight project budget :)) Maybe new ideas will fail, but they will provide you with invaluable experience and increased capabilities of the team.
6. Remove limits to growth - every process have some bottlenecks that slow growth. Removal of these barriers allow to move faster with better results.
   • Find your limits for growth and start eliminating them.

Limits to growth: reinforcing process is set in motion to produce a desired results. It creates a spiral of success but also creates secondary effects which eventually slow down the success.

Another method of finding fundamental problems is 5 Whys. The 5 Whys is a question-asking method used to explore the cause/effect relationships underlying a particular problem. Ultimately, the goal of applying the 5 Whys method is to determine a root cause of a defect or problem.

External Forces 

A software team has power to change things, but external forces are very strong.

• environment can change
• opponents become stronger
• product idea become useless

Three small case studies show dialectic of evolution and revolution.

Cambrian period - environment triggers revolution within evolution.
The Cambrian explosion was the seemingly rapid appearance of most major groups of complex animals around 530 million years ago. Before about 580 million years ago, most organisms were simple, composed of individual cells occasionally organised into colonies. Over the following 70 or 80 million years the rate of evolution accelerated by an order of magnitude and the diversity of life began to resemble today’s.

Relatively small changes in the environment are considered one of the main reason for the explosion. One of them is increase in oxygen levels. Shortage of oxygen prevented the rise of large, complex animals. An increase in the concentration of oxygen in air or water increase the size to which an organism could grow without its tissues becoming starved of oxygen. 
Raise of complex animals triggered arms races between predators and prey that caused explosion of huge variety of animals shaped by natural selection. Many species become extinct that could be considered as an extreme version of retreat.

Auto industry - winning opponents stop revolution
100 years ago hundreds of automobile companies competed and produced many revolutionary ideas – from steamed cars to battery powered vehicles. Innovation was rapid and rampant, with no clear standards for basic vehicle architectures, body styles, construction materials, or controls. Development of automotive technology was rapid, due in part to a huge number of small manufacturers all competing to gain the world’s attention.

Revolution continued until gasoline-powered cars makers won with a cost effective design, standardized components and better mass production techniques such as an assembly line. After 1930, the number of auto manufacturers declined sharply as the industry consolidated and matured. Starting from 1960 nearly all modern passenger cars are front wheel drive unibody designs with transversely-mounted engine running on oil products.

Fundamental changes in the game (ecology, fuel cost) seem to trigger a new revolution with gas-electrig hybrids, electric, fuel cells, biomass powered automobiles, but this is much smaller revolution than 100 years ago.

Search engines - a simple algorithm and profitable business model renders evolution pointless
From creation of the Internet in 1993 multiple search engines appeared and become popular and faded – Archie, Excite, Yahoo, Lycos, Infoseek, Alta Vista and many others.
Search engines revolution produced many ideas. Yahoo created manual directory. Wanderer deployed web spiders. Excite brought statistical analysis of words. Alta Vista introduced natural language processing and Boolean operators. Inktomi popularized the paid-per-click model.

However, Google won this revolution with Page Rank algorithm counting incoming links as votes with different degree of authority. After adding profitable business model based on pay-per-click AdWords service, Google become unstoppable - 70% of searches in United States with constantly increasing share.
Can anybody beat Google in search? Evolution won’t help here – revolutionary ideas are needed.

Afterword

I don’t know what is the best strategy for your team – there are so many factors to consider. But I know, your team will win if players honestly recognize reality, pragmatically select present strategies and move to enable the best strategies for achieving success in this complex world.

What does produce better ideas in software development – revolution or evolution? Revolution is a rapid triumph of the new ideas and breaking open of the old concepts. Evolution is the process of small frequent changes to improve and adapt to environment. The main difference – revolution replaces old ideas with the new promising unproven ideas, evolution gradually and continuously improves existing working ideas.

We often face this dilemma in software development – should we enhance existing features and improve the ways we work or should we instead come up with something radical and revolutionary.

Revolution!

Could we have our modern cars if people concentrated on improving horses only? Jet planes, Apple iPhone and Quantum Mechanics are products of revolutionary ideas, not improvements of old. Our intellectual creativity and brilliant unexpected ideas are the spark plug for our progress and domination on this planet. This is unique and most beneficial human trait.  Probably computers will never become truly intelligent, because they cannot create new revolutionary ideas and can only use logical reasoning based on initial programmed set of ideas (see Gödel Incompleteness Theorem).

The best revolutionary (disruptive) ideas give you enormous advantage over competitors. You move to the new unsettled territory with the new framework of ideas, leaving far behind people with old ideas . Incremental evolutionary changes will not allow you to change situation fast and overhaul hindering ideas.

Computer mouse, World Wide Web and MapReduce are the outcomes of revolutionary ideas. Google innovative ideas enable it domination in search.

Revolution is the fastest way to move forward, but outcome is very dependent on the quality of ideas.

Revolutions could be risky and even dangerous. Great revolutionary ideas will move us to a much better place, bad revolutionary ideas will throw us to the hell. Revolutionary ideas could be just unreal fantasies. The big leap forward doesn’t use invaluable feedback from the environment and learning from own results – there is no much time to correct something in the jump. These ideas are pushed by strong individuals, whose energy and conviction could blind rational and objective minds and lead to authoritarianism and destruction.

Finally, revolution cause instability, resistance and uncertainty that have negative impact on the system and team dynamic for some time.

Revolution is the matter of chance, there are so many things should come together right for successful revolution even if it is based on the great ideas. And it is really difficult to constantly and reliably come up with revolutionary ideas.

Or Evolution?

We, humans, are the product of evolution and I think it is a strong enough evidence that evolution works. Evolution introduce small incremental changes, improve beneficial and weed out harmful traits in interaction with the environment. Highly fit, adaptive and efficient solutions come as a result.

Evolution’s strength is reliable, systematic and effective mechanism to improve and advance in the complex and unpredictable world.

Retrospectve is evolutionary mechanism for improving agile teams work. Refactoring and eliminating of code smells is mechanism for improving the code. User focus groups and marketing research are the mechanisms for improving product ideas.

We cannot decline the fact that evolution is creative too, but in another way than revolution. Evolution could create completely new solutions in the long run by selecting, improving, and optimizing existing traits.  Biological evolution provides some examples:

1. Creating global solutions from optimization of local. The first hard mineralised structures to evolve in our ancestors were the teeth of early fishes known as conodonts. Once the ability to form hard hydroxyapatite had evolved, it could be exploited elsewhere in the body and have been the basis of the bony skeletons of all vertebrates [link].
2. Emergence of the new features. Our mind is emergent result of the mammals neocortex development for better evaluation of probabilities and predicting the future. The lobed fins of early fish have turned into structures as diverse as wings, fins, hoofs and hands.
3. Repurposing - Reptilian jaw bones turned into mammalian ear bones, without the loss of the jaw. The neural circuitry that allows us to make fine limb movements have been adapted to produce speech as well.

It is similar in software – new ideas come up when we fight for optimization, refactoring produces new better components and we reuse well crafted libraries for the new unforeseen tasks.
There is a limit for evolution creativity - solutions should have immediate practical purpose. Some ideas could be never reached with incremental improvements. For example, two-way radio won’t be developed evolutionary as one-way radio is useless, improvements of the best horse breeds will never lead to the powerful cars and improvements of propeller planes will not lead to the jet planes. These breakthroughs need revolutionary ideas.

Evolution main deficiency is slowness. Modern businesses often don’t have time for many improvement cycles – they want results now. They are willing to risk, speculate and go with revolutionary ideas than patiently wait for evolution.

Another reason for preferring revolution over evolution is the scale of necessary changes and seriousness of the problem – sometimes it is more effective to implement new ideas than evolve old ones.

What is better?

Dear readers, what do you think is better – revolution or evolution? Is it possible to combine both? Microsoft, Google, Yahoo, Facebook, Apple – which companies are revolutionary and which are evolutionary? I’ll continue topic about revolutionary and evolutionary ideas in the next post.