The manager slams a door and tells us that we are in a big trouble. Our old customers complain about many bugs and bad performance, new customers complain about delays and lack of dedication. And, top management considers our department financially unsustainable and wants to deeply cut expenses.
The manager tells that we are brilliant programmers, work very hard and create cool software solutions. But there is something wrong and we cannot work this way anymore.

Anxiety started to penetrate our souls. We know what is wrong: our team is short of people, we have too many commitments, our code is becoming a big mess, new technology and our new software version makes something bad with servers. A snowball of different problems makes us stressed, distracted and incapable of productive work.

What could our manager do next?

1. Distrust. Become a dictator, make own decisions including hiring external consultants to recommend what to do or even replace us. However,
   • we are good programmers and know our business well – the problem is not in lack of skill and knowledge
   • external people will take a lot of time to understand the system and they will have different motivation and won’t care about the long-term success
   • people will be demotivated and the manager cannot make effective decisions without active team involvement
2. Faith. Give to team the full power to fix a problems and make own decisions in hope that smart people, motivation and technical expertise will do magic. However,
   • fresh outlook and thinking out of box are hard when a team immersed for a long time into difficult situation
   • a team possibly doesn’t have understanding and control over external forces – management, customers, finances
   • changing of reality is tough (especially in people heads) and requires more than technical experience

There is a third way. Place improvement practices in the core of development process. Make self-improvement inevitable and required for any activity. Do it every day.

Toyota Way is the best example of large-scale reliable self-improvement process. It focuses on eliminating waste, solving problems at root cause and making right decisions. Toyota Way reduces problems, increases internal efficiency and makes a company successful. This is the best receipt for coming out of crisis.

Targets:

• Problems - emergencies, fires that require immediate fix: bugs, server crushes, deadline slips

• Waste - inefficient and non-value adding activities: waiting, misinformation, stress

• Challenges - adaptation to external forces (market, competitors, customers, society): new trends and technologies, changes in users expectations for user interface and functionality

Practices to see waste and stop to fix problems

1. Seeing waste

The team and managers should learn to see targets – real problems and waste. Otherwise improvements will be wild shots in the dark.

There are many targets in software development:

• stressed people – reduced energy, less productivity, more mistakes
• waiting – delays, tools / system problems and downtime, capacity bottlenecks (waiting for results of other people work)
• over-engineering – producing features and complicate design without real need
• unfinished work – functionality not used in a live system, probably still in design or under development or simply discarded (but not removed)
• defects – complete waste of time and money
• unused creativity – loosing time, ideas, skills, improvements and learning by not engaging or listening to your employees
• inadequate information – unclear, misleading or simply wrong information that causes useless activity and leads to rework at the end

• over-processing – taking unneeded steps to build software because of poor process and system design, overhead, bureaucracy, compliance, cumbersome tools
• motion – how much effort to get necessary information, access systems or use tools
• multi-tasking – losing time to switch between projects, tasks or different activities

2. Jidoka - stop to fix the problem to get quality right the first time

It is not enough to see your targets. The Team should carry commandment to shoot targets immediately. Otherwise the best intentions will be buried under growing avalanche of problems.

How to stop:

• quality for the customer drives all the processes – prevents temporary patches and bad for quality decisions
• low tolerance for quality problems and immediate detection are core work principle
• use Andon – a system to signal for help, notify about a problem and stop the process

• build in tools and process capabilities of detecting problems and stopping itself
• use all modern QA methods available
• managers encourage stops to fix problems and support implementation of counter measures

Making Decisions

“Make decisions slowly by consensus and thoroughly considering all options, implement rapidly.“ – Toyota Way

Even knowing problems and committed to solve them, the Team should make right decisions how to do it.

Make decisions slowly

1. Go and See (Genchi Genbutsu)
2. Understand underlying causes (Kaizen)
3. Broadly consider alternative solutions and develop a detailed rationale for preferred solution delaying certain decision as long as possible (Set-Based concurrent engineering)
4. Build consensus within a team and partners where group decision is preferred (while management can step in if consensus is not achieved)
5. Use very efficient communication devices – preferably one side of one sheet of paper

Implement rapidly

1. Put in place solutions and counter measures.
2. Evaluate the results
3. Standardize if solution is effective.

Practices to Eliminate Waste and Solve Problems

3. Genchi Genbutsu – go and see for yourself to thoroughly understand situation

How often do we jump to conclusion based on partial information, vague assumptions and what other people say? Information creates reality in your mind. This reality is a base for your decisions. So, you and the Team should get right information to make right decisions:

• observe situation with blank mind
• avoid assumptions and preconceptions
• use personally verified information

In short, base decisions on what really is going on

4. Kaizen (5 why’s) -continuous learning and improvement

“We view errors as opportunities for learning” – Toyota Way
The Team should find the root causes of the problems. Kaizen helps to find the root cause by repeatedly asking why the problem occurs.

Example of Kaizen
Problem: there are persistent javascript errors on a live site

1. Why? A developer didn’t build correct logic for complex web UI components interaction
2. Why? A developer built own solution without guidance and enough experience in this area
3. Why? A team expert didn’t tell about existing proven solutions, didn’t help and didn’t share knowledge
4. Why? The team is under stress, over-committed and don’t have time to communicate
5. Why? Managers accept too much work without consulting with development team
6. Why? you can continue…

Kaizen forces us to overcome desire to find a first convenient explanation and patch problems without fixing root causes. By ruthlessly applying this practice, we get deeper insight into reality and better learn our product, processes, people, environment and tools. Kaizen is a core practice to see waste, solve problems and improve process.

To avoid forgetting learning from Kaizen, it is important to standardize the improved process and make it a base for further improvements.

Practices to Support Flow

5. Standards – best you know today which is to be improved tomorrow

Standardized work is easier, cheaper and faster – stable repeatable methods can maintain predictability, high productivity and support quality.
Effective standards are not coming from theories, they come from

• best practices
• accumulated learnings and individual experience
• lessons from applying existing standards

The Team should try to use standards in many areas: project phases and activities; development practices; architecture and design approaches; code conventions; tools; programming techniques; libraries and third-party code; reuse of components and solutions; testing and so on.
Standardization in software development is a controversial topic – some theorists want to bring programming closer to standard-dominated engineering, practitioners are keen to reduce standardization to minimum promoting creativity and self-organization. In the rigid interpretation, standards are “must to follow” rules for any situation, in other interpretation standards are well defined steps and guidelines highly recommended for specific context. I support the latter definition. A productive team should have standards in place to focus on customer needs instead of fighting with the same puzzles and problems over and over again.

The system of standards shouldn’t be a heavy bureaucratic conduit, but a light and fluid book of knowledge. The book that contains most helpful and important rules and checklists. Standards will be effective if they are minimal, reviewed often (Kaizen) and followed by every team member.

6. Reliable thoroughly tested technology

The Team should be conservative with new technologies. Software development and IT thrive on change and innovation. However, Toyota Way suggests to be conservative in adapting technology and considers stability and reliability of operations as much more important goal than keeping on the cutting edge of technology.

Considerations for using technology

• primary goal is to improve flow and support people, process and values.
• process is driven by business, not technology concerns; software and tools do not eliminate themselves waste
• technology is visual and intuitive – people can use it correctly and effectively

• process manually before adapting technology to support the process – understand what problems it solves and how technology could help
• important: people / processes are easy to Kaizen, machines are difficult

Adopting new technology:

• new technology is unreliable and difficult to standardize, therefore it endangers flow
• proven process takes precedence over new and untested technology
• conduct actual tests before adapting new technology
• reject technology if it conflicts with culture or might disrupt stability, reliability and predictability

In the same time encourage people to consider new technologies while looking into new approaches. If technology improves process and flow – quickly implement after thorough testing.

7. Visual Controls

The Team should have clear status of information. Visual controls can convey complex information in fast and efficient for our brains way. We can use controls as a user story board; status of projects, servers or code build; burn down charts and others.
Simple visual indicators help people determine immediately whether they are deviating from the standards, provide quick gist of situation and direction for solving problems.

• use simple and most important indicators

• than provide clear picture for decisions and what to do next
• reduce reports to one screen / piece of paper even for the most important decisions

People, leaders and teams

8. People

People who build software are the people who should improve the process. They are directly involved and have first-hand experience of problems and waste.

Toyota Way expects that each team member is a problem solver and values job experience more than theoretical knowledge. The Team will beat any external consultants and find better way to work if people are open about problems and eager to find good solutions.

9. Leaders

The Team desperately needs strong leaders to build great products to overcome problems. Toyota grows leaders within who thoroughly understand the work, live the philosophy and must understand the daily work in great details

Chief Engineer is a key person in Toyota projects.:

• blessed by top management
• has control over project
• exceptional engineer
• critical link: between engineers and customer satisfaction
• coach for other engineers
• focus on concepts first, technicalities later

Chief Engineer concept is an excellent example for software technical leadership. A software team leader often lacks authority or makes too technical decisions without good understanding of customer needs.

10. Teams

The Team should be diverse and capable of solving wide range of problems. Toyota builds cross functional product teams, which

• use integrative decision making
• fast and accurate in implementation
• enhance process and flow by solving difficult technological problems

Software developers and their leaders are foundation of success in any project. Management, process and technology can only support them. And anyway, the process is as good as people follow it. Therefore, it is important to make software teams a key player in process improvements – they know problems, they understand work and they are capable to find good solutions.

Using Toyota Way

Can The Team revert a situation and win? Can it build the optimal process and expertise for fast development of high quality and low cost solutions?

This post shows the most effective option – build continuously improving process into the heart of development. The process that focuses on quality, eliminates waste and fixes problems at root cause. I believe this approach is a foundation of long term success. Your managers and company would love it!

A software project is a creative, unique and therefore unpredictable endeavor. We are not building the same thing over and over again, but solve new problems, address increasing demands and use perpetually changing technologies. Under these conditions, people – smart, creative and productive – are the most important factor of success . Software development process can only support and compliment these people, but it cannot guarantee success alone and make the factor of people negligible.

But, business wants predictable, reliable and successful results. I bet they don’t want to be at mercy how cards are shuffled in their talented development team. The answer is in establishing a process that increases chances of success and aligned with present nature of software development (unpredictable, empirical and heavily dependent on people).

The Toyota Way can be a great example that worth to learn. Toyota evolved from a small looming equipment shop to the largest car manufacturing company. The main foundation of successful growth is the system of few core principles that enables best quality, high productivity, lowest cost, shortest time and long-term success.

Agile and Lean Software Development adopted some practices from Toyota. Lean Software Development: An Agile Toolkit  (by Mary Poppendieck, Tom Poppendieck) is an excellent book that review Lean practices and principles in details. This post focuses more on philosophy and the system of practices as a whole that can turn software development team into a smart reliable machine.

1. Value

Why customer will pay money for your product? How does your company creates value?

All we are doing is looking at the time line from the moment the customer give us an order to the point when we collect the cash. And we are reducing that time line by removing the non-value-added wastes.

- Taiichi Ohno, founder of TPS

The only thing that adds value in software development is transformation of information and code into what customer wants. It is very important for software team to know how does it create value for customers. Or simply, decrypt the formula:
Customer (needs) -> Cash (delivered software)

Software Development Value Stream
Problems, needs and ideas are translated into units of development – functionality, user experience (interface, interactions, usability) and system qualities (performance, reliability, security, etc). A software development team transforms these pieces of requirements into a computer system using value-added and many not-so-value-added actions.

The ultimate goal of Toyota Way is to eliminate non-value-added activities while keeping the best quality of the product.

2. Flow

What is the process of creating value in your company?

Properties of the optimal flow for software development:

• cut back to zero the amount of time that any unit of work is sitting idle or waiting for somebody to work on it
• move design, code and information fast, link processes and people together that problems surface right away

One-piece flow
The most effective flow is one-piece flow – a customer need is immediately converted into a delivered software solution. In the same time this is the most demanding flow: a customer, designers, developers, testers and system administrators should be dedicated to the project, immediately available and work as one team together until the need is implemented in a live system.

One-piece flow has significant benefits (if it is implemented right):

• Quality - defects detected quickly; close proximity and connections ensure that necessary information is available immediately and important details are not missing; problems are solved quickly
• Flexibility - flexibility to respond and do what customer really wants in real time
• Productivity - the team engaged in high value added continuous work
• Morale - immediate results of work bring sense of accomplishment and job satisfaction
• Cost - lean production with minimal waste

However, in reality one-piece flow is difficult to achieve in many projects:

• a customer is not engaged all the time
• complex business domain requires analysis and preparation before development can start
• project activities involve people with different skills that are not available immediately
• deployment of mission-critical systems demands intensive testing and scheduling

• management, marketing, end-user expect specific time-lines, features and support

• and so on

Toyota uses two additional principles to overcome challenges of one-piece flow: pull systems and leveling out the workload.
However, Toyota Way says: “Flow where you can, pull where you must“. The big challenging goal is to turn eventually all the processes into one-piece flow.

Pull system (Kanban)
Kanban is a signaling system to trigger action. It adds to flow small buffers pulled by customer demand. Kanban allows optimal use of people and natural breaks in processes.

Agile iterations and Scrum sprints are close to Kanban ideas. A team pulls new user stories every time-boxed iteration from the backlog to design, implement, test and deploy within the same iteration.

However, it is not classical Kanban. An original idea would be to pull small batches of user stories for development after a team has finished a current batch (not at the end of fixed time iteration). In parallel, QA pulls implemented user stories for testing and so on. A customer can continue preparing next batches of work based on their priority of needs.

Main ideas

• provide dependent down stream teams with what they want, when they want and in amount they want
• just-in-time – started by high priority needs
• minimize work-in-progress and untested code
• frequently push what a customer can use right away
• be responsive to day-by-day shifts in demand instead of relaying on long-term schedules

Kanban is effective way to synchronize people and teams work without bringing them into continuous flow, where they should be always ready for the next unit of work.

Level Out the Workload (Heijunka)
The main idea of Heijunka is to level production by volume and activities mix.

• it is not built around actual flow of customer orders
• workload is leveled for the period considering previous history and scheduled projects

In software development workload could be leveled and scheduled when people could make reliable guess for what is expected. It is different from Kanban that works as just-in-time system adapting to the current workload.

Good candidates for Heijunka are

• standard procedures (automated testing, performance testing, deployment)
• recurring events (meetings, planning, demo, code reviews)

• scheduled tasks for people with limited availability (external consultants, customers, management)

• tasks with high confidence in estimation, usually done many times before (initial requirement exploration, design mockups, security audit, training new team members, reusing components and solutions)

Benefits:

• predictability and active planning

• balanced use of people, ability to schedule their involvement

• smoother demand on customers, shared resources and contractors

Toyota Way is focused on eliminating waste of non-value added work (Muda). Heijunka eliminates two additional types of waste:

• unevenness in project schedule (Mura)
• overburden of people and systems (Muri)

Building your own flow

These three approaches can help you to build most optimal flow. Consider cons and pros:

One-piece flow (push)
pros: minimal waste, fast implementation, low cost, quick problems resolution
cons: uneven workload, requires full availability and high level of engagement
when to use: for one collocated team work that is dedicated to one project

Kanban (pull)
pros: dynamic adaptation to load; synchronization of different teams and disconnected processes
cons: low predictability for completion; buffers hide waste
when to use: disconnected or involved in the multiple projects teams; activities that need people with limited availability

Heijunka (level)
pros: good predictability, ability to plan, balanced use of people: avoid overburden and uneven workload
cons: non-adaptive, inefficient for tasks with unpredictable effort (a lot in traditional software development)
when to use: activities that are standardized, highly predictable or have to be scheduled

You cannot build reliable and optimal flow from the scratch based on a theory only. The best process for your projects will emerge as result of evolution, problem solving and eliminating waste. This is a topic of the next post.

If some problem occurs in one-piece flow manufacturing then the whole production line stops. In this sense it is a very bad system of manufacturing. But when production stops everyone is forced to solve the problem immediately. So team members have to think, and through thinking team members grow and become better team members and people.

- Teruyuki Minoura

References:
The Toyota Way by Jeffrey Liker
Lean Software Development: An Agile Toolkit  by Mary Poppendieck, Tom Poppendieck

Pair Programming is a great way to build software systems. When Pair Programming works, it has significant benefits:

• better ideas – continuous brainstorming, larger knowledge pool, less gaps in understanding and more brain power to solve design problems;
• better quality – fewer bugs, instant validation of ideas, consistent approach and stricter adherence to team conventions;
• better knowledge – experience and knowledge sharing, deeper understanding of why, how and what was done;
• increased productivity – better focus and higher intensity, pushing each other and motivating to achieve best results, less procrastination and wasting of time;
• more enjoyment – most people like to work in groups and solve together interesting problems.

Extreme Programming leaders insist that all significant development should be done in pairs. But can we say that Pair Programming is the best method in all situations?

Programmers can find plausible alternatives to Pair Programming that don’t require two people sitting together all the time:

• ideas - frequent team brainstorming combined with short pair (or team) programming sessions for the most complex tasks;
• quality - dedicated tester closely works with developers, writing together automated tests;
• knowledge - frequent discussions, code reviews, training sessions;
• productivity - clear intent and pragmatic approach that bring focus, clarity and efficiency;
• enjoyment - close cooperation and mutual support

When Pair Programming is the most effective method?

The most important factor is match of skills and challenges. In Solo Programming the most productive mode is Flow when skills and challenges are matched. Pair Programming adds one more effective mode – Coaching that increases overall team productivity for the present and future tasks.

Successful modes
1. Flow – two programmers work on the interesting and challenging problem. They can have different skills and challenges, but both are capable to find good solution. For example, one can be javascript expert, another super backend programmer. They can get in flow working together on complex AJAX task combining their brainpower, knowledge and experience to create the best solution.
2. Coaching – expert programmer have experience and knowledge for solving the problem and shares it with other programmer, who cannot effectively solve problem alone. This more junior programmer has fundamentals to understand the solution and implementation. He learns and grows to become better programmer.

Failure modes
3. Wasting expert time – the problem is too simple that makes involvement of expert ineffective even for coaching
4. Overwhelmed novice – the problem is too complex or requires too much new knowledge that prevents the programmer from learning anything useful

Questionable modes
5. Two experts for a manageable task – pair programming doesn’t have much benefits if both programmers know how to implement the task and successfully solved similar problems before.
6. One programmer is in Flow, other is Learning – it is difficult to stay in Flow if other programmer frequently interrupts and needs many explanation for basic things that are not directly related to challenging problem.
7. One programmer is in Flow, other is Coaching – to make this mode successful for the growth, the coach should be open minded, avoid directing too much and should give opportunity for another programmer to come up with own (maybe even better) solutions.

In addition, psychological problems could prevent successful pair programming:

• Intimidation by expert – one programmer is intimidated by superior skills of other programmer and afraid to be considered as incapable
• Need time to think – a programmer needs more time to think over before pair programming, but he is forced to start pair programming session before thinking over his ideas
• Prefer to work alone – an introvert programmer likes to work alone (loner).
• Personal antipathy – programmers don’t like each other

Can we make Pair Programming always effective?
Yes! Match tasks, skills and pairs. Try to find pairs for 2 productive modes  – Flow and Coaching. A team will be most productive if the programmers pair solves together interesting problems matching their skills and learn from each other.

However, Pair Programming should be a free choice and preferred method, but it should not be a forced mandatory practice. As you can see from this post, there are some modes and psychological situations when Pair Programming is not working well.

In short, Pair Programming is one of the most useful tools in arsenal of the software team. Know when and how to use it.

Inspiring resource: Flow, Mihaly Csikszentmihal

“If you do not change direction, you may end up where you are heading.” – Lao Tzu

Software teams have three main strategies to achieve success: retreat, evolution or revolution.

• Retreat – refusal to act or the art of knowing when to say NO. 
• Evolution – continuous improvement and generation of ideas stemmed from existing set of ideas.
• Revolution – rapid advance with radical and disruptive ideas, overhaul of existing core ideas.

How can software teams choose the best strategy? They should consider three components:

1. The Players
2. The Game
3. The Dynamics

1. The Players
A software team includes programmers, designers, domain experts, system administrators, testers and other active contributors. Each players has 2 main characteristics

• motivation to achieve success
• capabilities (experience, knowledge, skills, ability to learn)

2. The Game
Each project could be considered as The Game where the main goal is successful delivery of software. The game represents various factors: client’s needs and goals, technology, project context, constraints and environment. The game has two main variables:

• novelty – how much players know and understand about The Game, and their ability to make intelligent moves
• complexity – how complex, predictable and controllable is The Game

3. Dynamics
Interaction of a software team with external and internal forces. Characteristics:

• pressure – customers, users, competitors, market, environment
• trend – positive or negative dynamic of the project and software product

Strategies overview

1. Retreat

“A good retreat is better than a bad stand” - Irish sayings

Pros

• saving energy, time and money
• maintain capacity to join the game later under more favourable conditions or move to another game

Cons

• no chances to succeed

How?

• Don’t do anything

When? 

• poor players – lack of expertise, skills or low morale
• too complex, uncertain and overwhelming game
• a project is collapsing or hopelessly loosing positions in a highly competitive environment

2. Evolution

“Evolution is not a force but a process. Not a cause but a law.” - John Morley

Pros

• Reliable mechanism
• Continuous improvement, adaptation to changes and complex environment
• Emergence of unforeseen advantageous traits
• Supports stability and full functioning of the project
• Enables productive focus, collaboration and harmony; keeps morale high

Cons

• Slow, requires many improvement cycles
• Reliance on existing fundamentals and core ideas
• Low chances of emergence of disruptive ideas and breakthroughs

How?
Select working existing ideas and improve them, dump harmful ideas

When? 

• interested and learning players
• complex, unpredictable, but promising game
• moderate pressure, positive dynamic, absence of serious fundamental problems

3. Revolution

 “It is impossible to predict the time and progress of revolution. It is governed by its own more or less mysterious laws.” - Vladimir Lenin 

Pros 

• Fast leap forward, rapid advance
• Quick removal of barriers, fundamental problems and weaknesses
• Overcoming people inertia and group thinking

Cons

• Sporadic, discontinuous, unreliable, risky
• Possible resistance and fear of change
• Provides little time for adaptation and ignores complex environment interactions and feedback loops
• Causes destabilization, temporary chaos in the project

How?
Come up with new crazy ideas, close your eyes and jump to implement them

When?

• the best and highly motivated players
• manageable and well understood game
• radical shift is required to sustain pressure and overcome limits to growth or fundamental problems

The Road to The Best Strategy

As you can see, there is no best strategy for every situation.

Evolution is the most preferable, comfortable and reliable strategy in the complex uncertain world. Few revolutionary ideas will be very successful, come at right time and have significant impact, but most will fail and fade. Most of our achievements are result of evolution than revolution. However, revolution could promote new disruptive ideas, solve fundamental problems and achieve radical shift and advantage. Retreat is a conservative and disaster preventing strategy. A software team can apply these strategies on different stages for different levels of the project.

Software team should think about two goals:

1. The best strategy for the present moment (criteria are discussed above)
2. Actions to enable better strategy in the future

What can a team do to prepare for the better strategy?

1. Grow - increase capabilities and motivation of players:
   • Experiment, make mistakes, acquire knowledge and shape skills. 
   • Consider personalities, group psychology, interests to make people energized and motivated to achieve success.
   • Create stimulus for growth
2. Learn - you cannot actively change the game, but you can learn the game and make it more predictable and manageable. 
   • Apply system thinking to understand core principles, relations and forces.
   • Become an expert, whose knowledge and intuition could suggest the best moves
3. Watch - you should watch the internal and external changes
   • Set up feedback loops (e.g. retrospectives) and intelligence that serve as early warning and prevention system
   • Honestly evaluate existing situation without self-deception and blaming others.
4. Adapt - constantly improve, adjust and respond to become fit for the changing environment, people minds and new opportunities
   • Be Agile
5. Innovate - strive for the new revolutionary ideas that could change the game or at least give you advantage (within tight project budget :)) Maybe new ideas will fail, but they will provide you with invaluable experience and increased capabilities of the team.
6. Remove limits to growth - every process have some bottlenecks that slow growth. Removal of these barriers allow to move faster with better results.
   • Find your limits for growth and start eliminating them.

Limits to growth: reinforcing process is set in motion to produce a desired results. It creates a spiral of success but also creates secondary effects which eventually slow down the success.

Another method of finding fundamental problems is 5 Whys. The 5 Whys is a question-asking method used to explore the cause/effect relationships underlying a particular problem. Ultimately, the goal of applying the 5 Whys method is to determine a root cause of a defect or problem.

External Forces 

A software team has power to change things, but external forces are very strong.

• environment can change
• opponents become stronger
• product idea become useless

Three small case studies show dialectic of evolution and revolution.

Cambrian period - environment triggers revolution within evolution.
The Cambrian explosion was the seemingly rapid appearance of most major groups of complex animals around 530 million years ago. Before about 580 million years ago, most organisms were simple, composed of individual cells occasionally organised into colonies. Over the following 70 or 80 million years the rate of evolution accelerated by an order of magnitude and the diversity of life began to resemble today’s.

Relatively small changes in the environment are considered one of the main reason for the explosion. One of them is increase in oxygen levels. Shortage of oxygen prevented the rise of large, complex animals. An increase in the concentration of oxygen in air or water increase the size to which an organism could grow without its tissues becoming starved of oxygen. 
Raise of complex animals triggered arms races between predators and prey that caused explosion of huge variety of animals shaped by natural selection. Many species become extinct that could be considered as an extreme version of retreat.

Auto industry - winning opponents stop revolution
100 years ago hundreds of automobile companies competed and produced many revolutionary ideas – from steamed cars to battery powered vehicles. Innovation was rapid and rampant, with no clear standards for basic vehicle architectures, body styles, construction materials, or controls. Development of automotive technology was rapid, due in part to a huge number of small manufacturers all competing to gain the world’s attention.

Revolution continued until gasoline-powered cars makers won with a cost effective design, standardized components and better mass production techniques such as an assembly line. After 1930, the number of auto manufacturers declined sharply as the industry consolidated and matured. Starting from 1960 nearly all modern passenger cars are front wheel drive unibody designs with transversely-mounted engine running on oil products.

Fundamental changes in the game (ecology, fuel cost) seem to trigger a new revolution with gas-electrig hybrids, electric, fuel cells, biomass powered automobiles, but this is much smaller revolution than 100 years ago.

Search engines - a simple algorithm and profitable business model renders evolution pointless
From creation of the Internet in 1993 multiple search engines appeared and become popular and faded – Archie, Excite, Yahoo, Lycos, Infoseek, Alta Vista and many others.
Search engines revolution produced many ideas. Yahoo created manual directory. Wanderer deployed web spiders. Excite brought statistical analysis of words. Alta Vista introduced natural language processing and Boolean operators. Inktomi popularized the paid-per-click model.

However, Google won this revolution with Page Rank algorithm counting incoming links as votes with different degree of authority. After adding profitable business model based on pay-per-click AdWords service, Google become unstoppable - 70% of searches in United States with constantly increasing share.
Can anybody beat Google in search? Evolution won’t help here – revolutionary ideas are needed.

Afterword

I don’t know what is the best strategy for your team – there are so many factors to consider. But I know, your team will win if players honestly recognize reality, pragmatically select present strategies and move to enable the best strategies for achieving success in this complex world.

Recently I was reviewing requirements for Microsoft Architect Certification and had found description for architect roles. After reading I had a question: do we need this kind of architects in a software team? Can they really help with building successful software?

The summary of architect responsibilities (as per Microsoft):

• Enterprise architects: set the overall vision and framework for the IT environment from a strategic business perspective.
• Solutions architects: design the solution to take advantage of the existing assets, integrate them into the existing environment, follow the enterprise architecture, and solve the business problems of the business owner or unit.
• Infrastructure architects: responsible for creating an architecture that meets the business and service level agreement requirements of the business owners and supports the applications and solutions that are required to operate their day-to-day businesses.

The bottom line is that company hires expensive and canny software architects before start of the project. They talk with the business, come up with solutions and make bright key decisions about architecture. But they don’t program. During the course of the project they ensure that developers don’t spoil this great architecture.

Now business owners can hire not so bright programmers, who should just follow directives and implement this great architecture. There is even correspondence with theories of some management consultants that programmers are just house painters.

Completely irrelevant questions: Can you trust advise of your friend about Paris, if he saw the city only on TV? Can you trust battle plans if battle didn’t begin and your commander knows little about enemy? How many chess moves ahead can you predict with high certainty before start of the game?

I don’t reject a need in architecture, but I don’t agree who, when and how should make architecture decisions in a software project.

Top 10 reasons why you don’t need Software Architect

The Architect (dedicated non-programming technical decision maker and problem solver for business):

1. Has outdated programming knowledge and experience, loss of touch with modern development approaches and practices.
2. Don’t program and don’t know much about evolving system internals, but makes key technical decisions. Often has completely irrelevant and unreal picture what is happening with the system.
3. Tends to complex, premature and generic solutions when the system is still in infancy and nothing is clear. Applies latest modern buzzword technologies as SOA, MDA, SaaS, Software Factories, etc. which look so beautiful in technical magazines, conferences and CV, but cause unnecessary headache for developers.
4. Plays role of the middleman introducing complexity in coordination and project responsibilities. Represents software team in interactions with business customers reducing communication value for the rest of the team and impacting idea flow.
5. Reduces quality of decisions, which become limited to one perspective; decision making starts lacking diversity, independence and decentralization, which are essential attributes of collective intelligence.
6. Creates tension with developers who experience mismatch between grand design and reality. Often continues pushing design decisions until the system becomes overly complex, difficult to change and becomes completely unusable.
7. Secures job and justifies high salary – becomes authoritative center for solving business problems without much input from the team.
8. Causes loss of sense of ownership, motivation and accountability in developers by detaching them from the key architecture decisions.
9. Concentrates project knowledge and the big picture in one head, limiting (and sometimes preventing) complete understanding for others.
10. Contributes to creation of specialized IT verticals that hurt relations with the business.

I’m sure that some architects bring strong technical leadership, excellent solutions and overall project success. However, the architect role itself, as described by Microsoft and employed by other companies, doesn’t encourage productive development and effective solutions.

What is effective architecture? Who is effective architect?

Emerging and evolutionary architecture is a core for a successful agile (and not only) software system, emerging from the implementation of business needs, learning from working code, problems and interactions with people, other systems and operational environment.

Effective architecture:

1. Provides stable foundation and integrity for the growing software system, enabling desired system qualities for the business solutions.
2. Ensures seamless and consistent integration, well defined interfaces and interaction between subsystems, external systems and operational environment.
3. Supports emerging abstractions, key system elements and frameworks for conceptual integrity, efficiency and reuse.

Effective architect:

1. Guards a system against failure, sensing worrying changes in the project dynamic, system code and business requests.
2. Keeps the trinity of system qualities (known from time of Roman empire) in a balance and prevent degradation and entropy:
   • Firmitas (Strength) – the system is reliable, secure, has good performance and easy to support
   • Utilitas (Utility) – the system is usable, meets business business needs and add business value every day instead of drifting into technology trenches
   • Venustas (Beauty) – code and system structure are clean, easy to understand, minimal
3. Encourages constant improving of the code design, enhancing system abstractions and structure, removing duplication, defining boundaries and interfaces of the subsystems.
4. Keeps solutions as simple as possible, maintains intellectual control over system and avoids over-engineering.
5. Most important – grows and coaches other developers to become architects

If your company can afford dedicated non-programming technical person, hire an expert for complex technical areas, a coach to share best practices and experience or a technical supporter to coordinate with different external groups and help to resolve problems and make them all part of the team.

But don’t take technical decisions and concern about architecture from the development team.

The ultimate solution – every developer covers these architect responsibilities. Hire few experienced, motivated and capable developers and start project with them. Developers will constantly evolve the system based on real needs and shape effective architecture. You can trust that system will be healthy, meet business needs and bring success. Often only few important architecture decisions should be made before start of the project – trust them to the developers, who will implement the system.

So, here is the answer regarding who, when and how should make architecture decisions:

Who – every developer timely makes architecture decisions and constantly improves evolving architecture
When – mostly after start of implementation, when system becomes more complex and need design improvements to maintain system qualities, intellectual control and conceptual integrity.
How – preventing degradation and entropy by making design changes and refactoring; learning from the project dynamic, working code and business requests and adjusting a way how the development team works.

Do you still think that software projects need The Architect?

Useful resources:

Microsoft Architect Certification

Is Design Dead?, Martin Fowler

Big Ball of Mud, Brian Foote and Joseph Yoder

Marc Andressen quotes Frans Johansson’s book “The Medici Effect”:

In a [1987 study, researchers] concluded that brainstorming groups have never outperformed virtual groups. Of the 25 reported experiments by psychologists all over the world, real groups have never once been shown to be more productive than virtual groups. In fact, real groups that engage in brainstorming consistently generate about half the number of ideas they would have produced if the group’s individuals had [worked] alone.

It is hard to believe (as with any categorical statement) that this is a final verdict. Some bloggers correctly point that done right, brainstorming is a powerful tool (here and here). Mindtools.com and Wikipedia describe effective brainstorming process.

However, group brainstorming indeed has potential risks for generating less and lower quality ideas:

1. Ideas that challenge conventional wisdom are suppressed or considered mistaken
2. People can change or even disregard own ideas while listening to ideas from others, especially authoritative sources
3. Sticking with the crowd causes less willingness for risky and innovative ideas.

Effective brainstorm process avoids these problems when:

1. People investigated the topic before the meeting and come prepared with own ideas and considerations.
2. People are diverse, have different perspective and background. They bring specialized and tacit knowledge to the table.
3. People are encouraged to think independently, creatively and don’t criticize others. Everybody contributes as equal; ideas could be even presented anonymously.

I experienced myself both successful and unproductive brainstorming sessions. I found that good brainstorming session produces ideas, which are better and more than just selection from individual ideas. It is real enjoyment to see when people create ideas, enrich each other thinking and generate new wonderful and unseen ideas. A side effect is important: people consider themselves as a part of the process, feeling ownership and control. They accept final ideas readily, better understand these ideas and are committed to implement them. This is a way to become self-organizing team.

4 Effective Leadership Styles in Software Development

What should be the qualities of the best leader for the software team: strong decisive knowledgeable or quiet supportive cooperative? Best leaders have two main concerns – people and getting things done (Blake and Mouton’s Managerial Grid). Both concerns have cumulative effect – high concern for people makes them motivated and therefore more productive, high concern for production creates sense of achievement and makes people more satisfied at work.

However, there is no one leadership style that suits all situations, projects and individuals. There are 4 main leadership styles (based on The Hersey and Blanchard model and Vroom and Yetton’s Normative Model), which could be effectively used in software development: The Commander, The Coach, The Supporter and Self-Organization.

1. The Commander

In tough times people need a strong leader – the person who can take responsibility and make decisions, the person who could motivate and inspire, the person who could move against winds and resistance.
Commanders are needed

• in crisis, when bold, quick and direct decisions should be made. The project is doomed, customers are leaving or there is fire in the building.
• when changes are necessary and there is strong resistance and barriers to make them. The leader creates a vision, finds a way to accomplish it and leads a change. The company moves to the new market or adopts new approaches as Agile.
• when people lack commitment, faith in company and willingness to work, because of previous failures, degrading environment or unpopular management. The leader can wake them up, inspire and move forward.

Commanders make and influence most decisions. The downside of this leadership style is that leader can slide to autocratic methods, demotivate and annoy people. Often decisions are not optimal – they don’t take in consideration all available information, options and perspectives. This style is effective in short-term, in long-term it could be dangerous for people and projects.

Skills and traits of Commanders are very important as they are the main factor of success. The Commander should be energetic, confident, decisive. Also as Commanders make most decisions they should be knowledgeable, forward-thinking and broad-minded.

2. The Coach

The Coach is experienced, knowledgeable and supportive person, who is needed when team lacks focus, expertise and understanding what should be done and how.
Coaches are concerned about growing people, creating enabling and trusting environment. This leader makes decisions collectively with a team. The Coach explains rationale behind decisions, listens and provides feedback. He encourages personal growth, builds long-term capabilities and prepares team and individuals for independent work.
This style has high relationship focus. The downside of this style could be micro-management and low concern for productivity.

Important qualities for Coaches: knowledgeable, empathetic and cooperative.

3. The Supporter

pignews

Supporters are needed to help teams, remove barriers and coordinate activities. The Supporter is an ego-less quiet leader and facilitator. Supporter makes joint decision with the team as equals, delegating majority of decisions to the team. In addition, this leader concerns about creation of harmony and balance between team members.
Important qualities for The Supporter: good communicator and facilitator, socially skilled and diplomatic

4. Self Organization

A cohesive, motivated and confident Team doesn’t need formal leaders. The team makes most decisions, while every member could step in and become leader in specific areas and situations. People are highly capable, committed and self-driven. The team should have diverse and independently thinking team members to prevent groupthink (see Wisdom of Crowds).

It is most preferable and most difficult to achieve style. Usually the team transcends through previous steps and becomes truly self-organized team after experiencing victories and failures, growing and gaining experience together. The Team have shared purpose, principles and values. Team members trust, support and respect each other. Almost everything is possible for such team.

Selecting Style

Teams and leaders use these 4 styles depending on a situation and specific people. The best leaders guide their team and people through these stages to the point when formal leaders are no longer needed – self-organizing team. It is possible that a self-organizing team will need a strong leader in tough time, a coach when moving to the new domain or a supporter to resolve external problems. However, the team, which experienced flow, independence and self-organization will strive to get back to this state again.

Important factors

1. Commitment - are people ready, motivated and willing to accomplish goals?
2. Competence and clarity of direction: do people know what to do and how to do it?
3. Cooperation and cohesiveness: can team effectively work together, make decisions and avoid conflicts? The larger group need more coordination.
4. Resources and support: does team have necessary tools, money, people, etc? Do they face serious barriers and require support?
5. External coordination: does team need to collaborate with other groups?
6. Leader - last, but not least: does leader have necessary authority, knowledge and experience?

All these styles could be effectively used in software development. But at the end, the most important factor of success is people. The style should match people commitment and capabilities. It is mistake to command skilled and confident team and it is mistake to give reins to unmotivated and inexperienced people. Leadership style is one of the secret ingredients of the Software Creation Mystery – it could bring great success or miserable failure on the same project with the same people.