From the first sight biological evolution looks too unpredictable to have any value for the constrained software development. We just don’t have time, money and resources for these wild experiment, unlimited trials and errors. It really seems that Nature could learn from us how to make things fast and effectively.

However, there are some principles that helped evolution to come up with amazing and efficient designs that made life flourishing on the Earth. This post will explore what software development could learn from biological evolution. See my previous post for the review of evolution concepts and mechanisms and how they could be applied to software development.

Read full post >>

I can’t wait to share this simple secret with you right now.

The Secret: Effective Software Systems are the systems that easy to understand and operate with human brains.

Programmers are more productive with effective software systems. Programmers can better learn and grow these system. Programmers have less problems, work faster and make better decision with them.

Now, you can avoid spending time reading this post if you already know this secret and you know how to avoid building the software system that:

• almost impossible to understand in reasonable time
• has confusing and convoluted swamp of logic and structure
• scary to change as nobody has any clue what will be broken, but sure that it will be broken

If you are still interested, lets find out what makes software systems effective.

Software Development is a pure mental endeavor (except typing on keyboard) that includes 3 main activities:

• Understand – learn and know system concepts and implementation
• Evolve – build, modify and support growth of the system ideas in the code
• Share – communicate and exchange ideas about the system

Programmers should care about 7 areas to make the system better suited for our brains:

1. Knowledge Creation and Retention – parsing, memorization and comprehension of the system ideas
2. System Organization – elements, relations and structure in the system
3. Sustaining Emerging Order – support evolution of the system and gain control over chaos
4. Minimize Noise and Purify – avoid adding unnecessary stuff to the system
5. System Discovery and Learning – making sense of the system
6. Mental Models – our internal explanations for how things are working in the real system
7. Shared Knowledge – ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge.

Read full post >>

“Ruin in community is not caused by witches with broomstick but it’s something that starts in people’s heads.” – Prof. Preobrazhensky (Michail Bulgakov’s “The heart of a dog”)

The newborn software system looks beautiful inside after the few weeks of development. Several talented programmers put their souls together into this amazing software idea. But week after week the system become older and uglier. It could still look exciting for users, but the system rots inside. It is much more difficult to modify, it breaks often, development time and cost soars. Programmers become scared to work with it. The system becomes Frankenstein. Why could it happen?

Three reasons

There are hundred objective reasons why it could happen – complex technology, changing customer needs, management pressure, time to market and many more. But some teams still deliver a good system under these conditions. And some teams cannot deliver even in the most favorable conditions. Why?

Degradation of the system starts in the people heads and there are three main reasons:

1. Incomprehension – developers don’t understand the purpose, ideas, design or technology behind the system
2. Inarticulateness – developers unable to express ideas through clear and effective architecture, design and code
3. Inconsistency – developers cannot act consistently or don’t care about the system

Read full post >>

Agile approaches insist that customer cannot come up with the reliable requirements at the beginning of the project. Martin Fowler provides four reasons:

1. Software development is a design activity, and thus hard to plan and cost.
2. The basic materials (technology and tools) keep changing rapidly
3. So much depends on which individual people are involved, and individuals are hard to predict and quantify.
4. In today’s economy the fundamental business forces are changing the value of software features too rapidly.

However, the most interesting reason is inside customers minds – they cannot reliably predict what they will need from software in the future.

Lets face inconvenient reality – most of the software ideas, decisions and requirements are not the result of scrupulous research, well prepared theories and mathematical models. If it were so, software development would take so much time, money and energy that it will almost eliminate any business benefits. Also it is very difficult to formalize anything related to irrational human beings (who accidentally are the main users of the software). In the most cases, software ideas are born from intuition, insights, copying other ideas and own experience mixed with some vague expectations. This empirical mess becomes source of the vision, requirements and future problems.

Customer dissatisfaction with software is the problem number one in software development. And the root cause is that many of customer’s expectations and predictions are in vain – mostly because future is fundamentally different than it appears in their thoughts. Why? There are six challenges that customers are facing to predict the future and get their ideas about the software right.

1. Subjectivity and our individual experience

alikaragoz
Software ideas are mainly based on our individual experiences, thoughts and understanding. Our individual experience is unobservable to everyone except ourselves. It is very difficult to communicate it right and completely understand other people. Translations and interpretations of software needs are unreliable – too much depends on what experiences we had, our memory and level of knowledge.
We see problems and possible solutions through lenses of our individual experience and often they shape and distort what we see. In addition, even if software customers can get objective view on their problems, they would see it from completely different perspective than domain experts, developers and end-users – people who shape ideas, build and use software. Subjectivity makes expression and understanding of software ideas very difficult.

Read full post >>

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations – optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions – the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation – a software project consumes energy to keep going, mostly in the form of money 🙂
• Instability – software development deals with constantly changing situations
• Multiple equilibria – there are many possible satisfying solutions
• Complexity – any non-trivial software project has complexity
• Hierarchies – there are many perspectives and levels – organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Read full post >>

Recently I was reviewing requirements for Microsoft Architect Certification and had found description for architect roles. After reading I had a question: do we need this kind of architects in a software team? Can they really help with building successful software?

The summary of architect responsibilities (as per Microsoft):

• Enterprise architects: set the overall vision and framework for the IT environment from a strategic business perspective.
• Solutions architects: design the solution to take advantage of the existing assets, integrate them into the existing environment, follow the enterprise architecture, and solve the business problems of the business owner or unit.
• Infrastructure architects: responsible for creating an architecture that meets the business and service level agreement requirements of the business owners and supports the applications and solutions that are required to operate their day-to-day businesses.

The bottom line is that company hires expensive and canny software architects before start of the project. They talk with the business, come up with solutions and make bright key decisions about architecture. But they don’t program. During the course of the project they ensure that developers don’t spoil this great architecture.

Now business owners can hire not so bright programmers, who should just follow directives and implement this great architecture. There is even correspondence with theories of some management consultants that programmers are just house painters.

Completely irrelevant questions: Can you trust advise of your friend about Paris, if he saw the city only on TV? Can you trust battle plans if battle didn’t begin and your commander knows little about enemy? How many chess moves ahead can you predict with high certainty before start of the game?

I don’t reject a need in architecture, but I don’t agree who, when and how should make architecture decisions in a software project.

Read full post >>

Marc Andressen quotes Frans Johansson’s book “The Medici Effect”:

In a [1987 study, researchers] concluded that brainstorming groups have never outperformed virtual groups. Of the 25 reported experiments by psychologists all over the world, real groups have never once been shown to be more productive than virtual groups. In fact, real groups that engage in brainstorming consistently generate about half the number of ideas they would have produced if the group’s individuals had [worked] alone.

It is hard to believe (as with any categorical statement) that this is a final verdict. Some bloggers correctly point that done right, brainstorming is a powerful tool (here and here). Mindtools.com and Wikipedia describe effective brainstorming process.

However, group brainstorming indeed has potential risks for generating less and lower quality ideas:

1. Ideas that challenge conventional wisdom are suppressed or considered mistaken
2. People can change or even disregard own ideas while listening to ideas from others, especially authoritative sources
3. Sticking with the crowd causes less willingness for risky and innovative ideas.

Effective brainstorm process avoids these problems when:

1. People investigated the topic before the meeting and come prepared with own ideas and considerations.
2. People are diverse, have different perspective and background. They bring specialized and tacit knowledge to the table.
3. People are encouraged to think independently, creatively and don’t criticize others. Everybody contributes as equal; ideas could be even presented anonymously.

I experienced myself both successful and unproductive brainstorming sessions. I found that good brainstorming session produces ideas, which are better and more than just selection from individual ideas. It is real enjoyment to see when people create ideas, enrich each other thinking and generate new wonderful and unseen ideas. A side effect is important: people consider themselves as a part of the process, feeling ownership and control. They accept final ideas readily, better understand these ideas and are committed to implement them. This is a way to become self-organizing team.

“I will work harder” – the horse Boxer (from George Orwell’s Animal Farm)

bichuas

The System Thinking Laws from Peter Senge’s book “The Fifth Discipline” applied to Software Development.

1. Today’s problems come from yesterday’s solutions.

We, humans, are happy when we solve problems. We often don’t think much about consequences. Surprisingly, our solutions could strike back and create new problems.

• A company decides to reward few key members of the very successful team with bonuses and promotions. The rest of the team feel unfairness and loss of motivation. Eventually tension between members is increased. The following projects are no longer successful.
• A project manager frequently asks developers to fix a new bug or work on urgent requests from customers. Developers do their best to fulfil these requests. Frequent distractions prevent them from finishing their main tasks for the iterations. Project shows only little progress.

Read full post >>

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

Why it is so difficult to understand the nature of the Software Development? Because it is ideal and metaphysical. Software Development doesn’t stand on objective physical world, but deals with fragile and subjective world of human ideas. That is why Software Development is much more than Software Engineering – it just cannot fit into Procrustean bed of well defined requirements, processes and laws (maybe except programming for science or physical devices).

Lifecycle of the most software projects:

1. People generate software ideas based on people needs. Humans have difficulty understanding what they need and even more difficulty to clearly explain it to others.
2. Software professionals try to understand these ideas and communicate to each other. Before coming to programmers these ideas are interpreted (more often misinterpreted) by bosses, marketers, project / product managers, architects, business analysts, etc.
3. Programmers are trying to translate these mutated ideas into the program code. Many of them are more interested in creating technical puzzles (or doomed to fight them) than in complete understanding of business ideas – often changing and sometimes illogical.
4. People use this software. Does it meet their needs? How happy do you think they are?

How to improve the flow of ideas?

Read full post >>

Who is smarter – a diverse group or the best expert? Can a boss, a project manager or the most experienced specialist make better decisions than a whole team? What are the problems with decision making in traditional corporations?

James Surowiecki in his book The Wisdom of Crowds argues that given the right conditions a group consistently outperforms best experts in decision making. Few people can do better than the group, but each time they will be different people. A group answer will often be at least as good as an answer of the smartest member.

The author provides interesting examples:

1. 800 different people (mostly non-experts) tried to guess weight of an ox at the International Exhibition of 1884 in London. The average guess of the crowd was only one pound off (ox weighed 1,198 pounds) and was better than any estimate of the cattle experts.
2. Stock market knew what company was responsible for the Challenger disaster within a half hour of the shuttle blowing up. Mostly uninformed investors dumped shares of Thiokol almost simultaneously.
3. Team of the men with wide range of knowledge (e.g. mathematician, submarine specialists, salvage men) correctly estimated location of disappeared U.S. submarine Scorpion. The submarine was found 220 yards from collectively estimated spot (original search area was 20 miles circle) and it was better than any individual guess.

Crowd is wise when these criteria are present:

1. Diversity of opinions (each person have some private information)
2. Independence (people’s opinions are not determined by the opinions of others)
3. Decentralization (people are able to specialize and draw on local knowledge)
4. Aggregation (some mechanism exists for turning private judgments into a collective decision)

Read full post >>