Agile approaches insist that customer cannot come up with the reliable requirements at the beginning of the project. Martin Fowler provides four reasons:

1. Software development is a design activity, and thus hard to plan and cost.
2. The basic materials (technology and tools) keep changing rapidly
3. So much depends on which individual people are involved, and individuals are hard to predict and quantify.
4. In today’s economy the fundamental business forces are changing the value of software features too rapidly.

However, the most interesting reason is inside customers minds – they cannot reliably predict what they will need from software in the future.

Lets face inconvenient reality – most of the software ideas, decisions and requirements are not the result of scrupulous research, well prepared theories and mathematical models. If it were so, software development would take so much time, money and energy that it will almost eliminate any business benefits. Also it is very difficult to formalize anything related to irrational human beings (who accidentally are the main users of the software). In the most cases, software ideas are born from intuition, insights, copying other ideas and own experience mixed with some vague expectations. This empirical mess becomes source of the vision, requirements and future problems.

Customer dissatisfaction with software is the problem number one in software development. And the root cause is that many of customer’s expectations and predictions are in vain – mostly because future is fundamentally different than it appears in their thoughts. Why? There are six challenges that customers are facing to predict the future and get their ideas about the software right.

1. Subjectivity and our individual experience

alikaragoz
Software ideas are mainly based on our individual experiences, thoughts and understanding. Our individual experience is unobservable to everyone except ourselves. It is very difficult to communicate it right and completely understand other people. Translations and interpretations of software needs are unreliable – too much depends on what experiences we had, our memory and level of knowledge.
We see problems and possible solutions through lenses of our individual experience and often they shape and distort what we see. In addition, even if software customers can get objective view on their problems, they would see it from completely different perspective than domain experts, developers and end-users – people who shape ideas, build and use software. Subjectivity makes expression and understanding of software ideas very difficult.

Read full post >>

Surprisingly, Jim Highsmith, respected agile guru, argues against one of the core agile principles of self organization, associating it with anarchy. On the contrary, surprisingly again, the Army experts, Don Vandergriff and George Reed, embrace ideas of adaptability and decentralization in traditionally command-oriented military units.

Jim Highsmith says:

“I’ve been thinking recently that the term “self-organizing” has outlived its usefulness in the agile community and needs to be replaced. While self-organizing is a good term, it has, unfortunately, become confused with anarchy in the minds of many. Why has this occurred? Because there is a contingent within the agile community that is fundamentally anarchist at heart and it has latched onto the term self-organizing because it sounds better than anarchy.”

Army experts say:

“A culture of adaptability is one that accepts a lack of absolute control over events on and off the battlefield. Implementation requires revisiting mission orders or trust tactics. It necessitates raising the bar in the education, training and coaching of leaders and soldiers. It seems trite to suggest that an adaptive institution will reward those who, when the need arises, act without waiting for orders, but this also necessitates a climate that is supportive of those who act and fail to achieve stellar results. Instead of seeking perfection or optimum solutions, operators will find a solution that works locally and then exploit those results as a continual evolution facilitated by an organization adept at receiving and communicating such information.”

Read full post >>

Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Is it easy to build useful programs for humans? Failure rates and dissatisfaction with the software projects (more than 50% still fails or challenged) show that it is not quite easy task. Can AI help to build more successful projects, compete and eventually replace human programmers?

From ideas to specifications

Human programmers face objective challenges in building software systems, which AI will face in the future:

1. It is difficult to understand what people need.
2. Customer’s ideas are shifting, once they start using the real system and experience all the consequences and effects for interacting with it in the context of their problems.
3. Customer’s needs are changing constantly reflecting outside business and company trends, situation and problems.

Read full post >>

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations - optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions - the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation - a software project consumes energy to keep going, mostly in the form of money :)
• Instability - software development deals with constantly changing situations
• Multiple equilibria – there are many possible satisfying solutions
• Complexity - any non-trivial software project has complexity
• Hierarchies - there are many perspectives and levels – organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Read full post >>

I have just recently read an interesting post in Cedric blog:

“..software engineering is nowhere near as advanced as building engineering is. We are still working on these nuts and bolts, and whenever a new software project starts, we still can’t be sure it won’t collapse under its own weight after just a year. To make a parallel: imagine a world where every time a new construction takes place (say, a bridge), the future of that bridge depends on the team of engineers and workers you choose to build it…”

Bridges do collapse. Tragedy shows that even standard engineering solution which was following strict guidelines, regularly inspected and maintained could fail.

One bridge is a relatively simple construction. We can probably compare it to one week of programmer’s work considering complexity, number of elements and relations between parts. What could we compare in the engineering to the several month project with many developers and subsystems involved? It is comparable to the very complex construction like combination of hundreds interlinked bridges (don’t ask me to prove my ballpark estimates :)). It is comparable to the construction of Boeing. How much does research and development of Boeing 787 cost? $11 bln dollars.

Typical software projects have much lower cost, much less people and don’t have luxury of relying on stable laws of the physical world and established engineering principles. But programmers carry in their brains the same amount of information, many system relations and complexity as in advanced engineering projects . These are essential difficulties that software developers are facing. In addition they deal with the changeable world of people and business needs.

Major bridges collapse every 30 years and engineers learn from these mistakes and make better and more reliable designs. When we complain that Software Development doesn’t have the same sound practices as engineering, we shouldn’t accuse just immaturity of our profession. It is one of the most intellectually complex human activities dealing with very complex systems. I just hope we will experience less and less collapsing bridges both in physical and software worlds.

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

Why it is so difficult to understand the nature of the Software Development? Because it is ideal and metaphysical. Software Development doesn’t stand on objective physical world, but deals with fragile and subjective world of human ideas. That is why Software Development is much more than Software Engineering – it just cannot fit into Procrustean bed of well defined requirements, processes and laws (maybe except programming for science or physical devices).

Lifecycle of the most software projects:

1. People generate software ideas based on people needs. Humans have difficulty understanding what they need and even more difficulty to clearly explain it to others.
2. Software professionals try to understand these ideas and communicate to each other. Before coming to programmers these ideas are interpreted (more often misinterpreted) by bosses, marketers, project / product managers, architects, business analysts, etc.
3. Programmers are trying to translate these mutated ideas into the program code. Many of them are more interested in creating technical puzzles (or doomed to fight them) than in complete understanding of business ideas – often changing and sometimes illogical.
4. People use this software. Does it meet their needs? How happy do you think they are?

How to improve the flow of ideas?

Read full post >>

Eric Wise argues that software development is not software engineering. Reddit tries to compare it with science. Both discussions have many interesting and insightful comments, huge variety of opinions and show no signs of their reconciliation. I also have post related to understanding of the nature of software development as many other people in their blogs :).
Goethe said that between two opposite opinions you’ll find not the truth, but the problem. Can we ever define an unified theory of the nature of Software Development?

Software is everywhere: inside our computers, cars, phones and even toasters. Software tells these devices what to do.

Everybody can develop software. Hundreds of millions do. We use similar skills as in writing a cooking recipe or telling a friend how to find a shopping mall – we just need to come up with the set of instructions. Basic logic and knowledge of instruction language is enough. You don’t need to have a computer science degree or even finish courses to become a good programmer.

Does it sound simple? Creation of a program should be a routine job now as growing potatoes or building a bridge. And we have at least 2 good reasons to hope for this:

• We had more than 60 years to learn how to properly instruct computers what to do. The first computer ENIAC was built in 1946.
• Today 19 millions of professional programmers in the world and multi-billion industry are very serious about effectively building good software .

And still, software creation is unpredictable, unreliable and often fails.

Why? We should understand better what is software development.

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

We can add more to this definition.
Read full post >>