Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Is it easy to build useful programs for humans? Failure rates and dissatisfaction with the software projects (more than 50% still fails or challenged) show that it is not quite easy task. Can AI help to build more successful projects, compete and eventually replace human programmers?

From ideas to specifications

Human programmers face objective challenges in building software systems, which AI will face in the future:

1. It is difficult to understand what people need.
2. Customer’s ideas are shifting, once they start using the real system and experience all the consequences and effects for interacting with it in the context of their problems.
3. Customer’s needs are changing constantly reflecting outside business and company trends, situation and problems.

Read full post >>

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said - between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization - galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties - the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations - optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions - the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation - a software project consumes energy to keep going, mostly in the form of money :)
• Instability - software development deals with constantly changing situations
• Multiple equilibria - there are many possible satisfying solutions
• Complexity - any non-trivial software project has complexity
• Hierarchies - there are many perspectives and levels - organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization - autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve - self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Read full post >>

I have just recently read an interesting post in Cedric blog:

“..software engineering is nowhere near as advanced as building engineering is. We are still working on these nuts and bolts, and whenever a new software project starts, we still can’t be sure it won’t collapse under its own weight after just a year. To make a parallel: imagine a world where every time a new construction takes place (say, a bridge), the future of that bridge depends on the team of engineers and workers you choose to build it…”

Bridges do collapse. Tragedy shows that even standard engineering solution which was following strict guidelines, regularly inspected and maintained could fail.

One bridge is a relatively simple construction. We can probably compare it to one week of programmer’s work considering complexity, number of elements and relations between parts. What could we compare in the engineering to the several month project with many developers and subsystems involved? It is comparable to the very complex construction like combination of hundreds interlinked bridges (don’t ask me to prove my ballpark estimates :)). It is comparable to the construction of Boeing. How much does research and development of Boeing 787 cost? $11 bln dollars.

Typical software projects have much lower cost, much less people and don’t have luxury of relying on stable laws of the physical world and established engineering principles. But programmers carry in their brains the same amount of information, many system relations and complexity as in advanced engineering projects . These are essential difficulties that software developers are facing. In addition they deal with the changeable world of people and business needs.

Major bridges collapse every 30 years and engineers learn from these mistakes and make better and more reliable designs. When we complain that Software Development doesn’t have the same sound practices as engineering, we shouldn’t accuse just immaturity of our profession. It is one of the most intellectually complex human activities dealing with very complex systems. I just hope we will experience less and less collapsing bridges both in physical and software worlds.

Software development is the translation of a user need or marketing goal into a software product. - Wikipedia

Why it is so difficult to understand the nature of the Software Development? Because it is ideal and metaphysical. Software Development doesn’t stand on objective physical world, but deals with fragile and subjective world of human ideas. That is why Software Development is much more than Software Engineering - it just cannot fit into Procrustean bed of well defined requirements, processes and laws (maybe except programming for science or physical devices).

Lifecycle of the most software projects:

1. People generate software ideas based on people needs. Humans have difficulty understanding what they need and even more difficulty to clearly explain it to others.
2. Software professionals try to understand these ideas and communicate to each other. Before coming to programmers these ideas are interpreted (more often misinterpreted) by bosses, marketers, project / product managers, architects, business analysts, etc.
3. Programmers are trying to translate these mutated ideas into the program code. Many of them are more interested in creating technical puzzles (or doomed to fight them) than in complete understanding of business ideas - often changing and sometimes illogical.
4. People use this software. Does it meet their needs? How happy do you think they are?

How to improve the flow of ideas?

Read full post >>

Eric Wise argues that software development is not software engineering. Reddit tries to compare it with science. Both discussions have many interesting and insightful comments, huge variety of opinions and show no signs of their reconciliation. I also have post related to understanding of the nature of software development as many other people in their blogs :).
Goethe said that between two opposite opinions you’ll find not the truth, but the problem. Can we ever define an unified theory of the nature of Software Development?

Software is everywhere: inside our computers, cars, phones and even toasters. Software tells these devices what to do.

Everybody can develop software. Hundreds of millions do. We use similar skills as in writing a cooking recipe or telling a friend how to find a shopping mall - we just need to come up with the set of instructions. Basic logic and knowledge of instruction language is enough. You don’t need to have a computer science degree or even finish courses to become a good programmer.

Does it sound simple? Creation of a program should be a routine job now as growing potatoes or building a bridge. And we have at least 2 good reasons to hope for this:

• We had more than 60 years to learn how to properly instruct computers what to do. The first computer ENIAC was built in 1946.
• Today 19 millions of professional programmers in the world and multi-billion industry are very serious about effectively building good software .

And still, software creation is unpredictable, unreliable and often fails.

Why? We should understand better what is software development.

Software development is the translation of a user need or marketing goal into a software product. - Wikipedia

We can add more to this definition.
Read full post >>