We can consider software development as the translation of our theories, concepts and ideas into the machine code. Therefore, how we gain, evolve and verify this knowledge is very important for building successful software. There are two main views how we gain the knowledge: Rationalism and Empiricism.

Rationalists appeal to reason as a source of knowledge or justification. They claim that our concepts and knowledge are gained a priori, independently of experience. Empiricists claim that experience is the ultimate source of all our concepts and knowledge. Rationalists come up with a theory and look for the evidence to support it. Empiricists assembles evidence, and builds on the evidence to form a theory.

The main question for the both camps: How do we know that our theory is true knowledge and not a guess?

Rationalism and empiricism have various success in different areas:

• Mathematics is triumph of rationalists – all the complex branches and theories are deducted logically from few intuitive basic premises.
• Natural sciences, which are dealing with dynamic complex systems combine rationalism and empiricism using scientific method: empirical observations and experiments with rational hypotheses and predictions. However, empiricism has the main emphasis – observational experiments are necessary to understand the physical universe.
• Social sciences, politics, economics show inability to come up with reliable rational theories. However, empiricism in these areas has own limitations as it is difficult to make comprehensive social experiments and deduct from them (and history) any true knowledge and theory about human groups and their societies. Human groups are complex, unpredictable and often irrational systems. They don’t behave consistently and don’t submit to permanent laws as physical world does.

What view does work better for software development?

How do these views show up in software development? Pure rationalists would prefer developing complete concepts and knowledge behind the software system using intuition, logic and creative insights before start of programming. Pure empiricists would immediately dive into experiments, programming and getting early practical results and build their knowledge based on this experience.

Read full post >>

Wired magazine published a very interesting article about change in US military philosophy during the Iraq war. US started with idea of Network-Centric Warfare:

The US military could use battlefield sensors to swiftly identify targets and bomb them. Tens of thousands of warfighters would act as a single, self-aware, coordinated organism. Better communications would let troops act swiftly and with accurate intelligence, skirting creaky hierarchies.

The Army committed more than $230 billion to a network-centric makeover, on top of the billions the military had already spent on surveillance, drone aircraft, spy satellites, and thousands of GPS transceivers.

Advanced technology helped to achieve quick victories – the opposing forces in Iraq (and Afghanistan) were broken in matter of days. But years of struggle to establish the new order came after these victories. Seizing territory and destroying enemy forces was not enough for the success. American military have learned from these failures and started to change approach recently. They realized importance of winning trust, minds and hearts of local people to win the war.

General David Petraeus, commander of Multi-National Force in Iraq, knows all about these mind games. He oversaw the writing of the new counterinsurgency manual. The book counsels officers to reinforce the local economy and politics and build knowledge of the native culture, “an operational code’ that is valid for an entire group of people.” And the manual blasts the old, network-centric American approach in Iraq. “If military forces remain in their compounds, they lose touch with the people, appear to be running scared, and cede the initiative to the insurgents,” it says.

Sometimes software development projects are closer to military operations than to the engineering process. The situation often changes, pressure is building up and quick decisions are required. The software system is the end result of these operations and developers are in the forefront. The software system campaign is not finished after the initial development is over: people start using the system, administrators support it and developers continue evolving it over time.

What skills can developers learn to contribute to the overall success of the long software campaign beside coming with technical solutions?

We often rely on technology, new tools and smart strategies to win our software battles. But at the end, people will decide the fate of the software. Should we teach programmers how to win minds of people who use their software in addition to mastering superiority in technology solutions?

Agile approaches insist that customer cannot come up with the reliable requirements at the beginning of the project. Martin Fowler provides four reasons:

1. Software development is a design activity, and thus hard to plan and cost.
2. The basic materials (technology and tools) keep changing rapidly
3. So much depends on which individual people are involved, and individuals are hard to predict and quantify.
4. In today’s economy the fundamental business forces are changing the value of software features too rapidly.

However, the most interesting reason is inside customers minds – they cannot reliably predict what they will need from software in the future.

Lets face inconvenient reality – most of the software ideas, decisions and requirements are not the result of scrupulous research, well prepared theories and mathematical models. If it were so, software development would take so much time, money and energy that it will almost eliminate any business benefits. Also it is very difficult to formalize anything related to irrational human beings (who accidentally are the main users of the software). In the most cases, software ideas are born from intuition, insights, copying other ideas and own experience mixed with some vague expectations. This empirical mess becomes source of the vision, requirements and future problems.

Customer dissatisfaction with software is the problem number one in software development. And the root cause is that many of customer’s expectations and predictions are in vain – mostly because future is fundamentally different than it appears in their thoughts. Why? There are six challenges that customers are facing to predict the future and get their ideas about the software right.

1. Subjectivity and our individual experience

alikaragoz
Software ideas are mainly based on our individual experiences, thoughts and understanding. Our individual experience is unobservable to everyone except ourselves. It is very difficult to communicate it right and completely understand other people. Translations and interpretations of software needs are unreliable – too much depends on what experiences we had, our memory and level of knowledge.
We see problems and possible solutions through lenses of our individual experience and often they shape and distort what we see. In addition, even if software customers can get objective view on their problems, they would see it from completely different perspective than domain experts, developers and end-users – people who shape ideas, build and use software. Subjectivity makes expression and understanding of software ideas very difficult.

Read full post >>

Surprisingly, Jim Highsmith, respected agile guru, argues against one of the core agile principles of self organization, associating it with anarchy. On the contrary, surprisingly again, the Army experts, Don Vandergriff and George Reed, embrace ideas of adaptability and decentralization in traditionally command-oriented military units.

Jim Highsmith says:

“I’ve been thinking recently that the term “self-organizing” has outlived its usefulness in the agile community and needs to be replaced. While self-organizing is a good term, it has, unfortunately, become confused with anarchy in the minds of many. Why has this occurred? Because there is a contingent within the agile community that is fundamentally anarchist at heart and it has latched onto the term self-organizing because it sounds better than anarchy.”

Army experts say:

“A culture of adaptability is one that accepts a lack of absolute control over events on and off the battlefield. Implementation requires revisiting mission orders or trust tactics. It necessitates raising the bar in the education, training and coaching of leaders and soldiers. It seems trite to suggest that an adaptive institution will reward those who, when the need arises, act without waiting for orders, but this also necessitates a climate that is supportive of those who act and fail to achieve stellar results. Instead of seeking perfection or optimum solutions, operators will find a solution that works locally and then exploit those results as a continual evolution facilitated by an organization adept at receiving and communicating such information.”

Read full post >>

Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Is it easy to build useful programs for humans? Failure rates and dissatisfaction with the software projects (more than 50% still fails or challenged) show that it is not quite easy task. Can AI help to build more successful projects, compete and eventually replace human programmers?

From ideas to specifications

Human programmers face objective challenges in building software systems, which AI will face in the future:

1. It is difficult to understand what people need.
2. Customer’s ideas are shifting, once they start using the real system and experience all the consequences and effects for interacting with it in the context of their problems.
3. Customer’s needs are changing constantly reflecting outside business and company trends, situation and problems.

Read full post >>

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations – optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions – the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation – a software project consumes energy to keep going, mostly in the form of money 🙂
• Instability – software development deals with constantly changing situations
• Multiple equilibria – there are many possible satisfying solutions
• Complexity – any non-trivial software project has complexity
• Hierarchies – there are many perspectives and levels – organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Read full post >>

I have just recently read an interesting post in Cedric blog:

“..software engineering is nowhere near as advanced as building engineering is. We are still working on these nuts and bolts, and whenever a new software project starts, we still can’t be sure it won’t collapse under its own weight after just a year. To make a parallel: imagine a world where every time a new construction takes place (say, a bridge), the future of that bridge depends on the team of engineers and workers you choose to build it…”

Bridges do collapse. Tragedy shows that even standard engineering solution which was following strict guidelines, regularly inspected and maintained could fail.

One bridge is a relatively simple construction. We can probably compare it to one week of programmer’s work considering complexity, number of elements and relations between parts. What could we compare in the engineering to the several month project with many developers and subsystems involved? It is comparable to the very complex construction like combination of hundreds interlinked bridges (don’t ask me to prove my ballpark estimates :)). It is comparable to the construction of Boeing. How much does research and development of Boeing 787 cost? $11 bln dollars.

Typical software projects have much lower cost, much less people and don’t have luxury of relying on stable laws of the physical world and established engineering principles. But programmers carry in their brains the same amount of information, many system relations and complexity as in advanced engineering projects . These are essential difficulties that software developers are facing. In addition they deal with the changeable world of people and business needs.

Major bridges collapse every 30 years and engineers learn from these mistakes and make better and more reliable designs. When we complain that Software Development doesn’t have the same sound practices as engineering, we shouldn’t accuse just immaturity of our profession. It is one of the most intellectually complex human activities dealing with very complex systems. I just hope we will experience less and less collapsing bridges both in physical and software worlds.

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

Why it is so difficult to understand the nature of the Software Development? Because it is ideal and metaphysical. Software Development doesn’t stand on objective physical world, but deals with fragile and subjective world of human ideas. That is why Software Development is much more than Software Engineering – it just cannot fit into Procrustean bed of well defined requirements, processes and laws (maybe except programming for science or physical devices).

Lifecycle of the most software projects:

1. People generate software ideas based on people needs. Humans have difficulty understanding what they need and even more difficulty to clearly explain it to others.
2. Software professionals try to understand these ideas and communicate to each other. Before coming to programmers these ideas are interpreted (more often misinterpreted) by bosses, marketers, project / product managers, architects, business analysts, etc.
3. Programmers are trying to translate these mutated ideas into the program code. Many of them are more interested in creating technical puzzles (or doomed to fight them) than in complete understanding of business ideas – often changing and sometimes illogical.
4. People use this software. Does it meet their needs? How happy do you think they are?

How to improve the flow of ideas?

Read full post >>

Eric Wise argues that software development is not software engineering. Reddit tries to compare it with science. Both discussions have many interesting and insightful comments, huge variety of opinions and show no signs of their reconciliation. I also have post related to understanding of the nature of software development as many other people in their blogs :).
Goethe said that between two opposite opinions you’ll find not the truth, but the problem. Can we ever define an unified theory of the nature of Software Development?

Software is everywhere: inside our computers, cars, phones and even toasters. Software tells these devices what to do.

Everybody can develop software. Hundreds of millions do. We use similar skills as in writing a cooking recipe or telling a friend how to find a shopping mall – we just need to come up with the set of instructions. Basic logic and knowledge of instruction language is enough. You don’t need to have a computer science degree or even finish courses to become a good programmer.

Does it sound simple? Creation of a program should be a routine job now as growing potatoes or building a bridge. And we have at least 2 good reasons to hope for this:

• We had more than 60 years to learn how to properly instruct computers what to do. The first computer ENIAC was built in 1946.
• Today 19 millions of professional programmers in the world and multi-billion industry are very serious about effectively building good software .

And still, software creation is unpredictable, unreliable and often fails.

Why? We should understand better what is software development.

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

We can add more to this definition.
Read full post >>