Software Creation Mystery -

What is Software Development?

Tower of Babel

Software is everywhere: inside our computers, cars, phones and even toasters. Software tells these devices what to do.

Everybody can develop software. Hundreds of millions do. We use similar skills as in writing a cooking recipe or telling a friend how to find a shopping mall – we just need to come up with the set of instructions. Basic logic and knowledge of instruction language is enough. You don’t need to have a computer science degree or even finish courses to become a good programmer.

Does it sound simple? Creation of a program should be a routine job now as growing potatoes or building a bridge. And we have at least 2 good reasons to hope for this:

And still, software creation is unpredictable, unreliable and often fails.

Why? We should understand better what is software development.

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

We can add more to this definition.

1) Is it engineering?

The application of scientific and mathematical principles to practical ends – The American Heritage Dictionary

Software development is close to engineering. As in engineering we have

  • Problem – user needs and requirements
  • Solution – software system that solves these needs
  • Principles – accepted rules for building solutions
  • Constraints – limitations that our solution should meet
  • Trade-offs – finding a compromise between different options
  • Methods – how do we practically translate user needs into solution applying principles, meeting constraints and making trade-offs

IEEE defines software engineering as application of a systematic, disciplined, quantifiable approach to development, operation, and maintenance of software; that is, the application of engineering to software.

Many prominent people in software development don’t agree that engineering is the only approach to create software.

Pete McBreen: “The biggest problem with software engineering is the assumption that a systematic, disciplined, and quantifiable approach is the only possible approach. …software development is not a defined process. At best, all we can hope to achieve is an empirical process. The reason is that the sources of all software requirements are people. There is no way to automate requirements elicitation; rather, people must talk to each other so that the software team can learn what the users really need the application to do. At the same time, the users are learning the technical constraints and costs and then use that knowledge to adjust the feature set that they ask for.”

Martin Fowler: “Engineering methodologies have been around for a long time. They’ve not been noticeable for being terribly successful. They are even less noted for being popular. The most frequent criticism of these methodologies is that they are bureaucratic. There’s so much stuff to do to follow the methodology that the whole pace of development slows down.”

Software development is similar to engineering as it deals with complex problems, constraints and trade-offs. The difference is that engineering relies on proven laws of physical world, well defined principles and processes, while software development relies on uncertain mental concepts, shaky people needs and empirical principles and processes (guided by experience). Directly applying engineering approach to software development could introduce more problems than solve.

2) Is it business?


The social science of managing people to organize and maintain collective productivity toward accomplishing particular creative and productive goals, usually to generate profit. – Wikipedia

Software development became essential part of most businesses for generating profit and reducing cost. The relation is often unhappy. Only 35% of software projects started in 2006 could be considered as successful and this is an enormous improvement with 1994 figures – 16%. And still, there is a long way ahead to make software development economically successful and effective.

Kent Beck: “We need to make our software development economically more valuable by spending money more slowly, earning revenue more quickly, and increasing the probable productive lifespan of our project. But most of all we need to increase the options for business decisions.”

Business is chaotic and depends on unpredictable factors as economy, market trends and customer behavior. Business drives software projects, destabilizes them with frequent changes of directions and still expects predictable results. Finally, most of the software projects doesn’t make sense without business and economic demands.

3) Is it art and craft?

Antonio Stradivari

a. A system of principles and methods employed in the performance of a set of activities
b. Skill that is attained by study, practice, or observation
c. High quality of conception or execution, as found in works of beauty; aesthetic value
The American Heritage Dictionary

Andrew Hunt and David Thomas [from Pragmatic Programmer]: “Programming is a craft. At its simplest, it comes down to getting a computer to do what you want it to do (or what your user wants it to do). As a programmer, you are part listener, part advisor, part interpreter, and part dictator. You try to capture elusive requirements and find a way of expressing them so that a mere machine can do them justice. You try to document your work so that others can understand it, and you try to engineer your work so that others can build on it. What’s more, you try to do all this against the relentless ticking of the project clock. You work small miracles every day.”

Pete McBreen [from Software Craftsmanship ]: “We must insist that developers really know their craft before we trust them to create systems for us or with us. Whether you are a user, manager, or developer, it doesn’t make sense to trust an inexperienced software developer. This is why we focus on software craftsmanship rather than on software engineering. The best processes in the world will not save a project from failure if the people involved do not have the necessary skills to execute the process; conversely, really good developers can make any process work.”

Donal Knuth: “We have seen that computer programming is an art, because it applies accumulated knowledge to the world, because it requires skill and ingenuity, and especially because it produces objects of beauty… A scientific approach is generally characterized by the words logical, systematic, impersonal, calm, rational, while an artistic approach is characterized by the words aesthetic, creative, humanitarian, anxious, irrational. It seems to me that both of these apparently contradictory approaches have great value with respect to computer programming.”

Successful software solutions for complex problems are difficult to achieve by standard scientific and engineering approaches alone. Complex problems require high level of mastery, creativity and intuition. And these traits are hard to learn, acquire and apply consistently.

4) Is it psychology?


The study of mental processes such as perception, cognition, emotion, personality, as well as environmental influences, such as social and cultural influences, and interpersonal relationships, in order to devise theories of human behavior – Wikipedia

Alistair Cockburn [from Characterizing people as non-linear, first-order components in software development ]: “In the title, I refer to people as “components”. That is how people are treated in the process / methodology design literature. The mistake in this approach is that “people” are highly variable and non-linear, with unique success and failure modes. Those factors are first-order, not negligible factors. Failure of process and methodology designers to account for them contributes to the sorts of unplanned project trajectories we so often see.”

Tom DeMarco and Timothy Lister [from Peopleware]: “The major problems of our work are not so much technological as sociological in nature.”

Gerald M. Weinberg [from Psychology of Computer Programming]: “Because of the complex nature of the programming task, the programmer’s personality – his individuality and identity – are far more important factors in his success than is usually recognized.”

Software is built by humans and in most cases used by humans. The way how individuals and teams of software professionals think, feel and interact has significant affect on creation of the software. And humans are difficult to predict, rationalize and quantify.

Software Creation Forces.

Now we can better understand what is software development, why it has problems and how it is different from established practices of growing potatoes and building bridges. Software teams are engaged in following activities to build software.

  1. Trying to solve complex problems that are driven by unpredictable businesses and even more unpredictable users.
  2. Translating these needs into software based on uncertain world of mental concepts and empirical principles. We don’t have luxury of firm physical laws, sound scientific principles and established engineering methods.
  3. Relying on humans to do this translation. Humans have varying psychology, talents and creativity; they expose complex social behavior. These factors are playing important role for individual and team productivity and quality of work.

As a result, many strong and contrary forces affect software development. However, we can identify 3 main group of forces:

1. Economic forces – profit, efficiency and customer needs.
2. Human forces – psychology, productivity, motivation; talent, creativity, learning; group behavior, interactions and social complexity.
3. System forces – organization and evolution of the complex systems.

The goal of this blog is to understand and harness these forces to improve software creation – one of the most useful, intellectual and creative human activities.

Question for readers
: Is software development more engineering, business, art, psychology or something else?

AddThis Social Bookmark Button AddThis Feed Button


My personal answerer to your question:
Software development is mix of engineering, business, art and psychology. The problem is in the fact that these disciplines do not share many common points of view. This is reflected in the way a software artifact is developed and evaluated. Moreover in the past years the software engineering research community has focused is attention in developing good algorithms and good tools but they left out most of the social aspect of the development.

Comment by Francesco | January 16, 2009 5:34 am

This blog have little value without you and your comments, thoughts and discussions. Please, leave your comments. You are welcome to debate and criticize any idea, but, please, don't attack other people. Thanks for your contribution!

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Subscribe without commenting

Software Creation Mystery -
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 License .