Evolutionary Software Architecture or Why Developers Are Not Janitors
Sep 3rd, 2007 | Architecture, Concepts, Design, Process, System
Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.
As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.
Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.
Dynamic of the Complex Systems
Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.
Can we say that the development of the software system has features of self-organization? Mostly yes, as we have
- Fluctuations – optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
- Local interactions – the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
- Dissipation – a software project consumes energy to keep going, mostly in the form of money 🙂
- Instability – software development deals with constantly changing situations
- Multiple equilibria – there are many possible satisfying solutions
- Complexity – any non-trivial software project has complexity
- Hierarchies – there are many perspectives and levels – organizational, technological and solution domain
The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?
Why self-organized systems are better?
- Distributed control– “control” of the system organization is typically distributed over the whole of the system. All parts contribute evenly to the resulting arrangement. The free market is a good example. There are many independent market agents – companies and individuals. Problems with individual company or even market segment doesn’t effect economy much as there are always players who will use this opportunity and take over the vacant niche. The brain is another example. Its organization is distributed over a network of interacting neurons. Although different brain regions are specialized for different tasks, no neuron or group of neurons has overall control. This is shown by the fact that minor brain lesions because of accidents, surgery or tumors normally do not disturb overall functioning, whatever the region that is destroyed.
Therefore, in effective software projects any skilled and experienced developer with tacit and specialized knowledge should have control over architecture decisions in his area as oppose to giving this control to few architects (or what even worse managers). Global order should emerge from these local development decisions.
- Robustness – self-organized systems are relatively insensitive to perturbations or errors, and can restore themselves, unlike most human designed systems. One reason for this fault-tolerance is the redundant, distributed organization. Another reason for this intrinsic robustness is that self-organization thrives on randomness, fluctuations or “noise”, which could bring unexpected and better solutions. And third reason is stabilizing effect of feedback loops.
- Feedback – the dynamics of a self-organizing system is typically non-linear, because of circular or feedback relations between the components. Positive feedback leads to an explosive growth, which ends when all components have been absorbed into the new configuration, leaving the system in a stable, negative feedbackstate.
Effective feedback loops could be established only with open, adaptive and distributed approach contrary to closed, rigid and centralized traditional approach to software development.
- Emergence – property or feature not previously observed as a functional characteristic of the system. Self organization turns a collection of interacting elements into an individual, coherent whole. Elements produce their own emergent properties and form the next level of structure in the system, which in turn form the building blocks for the next higher level of organization, with different emergent properties, and so on.
Therefore, it is almost impossible to come up with these properties and decisions on the preliminary software design stage. The key for building great software systems is to learn how to tame and use emergence effectively instead of fighting against it.
- Bifurcations – which of the possible configurations the system will settle in depends on a chance. A system may settle in a range of stable configurations. Since small fluctuations are amplified by positive feedback, this means that even small and impossible to observe fluctuation can lead to the to completely different project outcome.
Therefore, even if we know business needs, environment and system specifications at the beginning of the process, the outcome is still unpredictable. It is waste of effort trying to predict and plan it – it is better to prepare for changes and adaptation.
- Adaptation – a configuration of a system may be called “fit” if it is able to maintain or grow given the specific configuration of its environment. An “unfit” configuration is one that will disintegrate the system under the given boundary conditions. Different configurations can be compared as to their degree of fitness, or probability to survive under the given conditions imposed by the environment. Thus, adaptation can be defined as achieving a fit between the system and environment. Systems may be called adaptive if they can adjust to such changes while keeping their organization as much as possible intact.
Considering above points, adaptation is very important for survival and success of the software project.
- Self Regulation – adaptation can be considered as a problem of regulation or control: minimizing deviations from a goal configuration by counteracting perturbations before they become large enough to endanger the system.
Therefore, the software development process should be able to produce variety of actions to cope with each of the possible perturbations and select the best action for the given problem. How it is possible without developers, creators of the software system? They should be empowered, trained and encouraged to do this.
How to control a self-organized system?
- Establish attractors (state of equilibrium, a preferred and stable position for the system). It is better to establish vision, goals for expected system, good motivation and compensation for the team instead of telling what to do to and how to achieve this state.
- Establish principles(or fitness criteria) for choosing the best action for the given circumstances.
- The most straightforward method is to let the environment itself determine what is fit by trial and errors. This can be dangerous for the company since it may lead to the failure of the software project.
- Another approach is to create internal models of the environment. This allows to try out a potential action “virtually”, in the model, and use the model to decide on its fitness. Theoretically, this is the most effective approach and many architect are trying to follow it, but it is very difficult in reality as this approach should take in consideration huge number of variables and create credible simulation.
- Use simple rules, empirical patterns and best practices – considering the current state of the software development it is the best approach.
- Create effective self-organized teams.
Developers are not janitors and prima donnas, because they are the most important people to maintain healthy self organization in the development of the software system, avoid system degradation and lead to the project success. Architects should help developers to establish better attractors, fitness criteria and internal models (or patterns) for selection of the optimal solutions, but they are not the best people to make decisions about these solutions. Certainly it is valid statement, if we agree that evolution, adaptivity and self-organization are important (bottom up approach). Please notice, in this post I even didn’t touch people, idea flow and psychology factors – I considered software development only from perspective of the theory of complex systems and self-organization.
Useful Resources:
Self-Organizing Systems FAQ
The Science of Self-Organization and Adaptivity, Francis Heylighen
Order Out of Chaos, Ilya Prigogine
I find this article extremely interesting, and I have some questions and comments. First, my background in this area is previous study and development of complex systems, primarily in the area of agent based systems, with a little exposure to genetic algorithms. I quickly read the Self-Organizing Systems FAQ you linked, but I found some discrepancies with my previous understanding of complex systems, particularly regarding emergence. I took a look at Wikipedia, and found it to be more in line with my understanding of these concepts.
Self Organizing Systems
Emergence
Perhaps you can point out what is fundamentally different between these definitions? I am particularly uncomfortable with the FAQ’s claim that an engine is an emergent system. An engine is not a collection of self controlling components, and it is not self organizing. It is a designed system of components specifically configured for one purpose.
I was similarly confused by what you were proposing. I am unclear whether you were saying the software, the development process or the developers themselves were the components of a complex system? The software itself seemed the least likely, as the engine analogy holds true here as well. While each unit of code can potentially be used in multiple places, the overall structure of the software is designed. It is meant to have one set of functionality, and if through emergence it displays other behavior that should be considered a bug.
If you are referring to the latter options then I would say that, as my understanding goes, self organizing systems fitness is based on iterative attempts. In agent based software this is simple rules for agents and the results when introduced to an environment in large numbers. In genetic algorithms this requires many ‘generations’ of attempts to solve a problem. If these concepts are carried over to software development then perhaps I would agree that a nearly unlimited arrangement of software development teams over a period of time would produce many versions of an application, some of which would be better than an architected counterpart. However, while this model may exist to some degree in systems such as Linux features, it is not in practice or even arguably cost efficient within a single organization.
You seem to make the point that each developer should decide how to implement their own subsystem. I do not subscribe to the theory that architecture removes decision making from the developer’s duties, but to the one that an architecture is an overarching set of constraints that directs the development of the software. An architect would for example; choose an OS, programming language, hardware, database and perhaps software framework or design patterns. If each developer on the team was not so constrained, it would be extremely difficult to produce a cohesive application. In many long standing teams such things are foregone conclusions, but that does not mean the decision was not made.