The main architecture goal is to strengthen and align system ideas, muscles and structures in areas most impacted by stress, changes and expansion.

In short, architecture covers every important aspect of the software system.

A good thoughtful software developer is torn apart by conflicting approaches for architecting complex software system:

1. Heavy upfront system design leads to rigid, costly and over-engineered solutions
2. Ignored business and technical perspectives cause business irrelevance or failure
3. Poor reliability, performance or flaws could kill promising beautiful software system

What is the best approach to build a sound and successful system?

Problems

Let’s consider several architecture challenges that a software team faces during system development.

1. Idealism (assumed known knowns) – mostly theoretical views, lack of practical experience, illusions. We often exaggerate our intellectual abilities to find right solution theoretically.
2. Open problems (known unknowns) – problems or needs without foreseeable solution. These evil inconvenient unknowns could seriously impact the course of the project
3. Uncertainty (unknown unknowns) – low predictability of the future in technology, business and Universe. Many good decisions could turn to be wrong after some time.
4. Over-complication – over-engineering for domain problems because of over-thinking, misunderstanding, lack of information or technical enthusiasm. Over-complication steals development time and increase cost and complexity.
5. Fragility – poor system quality (performance, reliability, uptime, security, etc.). Neglecting quality could cost reputation and harm system success.
6. Complexity – complex structures, unpredictable behavior and states that turn a system into the Ball of Mud.
7. Inconsistency – dissonance of the system approaches and parts; fragmented and incomplete knowledge in heads. Development teams produce incoherent system without coordination of ideas and implementation.
8. Inertia– difficulties to introduce disruptive changes (for business, users, operations). People and established processes resist changes and have inertia that is difficult to overcome.
9. Misalignment – with business goals, existing systems, infrastructure. Building a software system without considering business vision, existing IT landscape is the path to disaster.

Three Architecture Answers

There are 3 useful approaches to Software Architecture that address these challenges:

1. Strategic
2. Technical
3. Evolutionary

Read full post >>

The software system in the period of active growth is a really wild beast. Excited developers with creative minds and feature obsessed marketers consistently add the fuel to this fire of software creation.

Uncontrollable growth, race for features and engineering wonders sometimes give rise to a monster – a software system bloated with useless features, over-engineered internals and erratic behavior.

Read full post >>

From the first sight biological evolution looks too unpredictable to have any value for the constrained software development. We just don’t have time, money and resources for these wild experiment, unlimited trials and errors. It really seems that Nature could learn from us how to make things fast and effectively.

However, there are some principles that helped evolution to come up with amazing and efficient designs that made life flourishing on the Earth. This post will explore what software development could learn from biological evolution. See my previous post for the review of evolution concepts and mechanisms and how they could be applied to software development.

Read full post >>

Software Evolution is similar to a Bible Story: we, software creators, have a plan and goals and pursue them to create the perfect solution. We carefully select and add new features according to our grand design. We try to keep design clean and optimal. Things sometimes go wrong and the software system goes berserk, but we intentionally intervene and fix the problems to continue fruitful functioning of the system.

Biological Evolution is a different story: chaotic raise and fall of different species, without any plan and goals. New traits and variations appear all the time and many are useless or even harmful. Small modifications accumulate over time, cause significant changes and emergence of new species, better adapted for the evolving world. Survival of individual organisms and species is the matter of chance, pressure from environment and competition with other organisms.

Intentional growth of software versus chaotic evolution of organisms… Do they have anything in common?

Biological Evolution

In biology, evolution is the process of change in the inherited traits of a population of organisms from one generation to the next.

Species – a group of organisms that can reproduce with one another and produce fertile offspring.
Population – a localized group of individuals belonging to the same species.
Traits – particular characteristics of an organism.
Genes -portions of DNA molecule, which control inherited traits.
Genotype – complete set of genes within an organism’s genome.
Phenotype – the complete set of observable traits that make up the structure and behavior of an organism. These traits come from the interaction of its genotype with the environment.
Variations – the variation in phenotypes in a population reflects the variation in these organisms’ genotypes.

Most of the genome of a species is identical in all individuals of that species. However, even relatively small changes in genotype can lead to dramatic changes in phenotype: chimpanzees and humans differ in only about 5% of their genomes.

Read full post >>

I can’t wait to share this simple secret with you right now.

The Secret: Effective Software Systems are the systems that easy to understand and operate with human brains.

Programmers are more productive with effective software systems. Programmers can better learn and grow these system. Programmers have less problems, work faster and make better decision with them.

Now, you can avoid spending time reading this post if you already know this secret and you know how to avoid building the software system that:

• almost impossible to understand in reasonable time
• has confusing and convoluted swamp of logic and structure
• scary to change as nobody has any clue what will be broken, but sure that it will be broken

If you are still interested, lets find out what makes software systems effective.

Software Development is a pure mental endeavor (except typing on keyboard) that includes 3 main activities:

• Understand – learn and know system concepts and implementation
• Evolve – build, modify and support growth of the system ideas in the code
• Share – communicate and exchange ideas about the system

Programmers should care about 7 areas to make the system better suited for our brains:

1. Knowledge Creation and Retention – parsing, memorization and comprehension of the system ideas
2. System Organization – elements, relations and structure in the system
3. Sustaining Emerging Order – support evolution of the system and gain control over chaos
4. Minimize Noise and Purify – avoid adding unnecessary stuff to the system
5. System Discovery and Learning – making sense of the system
6. Mental Models – our internal explanations for how things are working in the real system
7. Shared Knowledge – ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge.

Read full post >>

“Ruin in community is not caused by witches with broomstick but it’s something that starts in people’s heads.” – Prof. Preobrazhensky (Michail Bulgakov’s “The heart of a dog”)

The newborn software system looks beautiful inside after the few weeks of development. Several talented programmers put their souls together into this amazing software idea. But week after week the system become older and uglier. It could still look exciting for users, but the system rots inside. It is much more difficult to modify, it breaks often, development time and cost soars. Programmers become scared to work with it. The system becomes Frankenstein. Why could it happen?

Three reasons

There are hundred objective reasons why it could happen – complex technology, changing customer needs, management pressure, time to market and many more. But some teams still deliver a good system under these conditions. And some teams cannot deliver even in the most favorable conditions. Why?

Degradation of the system starts in the people heads and there are three main reasons:

1. Incomprehension – developers don’t understand the purpose, ideas, design or technology behind the system
2. Inarticulateness – developers unable to express ideas through clear and effective architecture, design and code
3. Inconsistency – developers cannot act consistently or don’t care about the system

Read full post >>

Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Is it easy to build useful programs for humans? Failure rates and dissatisfaction with the software projects (more than 50% still fails or challenged) show that it is not quite easy task. Can AI help to build more successful projects, compete and eventually replace human programmers?

From ideas to specifications

Human programmers face objective challenges in building software systems, which AI will face in the future:

1. It is difficult to understand what people need.
2. Customer’s ideas are shifting, once they start using the real system and experience all the consequences and effects for interacting with it in the context of their problems.
3. Customer’s needs are changing constantly reflecting outside business and company trends, situation and problems.

Read full post >>

Intelligence is what you use when you don’t know what to do. – Jean Piaget

Part 1. Gaining processing power
Part 2. Becoming intelligent
Part 3. Interacting with humans
Part 4. Building useful programs
Part 5. Future of human programmers

Computers blindly follow our instructions. They are much faster than humans, but still computers are stupid things dependent on our algorithms and knowledge how to solve problems.
Even huge processing power is not enough to start programming. Non-trivial solutions require understanding of ideas, problem solving, learning from experience and much more – everything what we can define as intelligence. Can computers become smarter than human programmers?

What intelligence is required for building programs?

Read full post >>

Opinions about developers varied from janitors to “prima donnas” in comments to my previous post Do We Need Software Architects? 10 Reasons Why Not .
Beside discussion about the role of the software architects, the underlying philosophical problem is whether software development is primarily top down (centralized and planned) or bottom up (emergent and adaptive) process. If software development is top down, the architects are essential and crucial people on the project, who concentrate knowledge, establish technical leadership and guide development teams. If software development is bottom up, the developers become primary force for evolving the system, make key technical decisions and care about the architecture; architects (if they still needed) play coordination and mediation roles.

As Goethe said – between two opposite opinions you’ll find not the truth, but the problem. I incline to consider software development as a bottom up process, which occasionally (often in time of changing direction or crisis) needs centralized effort and top down approach.

Evolutionary, adaptive and emergent development of the software system leads to the most optimal solution. However, any software project usually has specific business goal, constraints and cannot evolve forever as natural systems do. Therefore, the main architecture concern is how to balance these two approaches.

Dynamic of the Complex Systems

Complex Adaptive System

Many complex systems show organization – galaxies, biological, market, society, etc. These systems can be explained and studied by referencing their parts, properties and laws (gravitational, supply / demand, etc.). Another approach is to look for the system as a whole, studying dynamic of the elements interaction and system properties – the science of self-organization.

Can we say that the development of the software system has features of self-organization? Mostly yes, as we have

• Fluctuations – optimal solution is not obvious and needs search; many useful vs. noise factors affect this search
• Local interactions – the software solution emerge from local interactions between involved people, business needs, technical platform and IT environment
• Dissipation – a software project consumes energy to keep going, mostly in the form of money 🙂
• Instability – software development deals with constantly changing situations
• Multiple equilibria – there are many possible satisfying solutions
• Complexity – any non-trivial software project has complexity
• Hierarchies – there are many perspectives and levels – organizational, technological and solution domain

The only fundamental property, which sometimes is missing for true self-organization – autonomy or absence of external control. Traditional corporation will always exercise some form of the control over software projects. And this is important to resolve – self-organized systems (evolving bottom up) achieve better solutions than rigid controlled systems. But why self-organized systems can produce better solutions? And what is the best way to control them?

Read full post >>

Recently I was reviewing requirements for Microsoft Architect Certification and had found description for architect roles. After reading I had a question: do we need this kind of architects in a software team? Can they really help with building successful software?

The summary of architect responsibilities (as per Microsoft):

• Enterprise architects: set the overall vision and framework for the IT environment from a strategic business perspective.
• Solutions architects: design the solution to take advantage of the existing assets, integrate them into the existing environment, follow the enterprise architecture, and solve the business problems of the business owner or unit.
• Infrastructure architects: responsible for creating an architecture that meets the business and service level agreement requirements of the business owners and supports the applications and solutions that are required to operate their day-to-day businesses.

The bottom line is that company hires expensive and canny software architects before start of the project. They talk with the business, come up with solutions and make bright key decisions about architecture. But they don’t program. During the course of the project they ensure that developers don’t spoil this great architecture.

Now business owners can hire not so bright programmers, who should just follow directives and implement this great architecture. There is even correspondence with theories of some management consultants that programmers are just house painters.

Completely irrelevant questions: Can you trust advise of your friend about Paris, if he saw the city only on TV? Can you trust battle plans if battle didn’t begin and your commander knows little about enemy? How many chess moves ahead can you predict with high certainty before start of the game?

I don’t reject a need in architecture, but I don’t agree who, when and how should make architecture decisions in a software project.

Read full post >>