The manager slams a door and tells us that we are in a big trouble. Our old customers complain about many bugs and bad performance, new customers complain about delays and lack of dedication. And, top management considers our department financially unsustainable and wants to deeply cut expenses.
The manager tells that we are brilliant programmers, work very hard and create cool software solutions. But there is something wrong and we cannot work this way anymore.

Anxiety started to penetrate our souls. We know what is wrong: our team is short of people, we have too many commitments, our code is becoming a big mess, new technology and our new software version makes something bad with servers. A snowball of different problems makes us stressed, distracted and incapable of productive work.

What could our manager do next?

1. Distrust. Become a dictator, make own decisions including hiring external consultants to recommend what to do or even replace us. However,
   • we are good programmers and know our business well – the problem is not in lack of skill and knowledge
   • external people will take a lot of time to understand the system and they will have different motivation and won’t care about the long-term success
   • people will be demotivated and the manager cannot make effective decisions without active team involvement
2. Faith. Give to team the full power to fix a problems and make own decisions in hope that smart people, motivation and technical expertise will do magic. However,
   • fresh outlook and thinking out of box are hard when a team immersed for a long time into difficult situation
   • a team possibly doesn’t have understanding and control over external forces – management, customers, finances
   • changing of reality is tough (especially in people heads) and requires more than technical experience

There is a third way. Place improvement practices in the core of development process. Make self-improvement inevitable and required for any activity. Do it every day.

Toyota Way is the best example of large-scale reliable self-improvement process. It focuses on eliminating waste, solving problems at root cause and making right decisions. Toyota Way reduces problems, increases internal efficiency and makes a company successful. This is the best receipt for coming out of crisis.

Targets:

• Problems - emergencies, fires that require immediate fix: bugs, server crushes, deadline slips

• Waste - inefficient and non-value adding activities: waiting, misinformation, stress

• Challenges - adaptation to external forces (market, competitors, customers, society): new trends and technologies, changes in users expectations for user interface and functionality

Practices to see waste and stop to fix problems

1. Seeing waste

The team and managers should learn to see targets – real problems and waste. Otherwise improvements will be wild shots in the dark.

There are many targets in software development:

• stressed people – reduced energy, less productivity, more mistakes
• waiting – delays, tools / system problems and downtime, capacity bottlenecks (waiting for results of other people work)
• over-engineering – producing features and complicate design without real need
• unfinished work – functionality not used in a live system, probably still in design or under development or simply discarded (but not removed)
• defects – complete waste of time and money
• unused creativity – loosing time, ideas, skills, improvements and learning by not engaging or listening to your employees
• inadequate information – unclear, misleading or simply wrong information that causes useless activity and leads to rework at the end

• over-processing – taking unneeded steps to build software because of poor process and system design, overhead, bureaucracy, compliance, cumbersome tools
• motion – how much effort to get necessary information, access systems or use tools
• multi-tasking – losing time to switch between projects, tasks or different activities

2. Jidoka - stop to fix the problem to get quality right the first time

It is not enough to see your targets. The Team should carry commandment to shoot targets immediately. Otherwise the best intentions will be buried under growing avalanche of problems.

How to stop:

• quality for the customer drives all the processes – prevents temporary patches and bad for quality decisions
• low tolerance for quality problems and immediate detection are core work principle
• use Andon – a system to signal for help, notify about a problem and stop the process

• build in tools and process capabilities of detecting problems and stopping itself
• use all modern QA methods available
• managers encourage stops to fix problems and support implementation of counter measures

Making Decisions

“Make decisions slowly by consensus and thoroughly considering all options, implement rapidly.“ – Toyota Way

Even knowing problems and committed to solve them, the Team should make right decisions how to do it.

Make decisions slowly

1. Go and See (Genchi Genbutsu)
2. Understand underlying causes (Kaizen)
3. Broadly consider alternative solutions and develop a detailed rationale for preferred solution delaying certain decision as long as possible (Set-Based concurrent engineering)
4. Build consensus within a team and partners where group decision is preferred (while management can step in if consensus is not achieved)
5. Use very efficient communication devices – preferably one side of one sheet of paper

Implement rapidly

1. Put in place solutions and counter measures.
2. Evaluate the results
3. Standardize if solution is effective.

Practices to Eliminate Waste and Solve Problems

3. Genchi Genbutsu – go and see for yourself to thoroughly understand situation

How often do we jump to conclusion based on partial information, vague assumptions and what other people say? Information creates reality in your mind. This reality is a base for your decisions. So, you and the Team should get right information to make right decisions:

• observe situation with blank mind
• avoid assumptions and preconceptions
• use personally verified information

In short, base decisions on what really is going on

4. Kaizen (5 why’s) -continuous learning and improvement

“We view errors as opportunities for learning” – Toyota Way
The Team should find the root causes of the problems. Kaizen helps to find the root cause by repeatedly asking why the problem occurs.

Example of Kaizen
Problem: there are persistent javascript errors on a live site

1. Why? A developer didn’t build correct logic for complex web UI components interaction
2. Why? A developer built own solution without guidance and enough experience in this area
3. Why? A team expert didn’t tell about existing proven solutions, didn’t help and didn’t share knowledge
4. Why? The team is under stress, over-committed and don’t have time to communicate
5. Why? Managers accept too much work without consulting with development team
6. Why? you can continue…

Kaizen forces us to overcome desire to find a first convenient explanation and patch problems without fixing root causes. By ruthlessly applying this practice, we get deeper insight into reality and better learn our product, processes, people, environment and tools. Kaizen is a core practice to see waste, solve problems and improve process.

To avoid forgetting learning from Kaizen, it is important to standardize the improved process and make it a base for further improvements.

Practices to Support Flow

5. Standards – best you know today which is to be improved tomorrow

Standardized work is easier, cheaper and faster – stable repeatable methods can maintain predictability, high productivity and support quality.
Effective standards are not coming from theories, they come from

• best practices
• accumulated learnings and individual experience
• lessons from applying existing standards

The Team should try to use standards in many areas: project phases and activities; development practices; architecture and design approaches; code conventions; tools; programming techniques; libraries and third-party code; reuse of components and solutions; testing and so on.
Standardization in software development is a controversial topic – some theorists want to bring programming closer to standard-dominated engineering, practitioners are keen to reduce standardization to minimum promoting creativity and self-organization. In the rigid interpretation, standards are “must to follow” rules for any situation, in other interpretation standards are well defined steps and guidelines highly recommended for specific context. I support the latter definition. A productive team should have standards in place to focus on customer needs instead of fighting with the same puzzles and problems over and over again.

The system of standards shouldn’t be a heavy bureaucratic conduit, but a light and fluid book of knowledge. The book that contains most helpful and important rules and checklists. Standards will be effective if they are minimal, reviewed often (Kaizen) and followed by every team member.

6. Reliable thoroughly tested technology

The Team should be conservative with new technologies. Software development and IT thrive on change and innovation. However, Toyota Way suggests to be conservative in adapting technology and considers stability and reliability of operations as much more important goal than keeping on the cutting edge of technology.

Considerations for using technology

• primary goal is to improve flow and support people, process and values.
• process is driven by business, not technology concerns; software and tools do not eliminate themselves waste
• technology is visual and intuitive – people can use it correctly and effectively

• process manually before adapting technology to support the process – understand what problems it solves and how technology could help
• important: people / processes are easy to Kaizen, machines are difficult

Adopting new technology:

• new technology is unreliable and difficult to standardize, therefore it endangers flow
• proven process takes precedence over new and untested technology
• conduct actual tests before adapting new technology
• reject technology if it conflicts with culture or might disrupt stability, reliability and predictability

In the same time encourage people to consider new technologies while looking into new approaches. If technology improves process and flow – quickly implement after thorough testing.

7. Visual Controls

The Team should have clear status of information. Visual controls can convey complex information in fast and efficient for our brains way. We can use controls as a user story board; status of projects, servers or code build; burn down charts and others.
Simple visual indicators help people determine immediately whether they are deviating from the standards, provide quick gist of situation and direction for solving problems.

• use simple and most important indicators

• than provide clear picture for decisions and what to do next
• reduce reports to one screen / piece of paper even for the most important decisions

People, leaders and teams

8. People

People who build software are the people who should improve the process. They are directly involved and have first-hand experience of problems and waste.

Toyota Way expects that each team member is a problem solver and values job experience more than theoretical knowledge. The Team will beat any external consultants and find better way to work if people are open about problems and eager to find good solutions.

9. Leaders

The Team desperately needs strong leaders to build great products to overcome problems. Toyota grows leaders within who thoroughly understand the work, live the philosophy and must understand the daily work in great details

Chief Engineer is a key person in Toyota projects.:

• blessed by top management
• has control over project
• exceptional engineer
• critical link: between engineers and customer satisfaction
• coach for other engineers
• focus on concepts first, technicalities later

Chief Engineer concept is an excellent example for software technical leadership. A software team leader often lacks authority or makes too technical decisions without good understanding of customer needs.

10. Teams

The Team should be diverse and capable of solving wide range of problems. Toyota builds cross functional product teams, which

• use integrative decision making
• fast and accurate in implementation
• enhance process and flow by solving difficult technological problems

Software developers and their leaders are foundation of success in any project. Management, process and technology can only support them. And anyway, the process is as good as people follow it. Therefore, it is important to make software teams a key player in process improvements – they know problems, they understand work and they are capable to find good solutions.

Using Toyota Way

Can The Team revert a situation and win? Can it build the optimal process and expertise for fast development of high quality and low cost solutions?

This post shows the most effective option – build continuously improving process into the heart of development. The process that focuses on quality, eliminates waste and fixes problems at root cause. I believe this approach is a foundation of long term success. Your managers and company would love it!

A software project is a creative, unique and therefore unpredictable endeavor. We are not building the same thing over and over again, but solve new problems, address increasing demands and use perpetually changing technologies. Under these conditions, people – smart, creative and productive – are the most important factor of success . Software development process can only support and compliment these people, but it cannot guarantee success alone and make the factor of people negligible.

But, business wants predictable, reliable and successful results. I bet they don’t want to be at mercy how cards are shuffled in their talented development team. The answer is in establishing a process that increases chances of success and aligned with present nature of software development (unpredictable, empirical and heavily dependent on people).

The Toyota Way can be a great example that worth to learn. Toyota evolved from a small looming equipment shop to the largest car manufacturing company. The main foundation of successful growth is the system of few core principles that enables best quality, high productivity, lowest cost, shortest time and long-term success.

Agile and Lean Software Development adopted some practices from Toyota. Lean Software Development: An Agile Toolkit  (by Mary Poppendieck, Tom Poppendieck) is an excellent book that review Lean practices and principles in details. This post focuses more on philosophy and the system of practices as a whole that can turn software development team into a smart reliable machine.

1. Value

Why customer will pay money for your product? How does your company creates value?

All we are doing is looking at the time line from the moment the customer give us an order to the point when we collect the cash. And we are reducing that time line by removing the non-value-added wastes.

- Taiichi Ohno, founder of TPS

The only thing that adds value in software development is transformation of information and code into what customer wants. It is very important for software team to know how does it create value for customers. Or simply, decrypt the formula:
Customer (needs) -> Cash (delivered software)

Software Development Value Stream
Problems, needs and ideas are translated into units of development – functionality, user experience (interface, interactions, usability) and system qualities (performance, reliability, security, etc). A software development team transforms these pieces of requirements into a computer system using value-added and many not-so-value-added actions.

The ultimate goal of Toyota Way is to eliminate non-value-added activities while keeping the best quality of the product.

2. Flow

What is the process of creating value in your company?

Properties of the optimal flow for software development:

• cut back to zero the amount of time that any unit of work is sitting idle or waiting for somebody to work on it
• move design, code and information fast, link processes and people together that problems surface right away

One-piece flow
The most effective flow is one-piece flow – a customer need is immediately converted into a delivered software solution. In the same time this is the most demanding flow: a customer, designers, developers, testers and system administrators should be dedicated to the project, immediately available and work as one team together until the need is implemented in a live system.

One-piece flow has significant benefits (if it is implemented right):

• Quality - defects detected quickly; close proximity and connections ensure that necessary information is available immediately and important details are not missing; problems are solved quickly
• Flexibility - flexibility to respond and do what customer really wants in real time
• Productivity - the team engaged in high value added continuous work
• Morale - immediate results of work bring sense of accomplishment and job satisfaction
• Cost - lean production with minimal waste

However, in reality one-piece flow is difficult to achieve in many projects:

• a customer is not engaged all the time
• complex business domain requires analysis and preparation before development can start
• project activities involve people with different skills that are not available immediately
• deployment of mission-critical systems demands intensive testing and scheduling

• management, marketing, end-user expect specific time-lines, features and support

• and so on

Toyota uses two additional principles to overcome challenges of one-piece flow: pull systems and leveling out the workload.
However, Toyota Way says: “Flow where you can, pull where you must“. The big challenging goal is to turn eventually all the processes into one-piece flow.

Pull system (Kanban)
Kanban is a signaling system to trigger action. It adds to flow small buffers pulled by customer demand. Kanban allows optimal use of people and natural breaks in processes.

Agile iterations and Scrum sprints are close to Kanban ideas. A team pulls new user stories every time-boxed iteration from the backlog to design, implement, test and deploy within the same iteration.

However, it is not classical Kanban. An original idea would be to pull small batches of user stories for development after a team has finished a current batch (not at the end of fixed time iteration). In parallel, QA pulls implemented user stories for testing and so on. A customer can continue preparing next batches of work based on their priority of needs.

Main ideas

• provide dependent down stream teams with what they want, when they want and in amount they want
• just-in-time – started by high priority needs
• minimize work-in-progress and untested code
• frequently push what a customer can use right away
• be responsive to day-by-day shifts in demand instead of relaying on long-term schedules

Kanban is effective way to synchronize people and teams work without bringing them into continuous flow, where they should be always ready for the next unit of work.

Level Out the Workload (Heijunka)
The main idea of Heijunka is to level production by volume and activities mix.

• it is not built around actual flow of customer orders
• workload is leveled for the period considering previous history and scheduled projects

In software development workload could be leveled and scheduled when people could make reliable guess for what is expected. It is different from Kanban that works as just-in-time system adapting to the current workload.

Good candidates for Heijunka are

• standard procedures (automated testing, performance testing, deployment)
• recurring events (meetings, planning, demo, code reviews)

• scheduled tasks for people with limited availability (external consultants, customers, management)

• tasks with high confidence in estimation, usually done many times before (initial requirement exploration, design mockups, security audit, training new team members, reusing components and solutions)

Benefits:

• predictability and active planning

• balanced use of people, ability to schedule their involvement

• smoother demand on customers, shared resources and contractors

Toyota Way is focused on eliminating waste of non-value added work (Muda). Heijunka eliminates two additional types of waste:

• unevenness in project schedule (Mura)
• overburden of people and systems (Muri)

Building your own flow

These three approaches can help you to build most optimal flow. Consider cons and pros:

One-piece flow (push)
pros: minimal waste, fast implementation, low cost, quick problems resolution
cons: uneven workload, requires full availability and high level of engagement
when to use: for one collocated team work that is dedicated to one project

Kanban (pull)
pros: dynamic adaptation to load; synchronization of different teams and disconnected processes
cons: low predictability for completion; buffers hide waste
when to use: disconnected or involved in the multiple projects teams; activities that need people with limited availability

Heijunka (level)
pros: good predictability, ability to plan, balanced use of people: avoid overburden and uneven workload
cons: non-adaptive, inefficient for tasks with unpredictable effort (a lot in traditional software development)
when to use: activities that are standardized, highly predictable or have to be scheduled

You cannot build reliable and optimal flow from the scratch based on a theory only. The best process for your projects will emerge as result of evolution, problem solving and eliminating waste. This is a topic of the next post.

If some problem occurs in one-piece flow manufacturing then the whole production line stops. In this sense it is a very bad system of manufacturing. But when production stops everyone is forced to solve the problem immediately. So team members have to think, and through thinking team members grow and become better team members and people.

- Teruyuki Minoura

References:
The Toyota Way by Jeffrey Liker
Lean Software Development: An Agile Toolkit  by Mary Poppendieck, Tom Poppendieck

What you are as a person is far more important that what you are as a basketball player. – John Wooden

People are amazing, surprising and interesting. They change reality with power of thought and make things happen. What is most exciting – all people are completely different in their attitudes and behavior. But this comes with price – it is difficult to understand people and even more difficult to find the best way to deal with them.
Many people, who see programmers as extensions of their computer systems, will be surprised to discover that programmers are amazing individuals too. Programmers exhibit similar to other people behavior, they have different personalities and need individual approach.

I offer in this post a simple theory about Three Dimensions of a Software Programmer that could help to put relations with these individuals on some rational basis.

Axioms

There are two basic axioms in foundation of the theory

1. Constancy - some programmers consistently outperform others under same conditions.
2. Variability - performance of a programmer varies under different conditions.

Talents (constants)

Talents are recurring patterns of thoughts, feelings and behavior that can be productively applied. – First, Break All the Rules
Genes and upbringing form an adult human with specific talents and personality, strengths and weaknesses. These core individual characteristics of a programmer are difficult to change (except some limited success with consumption of large quantities of beer). Programmers are different – creative, thorough, funny, systematic, laid-back, curious, analytical, spontaneous and so on. Some people cannot be programmers at all.
Each unique programmer has a base performance level that is constant and almost impossible to change.

Three Dimensions (variables)

Now we can move to the part of equation that can be changed. Can we make programmers permanently performing above their base level? To answer this question lets review three variables of programmer’s performance.

X. Energy
measures amount of work and power of ideas that could change and improve reality.
Here I consider a specific kind of energy – creative positive energy that is directed to build better software and aligned with company and customers goals. Individual’s desire to act, create and achieve is the source of this energy. As we discussed above, part of this energy is based on character and predetermined. However, there is a big variable component of individual energy that is driven by motivation and environment.

Y. Discipline
measures ability to focus and follow necessary steps to achieve goals with good quality.
Accountability, self-organization and focus are core elements that keep individuals to perform on a high level. Personality matters here – some people will promise and forget immediately, give up experiencing first difficulties or become distracted by more interesting tasks. Some will fight until the end and go beyond limits to meet commitments, exceed expectations and bring excellent results. However, significant part of programmer’s discipline is variable and depends on context and expectations.

Z. Expertise
measures knowledge and experience in programming, technology and a customer domain.
Expertise includes know how to build software, make right decisions and implement good solutions with minimal troubles. There is no doubt that expertise is growing with time, but individual variations are big. Many people consider expertise as the only important characteristic of a programmer. I believe that it has much lower value without two other variables.

Programmer’s Archetypes

FfHII

How to Improve

A programmer can be 10 times more productive than other programmer and 10 times more productive than himself under different conditions. Therefore right people under right conditions could be 100 times more productive than others.

How can you influence programmer’s performance with these three dimensions? Hopefully, you found right people with right talents. It is a waste of time to change personality and push in direction where talent is absent. However, it is possible to amplify natural strengths and create conditions for super productivity.

Energy
goal: make people empowered, motivated, interested and satisfied; engage them to perform above their base level of energy

job design

• interesting work – brings inspiration and motivation
• control – over own work and decisions – pushes people forward
• self-organization – opportunities to find own ways that bring best results
• creative tension – goals on the edge of capabilities to focus energy and open new energy sources
• defined outcomes – clear direction that reduces anxiety and doubt and increases productivity

psychology

• flow – put mind in a state of energized focus, full involvement, and success
• positive experience – good emotions and healthy team relations; comfortable and trusted environment
• personal interests – decision making considers and takes care of individual interests
• right incentives – compensation, rewards, recognition are the great source of energy and motivation

environment

• space for mistakes – tolerance and open ways to fix mistakes make people think about moving forward instead of covering assess
• productive environment – where people think about work instead of nuisances
• fit – between personal and company goals, culture and views – amplify people desire to work
• reasonable pressure – normal workload, minimal stress, time slack to avoid energy drains

Discipline
goal: make people focused, responsible and aligned; make them performing in the right way

support system

• established process – align actions with goals and support their execution (Agile is the best way to go)
• transparency – reasons for decisions should be clear, problems should surface quickly
• pull system (kanban) – signaling system to trigger action based on actual needs
• control mechanisms, check and balances, audit, risk management (e.g. iterations)

zone of responsibility

• clear areas of responsibility – what is included and what is not
• balance of responsibility and authority – adequate authority to accomplish goals; restrictions to avoid damage
• rules of engagements – when, where, and how force (bold actions) shall be used
• match confidence with responsibilities – an individual is prepared and ready for the new responsibilities

standards

• safety regulations – mandatory rules for everyone
• professional standards – recommended parameters of work and products (security, performance, availability, etc.)
• best practices – spread approaches that work

Expertise
goal: make people knowledgeable, experienced and growing based on their talents

career

• right career path – built on persons strengths – enable growth in area of competence and natural strengths without switching into paths that don’t match talents (e.g. management). Avoid consequences of raising to the level of incompetence (Peter Principle).
• broadbanding – build levels of achievements and pay schemes that allow to achieve and earn more within the same profession
• match expertise and challenges- keep people always interested and challenged, avoid boring or over complicated tasks
• assist self-discovery – help search for the better career and role

perspective

• diversity in tasks and activities – accelerates growth and understanding
• big picture, direct communication with customer – informed people make more optimal decisions
• exposure to different subject areas – broader knowledge opens new possibilities and solutions outside the narrow specialization

learning and practice

• feedback loop – continuous evaluation and improvements based on feedback
• opportunities for learning – grow expertise beside direct work tasks
• sharing, coaching, communication – peer and experts access and support for growth

There is no standard way to increase performance of an individual programmer. Each individual is unique and requires personal approach. The theory of Three Dimensions of a Software Programmer suggests framework for these approaches.
It takes a huge effort to understand every programmer in your team individually and craft personalized approach, path and conditions. But it makes sense to do. People will be productive, engaged and happy. The company will grow and succeed.

What could be a better outcome in our civilized hard working world?

Experts do not need rules to make decisions. They have qualities that allow them to consistently make good decisions and show high level of performance under different circumstances without any rules. This post discusses these core qualities that turn a novice into an expert.

Definitions
rule: prescribed guide for conduct or action
intuition: instinctive knowing (without the use of rational processes)
tacit knowledge: automatic, unexpressed knowledge that provides context for people, places, ideas, and experiences. Tacit knowledge is not easily shared. As Polanyi said: “We know more than we can tell.”
context: the set of circumstances or facts that surround a particular event, situation, etc.

Problems with rules

Most rules didn’t come from heaven. They come from ordinary people. They are product of practice, theories, traditions and fear.
Problems:

• context-free – rules reflect standard situations without considering your specific circumstances
• limited verification – most rules are empirical and do not pass vigorous analysis, strict prove and experiments
• time sensitive – many rules become outdated quickly in dynamic professions, industries and societies
• overcautious – fear fuels many rules and seeks to protect from the worst scenario, often imaginable
• low skills denominator – rules tuned to match capabilities of the majority without accounting for individual strengths and weaknesses
• misinterpretation – tacit knowledge of experts, which forms base for many rules, is difficult to transfer in correct and understandable form

Core 7 Qualities of the Expert

1. Motivation. Believe in self. Energy. Unsatisfaction
Motivation appears to be a more important factor than innate ability in the development of expertise. – Scientific American

You have to believe in self to become a successful expert. An expert need strength, energy and motivation to go beyond ordinary performance levels.
Top performers in different disciplines do this – envision own success and prepare their minds to achieve higher levels. Thinking can “wire” our brains for developing new capabilities and success.
Questions: Am I ready? How can I boost my strength, energy and motivation? How will I succeed?

2. Hard Work. Discipline. Focus.
The differences between expert performers and normal adults reflect a life-long period of deliberate effort to improve performance in a specific domain. – Anders Ericsson

Hard work is not as difficult after you start and dive into it. Often the problem is that you cannot start or concentrate. Procrastination ruins your progress. There are many barriers for focused and productive work:

• pressure and stress kills productivity
• low interest – without engaged mind your brain will resist to form knowledge
• multitasking – takes away precious concentration and mindset needed for productive work
• unclear goals – disoriented mind cannot focus effectively
• no specific time  – you will tend to postpone, delay and miss practice without strictly scheduled time
• distractions – inconvenient  environment and frequent interruptions don’t help

New Scientist had an advice how to get a grip on yourself and strengthen willpower. The main ideas – willpower is limited resource, easily depleted; it requires planning, boost and  practice.

Also, keep yourself interested. Switch if you start loosing focus and interest. Reboot. Have prepared practices in different areas to enable fresh start for your mind.
Questions:  How do I maintain focus, stay committed and interested? Do I have clear goals, action plan and productive environment?

3. Think critically. Think as a beginner
In the beginner’s mind there are many possibilities, but int the expert there are few – Suzuki Roshi

As an expert you cannot go with the flow, believe everything and hide behind authoritative opinions. You cannot stick with your own views forever without changing them. You have to think critically: challenge beliefs, existing theories and dominant ideas. Some of them are completely wrong. Many of them have flaws and can be improved. Most of them are not the best in your specific context.
5 Whys is an excellent method to discover reality.
Better unforeseen solutions exist if you look for them. If you don’t think critically, you will be a follower without much chances to grow your ideas and find better solutions. Keep your mind open and question ideas including your own.
Questions:  What are hard facts, assumptions and theories? What should I trust, dismiss or verify?

4. Full brain power. Use right brain (in addition to left).
The right hemisphere synthesizes over space. The left analyzes over time. – Jerre Levy

Your right brain (more accurately – right brain mode) is inherited from our animal ancestors and shaped by millions years of evolution. Right brain mode is much more powerful, reliable and faster than left brain mode. Right brain works in parallel with images, subconscious mind and deep vast memory (even when you sleep). On the contrary, left brain works with symbols and words; it is logical, analytical and linear. It can only work with few ideas in the same time (4-7) and needs focus and conscious effort. We need left brain to formulate, express ideas and communicate them to others.
Experts are using both brains – left for rational thinking, analysis and communication, right for intuition, imagination and creative insights. One of the best books how to start using you right brain is The New Drawing on the Right Side of the Brain.
Learn how to expand your brain power with the right brain mode. If you use only left brain mode, you’ll stay dull, rational and predictable specialist.
Questions: How can I engage my both brain modes? (Hint: interest, humor, senses, emotions, surprise.) What puzzles can I feed to my subconscious mind?

5. Continuous learning. Sharing
The real voyage of discovery consists not in seeking new landscapes but in having new eyes. – Marcel Proust

Experts never stop to learn. The day you believe you know everything is your last day as an expert. Experts learn from new research and existing knowledge, they learn from other people and they learn from own experience. Also they learn from disciplines outside of their professional realm that can provide new ideas.
For a example, a programmer could benefit from learning:

• psychology – better understand how users think and perceive, how to engage customers, how to make yourself and team productive and motivated;
• design, aesthetics, art – beautiful systems are more usable and successful
  
• management theories and practices – learn how to organize people and activities  (for example Agile learned a lot from Toyota Production System and Japanese product development)
• system theories – understand fundamental principles and laws governing complex systems
• economics, business – why people and business need software, how software fails or succeed on market; how your software fits into company business; how to start your own company
• culture, social life, demographics – trends in behavior, interest and problems of groups of people (by sex, age, education, profession, etc). This knowledge will explain current dynamic and suggest future opportunities.

Make learning essential part of your everyday life. If you don’t learn, you stop your journey to become an expert.
You gain deeper knowledge if you share your knowledge with other people – discuss, explain, teach, blog, speak, present. Other people – with different views and perspective – will quickly show weaknesses and quality of your knowledge. Sharing with others will push your learning further and deeper than keeping knowledge to yourself.

Questions: What do I not know? How will I learn it? How will I share it?

6. Self-improvement. Know yourself.
Why do you see the speck in your brother’s eye but fail to notice the beam in your own eye? – Matthew

We protect ourselves from inconvenient truth and often are blind to own mistakes. Our psychological defense systems help to avoid depression and anxiety by protecting our ego, but also distort our view and perception of true reality.
Good experts are capable to see truth, be objective and correct themselves. They try to avoid the trap of groupthink, crowd psychology and self white washing.  They know own strengths, weaknesses and biases.
Seek  the truth about self, know your internal beliefs and motivation and improve yourself. Otherwise, your rosy distorted pictures will hinder your growth and you will become the part of a problem, not the part of a solution.
Questions: What can I do better? How can I improve myself? Am I honest with myself?

7. Big picture. Systems Thinking. Creative solutions.
Opportunity ideas do not lie around waiting to be discovered. Such ideas need to be produced. – Edward de Bono

Experts main advantages are tacit knowledge and experience. They understand big picture, reality, context and how systems work. However, it is not enough. Experts should solve problems. And therefore, they should train themselves for problem solving, innovation and changing reality in the novel ways. The outcome, a creative solution,  is quintessence of hard work, deep knowledge and intuition.
Questions:  What is the big picture: forces, players and relations? How things can be done better? Do I see new ways?

Questions to Ask Yourself

1. Motivation – Am I ready? How can I boost my strength, energy and motivation? How will I succeed?
2. Focus – How do I maintain focus, stay committed and interested? Do I have clear goals, action plan and productive environment?
3. Critical thinking – What are hard facts, assumptions and theories? What should I trust, dismiss or verify?
4. Full Brain power – How can I engage my both brain modes? What puzzles can I feed to my subconscious mind?
5. Continuous Learning – What do I not know? How will I learn it? How will I share it?
6. Self Improvement – What can I do better? How can I improve myself? Am I honest with myself?
7. Creative solutions – What is the big picture: forces, players and relations? How things can be done better? Do I see new ways?

If you know answers to these questions, you don’t need rules to solve problems – you are ready to make good decisions and become an expert.

References:
Dreyfus Model of Skill Acquisition , Wikipedia
The Science of Experience, by John Cloud/Tallahassee, TIME
Pragmatic Thinking and Learning: Refactor Your Wetware, by Andy Hunt
The New Drawing on the Right Side of the Brain, by Betty Edwards
The Expert Mind , Scientific American
The Role of Deliberate Practice in the Acquisition of Expert Performance, K. Anders Ericsson, Ralf Th. Krampe, and Clemens Tesch-Romer

experts are made, not born – Scientific American

Disclaimer: This post is devoted to a person who wants to become an expert – the top player in a specific field as programming, soccer or chess. This post will be not interesting for people who are satisfied with their current performance and not interested to be the best.

If you want to become an expert, it is not enough to follow your work assignments or occasionally play with interesting stuff at home. You have to push yourself hard in specially designed way.

@Work
Your paid work tasks and projects are not designed to make you an expert. Your company expects results from your work: reliable, with minimal mistakes and focused on the company main goal – make money. Your employer could provide minimal training to help you with job requirements. However, your growth will be constrained by company needs, timelines, work assignments and acceptable methods. We cannot blame our organizations – this is part of the deal – they pay for your work and expect specific results. But… is this the best way for you to become an expert, acquire new skills and gain knowledge? To become an expert, you have to make many mistakes, learn from them, experiment with alternatives and work hard on your weaknesses. How many organizations do allow this risky, unproductive and unreliable way of working?

@Home
Your play at home with interesting stuff has problems too. To satisfy your programming instincts and curiosity, you will probably select what you enjoy to do and eager to try. You’ll immense in this activity and find great satisfaction from doing it. But…  is this the best way to become an expert by doing only what you like? Becoming an expert requires hard, sometimes unpleasant work, specifically designed to improve your performance and push you over comfort zone.

Genesis of an expert

Research shows that experts do not have more innate talent, but put more effort and dedication. Newcomers to the field need at least 10 years to become an expert. They move through several stages: from a novice, who blindly follows rules, to an expert, who does not need rules.
Most people will stop on the level of competent specialist acceptable for the majority of jobs in the field. Only few will continue pushing forward to achieve the top level.

We agree that expert performance is qualitatively different from normal performance and even that expert performers have characteristics and abilities that are qualitatively different from or at least outside the range of those of normal adults. However, we deny that these differences are immutable, that is, due to innate talent. Only a few exceptions, most notably height, are genetically prescribed. Instead, we argue that the differences between expert performers and normal adults reflect a life-long period of deliberate effort to improve performance in a specific domain. – Anders Ericsson

Even the novice engages in effortful study at first, which is why beginners so often improve rapidly in playing golf, say, or in driving a car. But having reached an acceptable performance–for instance, keeping up with one’s golf buddies or passing a driver’s exam–most people relax. Their performance then becomes automatic and therefore impervious to further improvement. In contrast, experts-in-training keep the lid of their mind’s box open all the time, so that they can inspect, criticize and augment its contents and thereby approach the standard set by leaders in their fields. – Scientific American

Kathy Sierra wrote that you, to become an expert who kicks ass, should keep reminding: “I’ll keep pushing myself. There’s always some way to do it better…”. How should we push ourselves?

If you do same tasks over and over again, you will not become an expert. If you do what you are good at, you will not improve your overall performance. You should focus on improving skills beyond your present competence. Outcome of your practice as not as important as developing your capabilities.
You can spend years doing your favorite activities and still stay at amateur level while dedicated novice can quickly leave you behind. Practice should concentrate around your weaknesses and areas that are important, but you probably try to avoid. You should repeatedly progress to the higher level through persistent training and overcome your limits.

Anders Ericsson (his research is the most influential in this area) calls this approach deliberate practice:

“Deliberate practice is different. It entails considerable, specific, and sustained efforts to do something you can’t do well-or even at all. Research across domains shows that it is only by working at what you can’t do that you turn into the expert you want to become.”

Design your practice

Follow 7 steps

1. Who is an expert?
You should understand what is the profile of an expert that you want to be – required capabilities, knowledge, areas of expertise and types of the problems experts work on.
Have a beer with guru and ask for opinion (early enough), study job advertisements for high skilled positions (don’t be carried away), read about top players – why they are considered experts, what makes them special (skip part how they spend money) and how they achieved top level.

2. Find your gaps and weaknesses.
You can identify areas which require improvement once you know the profile of the ideal expert. Put them on the list and assign importance.

3. Build roadmap.
Answer the questions: in what order you want to work on these areas and what are the major steps.
Instead of focusing for a long time on one area, I would recommend to go in spiral – learn and practice various areas in parallel. Each spiral cycle should add to your experience small, but integral chunk of core expertise. Often developing diverse skills together brings synergy – deeper perspective, holistic view and increased capabilities for wider range of tasks.

4. Design practice to improve performance
Set specific goals and methods – how do you improve performance, gain skills and knowledge for each area.
Consider famous SMART goals for practice: specific, measurable, achievable, relevant, time-boxed. Coach, who can effectively guide you, is very helpful on this stage. However, you can go alone if you are motivated and disciplined enough.
Don’t burn yourself too much – your mind and body should be ready and eager to practice. Give yourself time for recovery, life and other favorite activities, otherwise you’ll soon hate your practice.

5. Everyday practice on the edge of capabilities – at least 30 minutes
Deliberate practice is highly demanding and requires intensive concentration and focus. The practice should be scheduled, clear and focused – don’t give yourself chances to procrastinate. Concentrate on the excellence of execution – sloppy practice is not acceptable for top players. You should practice challenging tasks on the edge of your capabilities. Compare playing basketball with professional players or beginner kids from the neighborhood. Who will push you more and show how good you are?
Observe and understand how and what you do – this process will develop intuition and patterns in your brains for future fast automatic decision making. Train yourself to think and act on the higher level – what is difficult for other people, should be trivial for you.

6. Feedback
Constant feedback loop is the must in deliberate practice. Correct assessment of performance and results with following adjustment of practice will move you much faster. And certainly, interested and qualified coach capable of giving constructive and painful feedback is very important. Each day of practice should move you forward against higher and higher standards.

7. Correct your roadmap and practice.
Feedback, self-evaluation and practical use of your skills will tell if you are on the right track. Be fluid and open for change of direction, new knowledge and unexpected opportunities. Learn and adjust goals, plans and methods on the go. As Eisenhower said: plans are nothing, planning is everything. Your rise as an expert will open for you new horizons that definitely change your plans and original intentions.

Additional Requirements

• Discipline, motivation, unsatisfaction – top experts are extremely dedicated, disciplined and motivated to grow and improve performance. Without the goal of improving performance, the motivation to engage in practice vanishes, and practice becomes boring routine.
• Support – you’ll have ups and downs, moments of despair and disbelief. You need somebody who can understand, compassionate and encourage – friends, family, colleagues or a coach.

Deliberate practice in context

Practice @Work
Deliberate practice methods and work objectives are different. However, it is possible to grow as an expert and work productively.

• First, your company could have work in target areas of your expertise growth. Position yourself to take advantage of any opportunity to move there.
• Second, any company will benefit from motivated and competent player, who wants to grow his capabilities. A smart company will support and encourage your drive… and use for own benefits.

Strategy:

1. Job intelligence – what are opportunities, potential projects and tasks to assist with your progress as an expert? How can your help your company while upgrading your skills? Invent new opportunities. You have to build case for your advance and show benefits for the company.
2. Call for the job redesign – many companies will be flexible for people who wants to progress and contribute more (if no, maybe you are in the wrong place)
3. Productive work and intensive practice – work in the best way to help with your goals, but don’t make a company victim of your practice. Separate prototyping and experiments from production work. Make them open, controllable with a clear statement of benefits. Seek for challenging assignments on the edge of your capabilities.
4. Tap into company pool of talents, expertise and knowledge – this is free (actually paid) opportunity to use coaching and valuable resources for growth acceleration. Ask for advises, guidance and feedback.

You will find, as your star is rising (expertise growing), that you have more freedom to work in areas that have the most interest for you.

Practice @Home, @Train or @Cafe
This is time for

• research, learning, thinking
• crazy experiments
• irrelevant to your work practice
• practice that need special focus, time and conditions, difficult to achieve at work

Formal training
Qualified and relevant training can move you much faster. If you decide to invest in education be very specific what you expect and make sure that you’ll not waste your time.

Sample Road to become Web Development Expert
You can identify few key areas:

• HTML / CSS
• Graphic Design / Art
• Usability / Aesthetics
• Information Architecture
• Client – side programming / javascript / AJAX
• RIA (e.g. Silverlight / Flash)
• Server – side programming and databases

And design practice for each of them, for example:

Experts: Executive Summary

Experts are essential for the company/team competitive advantage. They enable revolutions and support evolution.

Experts are made, not born. Experts are made by themselves. It is a long, thorny and hard road. But this road makes their life interesting, positive, meaningful and brings happiness from achieving a rare gift – mastery in the field they love.

Albert Camus said: But what is happiness except the simple harmony between a man and the life he leads?

For few people the harmony is a journey to become The Master.

Resources:
The Expert Mind , Scientific American
How to be an expert , Kathy Sierra
The Role of Deliberate Practice in the Acquisition of Expert Performance, K. Anders Ericsson, Ralf Th. Krampe, and Clemens Tesch-Romer

A really great talent finds its happiness in execution. – Johann Wolfgang von Goethe

source

Qualities of well composed code:

1. Quick discovery and understanding of programming logic and components
2. Clear organization (for human brains)
3. Ease of reuse, modification and evolution
4. Close connection between customer ideas and system implementation

Style Components:

• Intention - understand your task and how to get it done
• Approach - basic principles of writing code
• Composition - organization of code
• Expression - expressing ideas in code
• Object Oriented Pragmatic Style

1. Design for customer problems
Make the customer problem space a core for the software system design. Make every line of code accountable for solving customer needs. It is true that any software system requires a lot of plumbing beyond pure implementation of customer needs. However, technology concerns shouldn’t prevail. High-level languages and modern programming platforms give us a power to concentrate on customer problems more than on solving technical problems.

Code could be divided into two categories:

• customer oriented – solves customer problems directly, e.g. implementing domain logic, UI interactions and display of information;
• system oriented – solves technical and application specific problems – data access, event handling, interfaces between subsystem, utilities and other code required for using libraries, systems and platforms.

Customer-oriented code brings most value and should be primary design concern. System-oriented code puts in place infrastructure and support for running customer oriented code.

Therefore, start with design of customer-oriented code and support it with minimal system-oriented code. This doesn’t mean that system-oriented code is not important. Complex system environments, sophisticated user interfaces and challenging non-functional requirements (performance, availability, reliability, etc.) could demand major system-oriented development effort. But still customers needs should pull and direct this effort and the whole implementation.
There are many ways to make technical problems dominant over customer problems:

1. Making predetermined technical decisions (databases, programming platforms, messaging systems, expensive middle tier etc.) without considering simpler alternatives.
2. Coming up with complex up-front design instead of evolving a simple design into optimal design.
3. Starting programming around interesting technical problems forgetting what is important for a customer.
4. Building frameworks and system-oriented code before writing code for immediate customer problems.

For a example, some teams could start with this picture in mind stretching customer requirements to fit it (even for relatively simple applications).

source
2. Organize and evolve code around domain concepts and customer ideas
Make system design reflecting customer domain and problem space. Keep customer ideas and system implementation connected and synchronized all the time. This is as important as refactoring practice for effective creation of the software system.
Domain-driven design is an excellent way to build software systems. It puts emphasis on ubiquitous language, distilled domain knowledge and shared domain models that drive software development.

3. Think as a customer
What we see depends mainly on what we look for – John Lubbock
A programmer often considers programming as an opportunity for solving interesting technical problems. Often a programmer even doesn’t understand what customers really want. There are 2 consequences. First, the programmer is not interested in simpler solutions focused on the customer problem. Second, the programmer is missing coherent view on the customer problems and purpose of the software system. Over-engineered, complex and misaligned code is one of the most serious problems in software development. An army of business analysts, stressed project managers and smart customer proxies will not force indifferent and ignorant programmers to write code in the best interest of customer.
The programmer should constantly ask himself: “Will this solution work for my customer?”. To give correct answer on this simple question the programmer should understand a lot – customer language, needs, problems, a big picture and business purpose.

Certainly, it is difficult to think as a customer if you are at the bottom of software creation chain:

• separated from a customer by project and product managers, architects, designers, consultants, experts and panels of stakeholders
• pressed by the heavy process, rigid organization structure and inability to make decisions
• fed with over-processed, distorted and disjointed doses of information coming down through long chain

Beautiful software ideas can turn into monster applications – unusable and disconnected from real needs. Did you feel (as a user) frustrated by complex confusing and irrational program screens and logic? Yes, we, programmers, can easily produce bad stuff if we cannot think as people who need our systems.

4. Sketch programming ideas with unit tests.
Write unit tests first. Any programmer should understand what he is trying to solve with the new code and how it will be used – possible input, expected output and public interface. A programmer should think about behavior in isolation, under border conditions or within context of existing code. An unit test is an excellent place to sketch nonexistent code, play with it without compilation and see if it is convenient and makes sense. Once the programmer likes sketched ideas in unit tests, he just need to build simple implementation to satisfy these unit tests. The process of writing unit tests focuses a programmer on solving customer problems, understanding code intent and making optimal design decisions. As an additional side effect, the programmer receives comprehensive suite of automated regression tests and executable specifications.

For a example,

CustomerDB.Save(TEST_USER, TEST_PWD)
Customer customer = CustomerDB.Login(TEST_USER, TEST_PWD);
Assert.IsNotNull(customer, "customer should be logged in");
customer.Purchase(TEST_PRODUCT);
List<Order> orders = OrderDB.LoadFor(customer) or maybe... customer.Orders //what is better?
Assert.AreEqual(1, orders.Count, "count");

There are many design decision are made in this unit test without writing any production code. And now programmer can concentrate on implemention without throes of creation.

5. Eliminate duplication
Simplification is ultimate sophistication – da Vinci
Elimination of duplication is the moving force of evolutionary design. This is straightforward and powerful method to grow an optimal system. Duplication in the code tells that you need better design. Many interesting design ideas are born from the simple need to eliminate duplication in the growing system. These ideas are based on deeper understanding and gained experience as oppose to speculative up-front design ideas.
The main goals of removing duplication are

• reduce code
• simplify solution
• make the system more manageable for programmers’ minds.

Some sources of duplication:

• copy and paste – acceptable only temporary
• similar logic different in details – generalization is required
• another implementation for the same problem – better team communication and reuse are required

Master the core skill of good evolutionary designers – how to detect and eliminate duplication.
The knowledge of design patterns is the most useful on this step. The initial simple solution rarely requires design patterns. But as a system grows, design patterns can effectively and elegantly solve new complex design problems.
Example of elimination of high level duplication with Template method design pattern
This code was used in many data access classes almost the same way, but with different inner steps

        using (SqlConnection conn = NewSqlConnection())

        {

            SqlCommand cmd = conn.CreateCommand();

            cmd.CommandType = CommandType.StoredProcedure;

            cmd.CommandText = "SearchCustomers";

            cmd.Parameters.Add(new SqlParameter("@city", city));

            ... //long list of parameters

            conn.Open();

            try

            {

                using (SqlDataReader reader = cmd.ExecuteReader())

                {

                    foreach (DbDataRecord record in reader)

                        customers.Add(new Customer(record["id"], record["name"]))

                }

            }

            catch (Exception e)

            {

                //error handling logic here

            }

        }

The common logic could be moved into one template method with placeholders for different steps:

List<Customer> customers =  LoadRecordsFromSP<Customer>("SearchCustomers",

    (SqlCommand cmd) => {

         cmd.Parameters.Add(new SqlParameter("@city", city));

         ...     //long list of parameters

    },

    (DbDataRecord record) => { new Customer(record["id"], record["name"]); }

);

6. Reduce code smells
Write code that doesn’t smell. Train your sense of smell for code and design problems. They will provide hints where code is probably bad.
These are common bad smells from a long list

• Duplicated code
• Large method
• Large class
• Long parameter list
• Dead Code
• Over-generalized code
• Lazy class

7. Use design patterns and abstractions to make code simpler
Learn design patterns. At some level of complexity they will provide excellent solutions and simplify design.
However, wrong abstractions and over used design patterns will make code more complex and less manageable. So, apply them carefully.

8. Decouple and isolate components
Only talk to your immediate friends; Don’t talk to strangers – Law of Demeter.
Reduce connections between programming units – subsystems, modules and objects – as much as possible. More relations between components make software system more complex and rigid. Decoupled and isolated system components have significant benefits:

• better understanding – easier to keep in memory and make sense for less states and relations
• increased reuse – components with minimal well-defined public interface and relations less dependent on context and can be used in many scenarios
• decreased breakdowns – caused by unpredictable run-time combination of relations
• effective testing – highly connected system require much more testing beyond isolated testing of components

9. Keep related code together
Put related code to the same package. It will enhance

• discovery - easier to find and reuse
• design - allows isolation, minimal interactions and consistent structure for related components
• maintenance and testing - reduce effect of new changes to fewer packages and system components.

There are two package approaches:

• Feature – code related to the same problem placed together
• Application layers – code for similar programming concepts

Benefits of packaging:

• Feature: related code in one place, changes together and form self-sufficient granule for reuse and release
• Application: separation of concerns, logical layering and reduced dependencies,  consistent global structure

Composing the program – Putting Things Together

1. Understand what customer wants and form intent.
2. Translate customer ideas into programming concepts (technology, platform and language, UI, libraries, components)
3. Make them simple, minimal and clear with pragmatic approach.
4. Organize code around domain concepts and customer needs; use domain-driven design.
5. Describe UI ideas with paper sketches
6. Specify program ideas with automated unit tests.
7. Merge UI and program ideas
8. Package related ideas together, isolate subsystems and components
9. Express ideas in code and UI screens
10. Remove duplication and bad code smells; improve design; consider using design patterns
11. Evolve and refactor common code into framework; emerge layers and abstractions for similar concepts

Pair Programming is a great way to build software systems. When Pair Programming works, it has significant benefits:

• better ideas – continuous brainstorming, larger knowledge pool, less gaps in understanding and more brain power to solve design problems;
• better quality – fewer bugs, instant validation of ideas, consistent approach and stricter adherence to team conventions;
• better knowledge – experience and knowledge sharing, deeper understanding of why, how and what was done;
• increased productivity – better focus and higher intensity, pushing each other and motivating to achieve best results, less procrastination and wasting of time;
• more enjoyment – most people like to work in groups and solve together interesting problems.

Extreme Programming leaders insist that all significant development should be done in pairs. But can we say that Pair Programming is the best method in all situations?

Programmers can find plausible alternatives to Pair Programming that don’t require two people sitting together all the time:

• ideas - frequent team brainstorming combined with short pair (or team) programming sessions for the most complex tasks;
• quality - dedicated tester closely works with developers, writing together automated tests;
• knowledge - frequent discussions, code reviews, training sessions;
• productivity - clear intent and pragmatic approach that bring focus, clarity and efficiency;
• enjoyment - close cooperation and mutual support

When Pair Programming is the most effective method?

The most important factor is match of skills and challenges. In Solo Programming the most productive mode is Flow when skills and challenges are matched. Pair Programming adds one more effective mode – Coaching that increases overall team productivity for the present and future tasks.

Successful modes
1. Flow – two programmers work on the interesting and challenging problem. They can have different skills and challenges, but both are capable to find good solution. For example, one can be javascript expert, another super backend programmer. They can get in flow working together on complex AJAX task combining their brainpower, knowledge and experience to create the best solution.
2. Coaching – expert programmer have experience and knowledge for solving the problem and shares it with other programmer, who cannot effectively solve problem alone. This more junior programmer has fundamentals to understand the solution and implementation. He learns and grows to become better programmer.

Failure modes
3. Wasting expert time – the problem is too simple that makes involvement of expert ineffective even for coaching
4. Overwhelmed novice – the problem is too complex or requires too much new knowledge that prevents the programmer from learning anything useful

Questionable modes
5. Two experts for a manageable task – pair programming doesn’t have much benefits if both programmers know how to implement the task and successfully solved similar problems before.
6. One programmer is in Flow, other is Learning – it is difficult to stay in Flow if other programmer frequently interrupts and needs many explanation for basic things that are not directly related to challenging problem.
7. One programmer is in Flow, other is Coaching – to make this mode successful for the growth, the coach should be open minded, avoid directing too much and should give opportunity for another programmer to come up with own (maybe even better) solutions.

In addition, psychological problems could prevent successful pair programming:

• Intimidation by expert – one programmer is intimidated by superior skills of other programmer and afraid to be considered as incapable
• Need time to think – a programmer needs more time to think over before pair programming, but he is forced to start pair programming session before thinking over his ideas
• Prefer to work alone – an introvert programmer likes to work alone (loner).
• Personal antipathy – programmers don’t like each other

Can we make Pair Programming always effective?
Yes! Match tasks, skills and pairs. Try to find pairs for 2 productive modes  – Flow and Coaching. A team will be most productive if the programmers pair solves together interesting problems matching their skills and learn from each other.

However, Pair Programming should be a free choice and preferred method, but it should not be a forced mandatory practice. As you can see from this post, there are some modes and psychological situations when Pair Programming is not working well.

In short, Pair Programming is one of the most useful tools in arsenal of the software team. Know when and how to use it.

Inspiring resource: Flow, Mihaly Csikszentmihal

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. – Antoine De Saint Exupery

The approach to programming is concerned with finding the best ways to translate programmer’s intention into the good system design and code.

The programming is communication. The programmer continuously add, change and refine ideas in the code. Source code has two important goals: tell a computer what to do and tell people what the computer should do. The program code is the only true medium for storing and communicating ideas about the software system behavior. Quality of the ideas expression in the code directly affects overall quality of the system.

So, what are characteristics of the good code?

1. clear - easier to work with ideas;
2. minimal - less effort to understand and change ideas;
3. testable - easier to validate ideas.

These are 6 top reasons for bad design and code:

1. lack of expertise
2. unrestrained technical curiosity and creativity
3. missing big picture: system purpose and customer goals
4. blindly following popular methods and over-using technology
5. sloppiness; lack of attention to details
6. over-complicating design to have more work or increase job security

The programmer can write better code (and avoid most of these problems) by improving programming style and approach.

Style Components:

• Intention - understand your task and how to get it done
• Approach - basic principles of writing code
• Composition - organization of code
• Expression - expressing ideas in code
• Object Oriented Pragmatic Style

Approach

Two most important rules are

• work from the simplest design
• revise and refactor after each task

1. Work from the simplest design
The simplest design that solves business needs is the best design. The simplest design is the product of discipline and mastery. The simplest design requires creative thinking – it is easier to follow traditional ways than to come up the simple, but clear and powerful solution.

Agile development has famous principle DTSTTCPW (Do The Simplest Thing That Could Possibly Work) best explained by Ron Jeffries:

We’re saying consider all solutions to your task that could possibly work. Implement the simplest solution. Refactor from there if and when needed…  And we’re saying that when you do enhance the simple solution (and you generally will), you will always wind up with a system that is just right for what it does so far. And we’re saying that that is just where you want to be. Everything just right, nothing added that isn’t needed.

For example, if you love design patterns, MVC and object oriented programming, you may end up with design below for finding orders by customer’s postal code.

I love these patterns too, but I would still recommend to start with the simpler design.

The simplest design is the most effective way to deliver business value. You spend less time, build more reliable, easier to understand and change solution. You save energy for other tasks and reduce load on your hard-working brains.

Why do people not go for the simplest solution all the time? There are three main reasons:

• unintentional - the programmer didn’t find simpler solution (lack of experience, pressure and no time to think, relying on traditional ways, compliance with architect’s grand design and others)
• intentional - the programmer wants to over-complicate solution for personal benefits (play with interesting technology, build up resume or increase scope of paid work)
• foretelling - the programmer bets on the future advantage of more complex design

The last reason is the most compelling for many programmers. What if your experience and knowledge tells you that you will need this more advanced design and additional code?
Lets do cost-benefit analysis

What are the chances of mismatch with our initial expectations for a non-trivial project? Big. Mismatch with expectations happens when we learn more from technology, domain and customer feedback while programming. Mismatch happens when we understand the problem deeper and come up with the better solutions. Mismatch happens when the new requirements are coming, priorities are altered and the whole world around is changing (e.g. new company opportunities or credit crunch and economic downturn).

As we can see the best approach is the simplest design that requires little refactoring in the future and doesn’t prohibit anticipated design evolution. Therefore, use you experience and intuition to predict direction of the design evolution. Use this knowledge not for making design more complex to accommodate this prediction, but for ensuring that the simplest solution is not stupid, prohibitive for evolution and will survive coming changes.

However, the level of expertise has significant impact on the choice. The novice programmer should use standard well-described solutions and learn how to simplify them later. The expert programmer should be more comfortable to design simple, sound and easy to evolve solutions from the start.

2. Revise and refactor
Focus on the simplest local design and code for individual features without revisions and design improvements could sometimes create global mess and painful complexity .

For example, it is bad, if after adding several features, your OrderDB contains many methods like LoadForPostalCode, LoadForPostalCodeAndStatus, LoadShippedOrders and web pages are overloaded with domain logic. This kind of code calls for refactoring.

Revise and refactor after completing any programmer tasks.
You should ask yourself 4 questions:

• Can I reduce code?
• Can I make code clearer?
• Can I make code more testable?
• Can I make the whole system simpler?

Refactor, if you answer yes to any of these questions. Learn code smells to identify concrete places that need refactoring. Learn refactoring techniques to improve code.
Main methods:

• remove duplication and increase code reuse

• create abstractions to generalize individual cases
• use design patterns to simplify design

Refactoring is one the most important practice for productive software creation. Refactoring allows to keep the system in the good shape all the time – simple, well designed, ready to change. Implementing the simplest solutions without refactoring and evolutionary approach is risky.

Other rules should enhance your approach further.

3. Discover Form and Function in parallel.
Form follows Function is a traditional principle in design. In other words, customers requirements dictate user interface. However, discovery of complex interactions could affect what the customer wants and change understanding of needs. Sometimes starting with UI paper sketches and discussing them with customers could expose missing details, gaps in understanding and suggest better ways to satisfy customer needs.  Therefore, work on discovery of Form and Function of the system in parallel.

4. Avoid using new tools
New Systems Mean New Problems – The Fundamental Theorem of Systemantics
Do not use new languages, tools and libraries unless existing are bad and you can tolerate high risk and delays.
Every new programming tool claim to make you more productive and powerful. Few of them really do this. But , it is guaranteed, that every new tool takes time for learning and brings new problems.
I know that it is hard to continue using old tools for a long time and stay competitive in fast moving software development world. But try to limit impact on customer. It is not fair to make them victims of your learning and experiments. Create separate experimental prototypes and greenfield projects to evaluate new tools and approaches. Prefer to be honest and transparent with your client about risks of new technologies, tools and approaches.

5. Use creative ideas sparingly
Creative ideas can bring excellent solutions. However, any creative idea adds element of unexpectedness and departure from established ways to solve similar problems. Programmers will spend more time to understand creative ideas and will have challenges to support them. Therefore, use creative ideas when they bring significant advantages (especially to simplify solution) and avoid them in trivial situations. Don’t worry, you will always find where to apply your creative energy if you are productive, build good solutions and have more and more customers.

6. Prefer standard solutions to the non-standard
This rule compliments the previous rule. Standard solutions are based on experience of many people. Maybe they are not the best in your particular case, but they often are safe bet if these problems are new for you. You can find many examples, quickly learn how to apply them and understand consequences beforehand. New creative solutions will make life more interesting in expense of productivity and certainty in the final results. Therefore, prefer standard solutions to reinventing the wheel again. Don’t forget to learn how to make them simpler.

7. Separate production code and experiments
Restrain technical curiosity when writing production code. Many programmers find the joy of programming in solving complex problems. It is so tempting to solve difficult technical puzzles and try out new cool techniques without need. However, the pragmatic programmer solves complex customer problems with simple solutions, instead of creating complex solutions for the simple problems.
Think first about customer requirements and simple design. Experiment and look for the new solutions only if you cannot find simple solutions with familiar methods. Keep your production and experimental code separately. Adopt ideas from experiments without unnecessary complication of the solution.

8. Do not overwrite (or over-eingeneer)
Write minimal code relevant to present customer requests. Surprisingly, complex over-engineered solutions are easier to create – just follow recent trends and try to anticipate everything that came to mind. The simplest and clear solutions require extra effort.
Over-engineering harms the software system, takes away precious time and project money. Avoid over-engineering if you are motivated to build good system and satisfy your customer.

9. Write own code for the core and use external components for supporting functionality.
Concentrate your effort on writing code for the core functions specific to customer domain or code that is bringing competitive advantage. Don’t waste too much effort on secondary supportive functions. For example, there are many good enough libraries, for solving standard dynamic web user interface needs – prototype, jQuery, Microsoft AJAX, Yahoo user interface library and others – but many programmers are still building their own solutions. I like to have full control over my software systems and write as much as possible components, but one cannot embrace unembraceable (Kozma Prutkov). You have to trust some parts of the system to other libraries that will save time and do better job (but bring new problems).

10. Do not write code for future.
The dirty trick of predicting the future is that future is unpredictable. Everything else is fine. Written for the future code adds unnecessary complexity. This code is useless for the present problems. Avoid writing code for the future and strive for the simplest solutions for the present needs.

11. Write code for other programmers
Can other programmers understand your code? Do you have clear names, consistent formatting and self-explaining code? Writing code for others will make code better for you.

12. Switch between big picture and details
Good programmers can focus on the big picture and still pay attention to details. Clear intention and forward thinking are as important as reliable implementation.

13. Pair program for complex problems
Pair programming brings significant benefits:

• better ideas – two brains solving the problems
• better quality – two pair of eyes validating code
• better knowledge – two programmers understand implementation and ideas behind
• better productivity – problems are solved faster
• more enjoyment – people like to communicate and work together

I’m still not sure about serious benefits of pair programming for simple and routine programming tasks. But many Agile teams pair program for all production code and like results.

Putting It All Together

Inspiring reference: The Elements of Style, W. Strunk Jr. and E.B. White

I can’t wait to share this simple secret with you right now.

The Secret: Effective Software Systems are the systems that easy to understand and operate with human brains.

Programmers are more productive with effective software systems. Programmers can better learn and grow these system. Programmers have less problems, work faster and make better decision with them.

Now, you can avoid spending time reading this post if you already know this secret and you know how to avoid building the software system that:

• almost impossible to understand in reasonable time
  
• has confusing and convoluted swamp of logic and structure
• scary to change as nobody has any clue what will be broken, but sure that it will be broken
  

If you are still interested, lets find out what makes software systems effective.

Software Development is a pure mental endeavor (except typing on keyboard) that includes 3 main activities:

• Understand - learn and know system concepts and implementation
• Evolve - build, modify and support growth of the system ideas in the code
  
• Share - communicate and exchange ideas about the system

Programmers should care about 7 areas to make the system better suited for our brains:

1. Knowledge Creation and Retention – parsing, memorization and comprehension of the system ideas
   
2. System Organization – elements, relations and structure in the system
   
3. Sustaining Emerging Order – support evolution of the system and gain control over chaos
   
4. Minimize Noise and Purify – avoid adding unnecessary stuff to the system
   
5. System Discovery and Learning – making sense of the system
   
6. Mental Models – our internal explanations for how things are working in the real system
   
7. Shared Knowledge – ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge.

1. Knowledge Creation and Retention

Information is not knowledge. – Albert Einstein

Programmers should keep as less as possible things in the mind when work with the system. Less information, simpler and more logical ideas behind the code cause less brain damage and more understanding.
Main principles:

• Chunking – combining many units of information into a limited number of units and chunks, so that information is easier to process and remember. Logical groups (subsystems -> modules -> classes) should contain only few elements. These groups should emerge with growth of the system functionality and complexity. They should replace large clusters of difficult to remember similar and unstructured elements.
  

• Abstraction - generalize, find common meaning and remove redundancy in the similar concepts across the system. You can substantially reduce number of ideas required for understanding the system.
• Simplicity - simpler ideas make understanding (and programmers life) much easier.
• Isolation - reduce relations and dependencies between elements. These relations add significant complexity to the system and therefore effort to understand and remember them. The volume of knowledge about relations could easily exceed the knowledge about individual elements.

Examples of useful tools for supporting knowledge formation:

• Unit tests – capture knowledge about the system correct behavior. They evolve with the system and become a parallel verification system. Good unit tests reduce our effort and limit necessary for understanding scope as they focus on the system behavior in isolated components and functions.
  
• Visuals - take advantage of diagrams, mind maps and pictures to simplify, distill and integrate important knowledge and ideas about the system
  
• Storytelling - create imagery, emotions and understanding of the system behavior through stories (our episodic memory is very powerful). For example, create narratives how users accomplish tasks with your system. Or share dramatic stories how programmers doomed the system with ignoring important practices.
  

2. System Organization

Be regular and orderly in your system organization, so that you may be violent and original in your software solutions. (paraphrasing Flaubert)

Human programmers shouldn’t have hard time to memorize and comprehend a software system. Computers don’t care much about the organization as long as the logic is correct, but we, human programmers, do. Clear, logical and consistent organization make us much more productive.
Main principles:

• Modularity - managing system complexity by dividing large subsystems into multiple, smaller self-contained systems
• Layering - process of organizing information into related groupings in order to manage complexity and reinforce relationships in the information
• Hierarchy - hierarchical organization is the simplest structure for visualizing and understanding complexity
  

3. Sustaining Emerging Order

Evolution is not a force but a process. Not a cause but a law. – John Morley

The main challenge for effective system organization is the fact that a system is constantly changing during active development. Good organization today doesn’t mean good organization tomorrow with many new added features. New customer needs, contribution of fellow developers and growing complexity push a software system to chaos.
Therefore, we have to keep organization optimal for today needs and still ready for tomorrow changes with

• Refactoring – constant effort to improve the system internal structure and making it effective for recent changes to prevent system degradation and rule of chaos
• Simple principles and rules (that everybody follows) help to keep consistency and integrity of the system in time of intensive growth and modifications.
• Small independent, single-purpose components – enable rich behavior with minimum number of elements, which could participate in various and often unforeseen scenarios. Bigger components are less reusable, more specialized and lead to more code, complexity and duplication.
• Building adaptive system using agile approach

4. Minimize Noise and Purify

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. -Antoine de Saint-Exupery

Complexity is one of the biggest problem for the programmers brains. While good organization and representation of the system ideas help to fight it, the ultimate challenge is to avoid adding unnecessary stuff to the system.

Principals for Complexity Reduction:

• Increase Signal-to-Noise Ratio – ratio of relevant to irrelevant information in the system. Minimize noise by removing unnecessary elements and minimize the expression of necessary elements.

• Follow 80/20 Rule (Pareto’s Principle ) – a high percentage of effects in any large system are caused by a low percentage of variables. In other words, 80% of requested functionality will be used rarely or not used at all, while substantially increasing system complexity.
  
• Consider Good Enough Alternatives – don’t dash against the rock implementing exactly what customer asks. Slight modifications and alternatives could often cover need with less effort and complexity.
  

5. System Discovery and Learning

The real voyage of discovery consists not in seeking new lands, but in seeing with new eyes. – Marcel Proust

Learning and navigating the complex software system is a difficult task. Usually developers read code, documentation and debug the system. These methods take time and often are not efficient.

A better way to discover the system and make it discoverable with:

• Entry Points – points of attentional entry into the system. There are points in the system that doesn’t require much knowledge about the rest of the system. They could be starting points to understand other elements. For a example, UI screens or the system input / output. Other parts could start making sense and uncover their purpose in relation to these entry points. Discover these points and help newcomers to find them.
• Progressive Disclosure – a strategy for managing information complexity in which only necessary information is visible at any given time. Good design and coding style allow understanding of intent and logic of the studied system area without deeply diving into boundless waters of implementation details.

Code example:
Criteria criteria = Criteria.AddPriceRange(10, 100).AddDateRange(Period.LastMonth);
IList orders = Database.Orders.LoadBy(criteria);
decimal totalTaxes = orders.Sum(order => order.Tax)

• Recognition over Recall – memory for recognizing things is better than memory for recalling things. (e.g. multiple choice questions with possible answers vs. fill-in-the-blank questions). Minimize the need to recall information from memory whenever possible. Use tools to make available options clearly visible. For example, Visual Studio Intellisense and Resharper remove need in memorizing members, purpose and syntax of .Net library classes and methods. Build tools or extend a development environment to construct code without memorizing all details.
  
• Inverted pyramid – method of presentation in which information is presented in descending order of importance. Discover first what is most important and make it easily accessible following by less and less important information.
• Performance load – the greater the effort to accomplish a task, the less likely the task will be accomplished successfully. Most important tasks should be easy to accomplish and most valuable information should be easiest to access (including direct access to the people who has the best knowledge).
  
• Picture superiority effect – pictures are remembered better than words. Use pictures and words together to reinforce the learning.
• Associations with familiar – link new information to already familiar concepts. Build on this knowledge by comparing and contrasting to reach deep understanding.
  
• Examples – learning examples is always one of the best method to understand how to use the system (including best practices, how-to, and tutorials).
  

6. Mental Models

A mental model is an explanation in someone’s thought process for how something works in the real world. – Wikipedia

People understand systems and environments and interact with them by comparing the outcomes of their mental models with the system and real-world domain concepts. Good mental models bring better understanding and optimal decision making.

Schemata – networks of connected ideas or relationships that help to organize experience and information into a meaningful system. Conversation between individuals is most effective if both share common schemata.
Examples:

• Design and architecture patterns – general reusable solutions to a commonly occurring problems in software design.
• System metaphors – a simple story of how the system works. A system metaphor provides common vision and shared vocabulary.
  
• Conventions – standard and well known approaches in architecture, design, code and user interface. They make intuitive and common sense and save mental energy for present and future developers. Custom and novel approaches should be used only if they have serious advantage.
  

7. Shared Knowledge

I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant. – Robert McCloskey

The system is rarely envisioned, built and used by the same person. A successful system is a result of people interactions – intensive ideas exchange, reconciliation of opinions and creation of mutually enhanced knowledge. It is not easy for people to properly translate, communicate and understand each other ideas. Complex knowledge, different background and experience increase the gap that people should jump to completely understand each other.
Main principles:

• Ubiquitous language – the shared language that people with different perspectives use to exchange ideas about the system. The same meaning of the words is very important for understanding, but very difficult to achieve.
  
• Clear goals and intention – help to align people thinking and actions, and avoid wasteful effort caused by misunderstanding or confusion.
  
• Express business domain concepts in the code – represent business concepts in the code. Developers will better understand domain, synchronize implementation with customer ideas and align the system with relevant business concepts. As a result, the system can effectively grow with the refinement and expansion of business needs.
  
• Code readability – good naming and readable code enables better understanding by fellow programmers, reduce chances of errors and lead to correct future modifications.
  
• Shared mental models – frequent face-to-face discussions, pair programming and direct conversations with customers bring shared understanding, effective exchange of knowledge and most optimal solutions.
  

Evolution didn’t consider adaptation of human brains for software development. But our brains is the most important tool for programming. You cannot change brains (and human nature), so make software development compatible with them. Build effective software systems.

Resources:
The Programmer’s Brains At Work: Understanding The Software System
Universal Principles of Design, William Lidwell , Kritina Holden, Jill Butler

“The important thing is not to stop questioning” – Albert Einstein

A typical programmer faces thousands lines of code, huge number of details and millions of situations and states during the software system execution. A programmer should understand the system, know how to modify it and support knowledge about the system with explanations, justifications and answers. How can a software developer gain, maintain and operate this knowledge and make sense of these volumes of information and complex logic changing every day?

The ways how our memory works could give us few hints.

Our Human Memory Architecture

The Process

• Acquisition. New information enters your brain along pathways between neurons in the appropriate area of the brain. The key to encoding information into your memory is attention; unless you focus on information intently, it is immediately forgotten.
  
• Consolidation. If you’ve concentrated well enough to encode new information in your brain, the hippocampus sends a signal to store the information as long-term memory. This happens more easily if it’s related to something you already know, or if it stimulates an emotional response.
• Retrieval. When you need to recall information, your brain has to activate the same pattern of nerve cells it used to store it. The more frequently you need the information, the easier it is to retrieve it along healthy nerve cell connections.

Sensory memory
Sensory memory corresponds approximately to the initial 200 – 500 milliseconds after an item is perceived. The ability to look at an item, and remember what it looked like with just a second of observation, or memorization, is an example of sensory memory. This type of memory cannot be prolonged via rehearsal.
We use five sensory channels to perceive the world: sight, hearing, smell, touch, taste

Short Term Memory (Working memory)
Some of the information in sensory memory is then transferred to short-term memory. Short-term memory allows us to recall something from several seconds to as long as a minute without rehearsal. Its capacity is also very limited: 4-7 items. But short term memory capacity can be increased through a process called chunking – packing information units into small chunks.

The Central Executive is the controlling system of working memory and is more complex than either of the three slave systems. This system controls attention and it is necessary for cognitively demanding tasks such as problem solving and comprehension. The Central Executive also has a limited processing capacity, which is related to the capacity of working memory (4-7 items).

Long Term Memory
Long-term memory extends to hours, days and years. By contrast to sensory and short-term memories, long-term memory can store much larger quantities of information for potentially unlimited duration (sometimes a whole life).
It has two main divisions: declarative (explicit) and non-declarative (implicit) memory.

• Declarative Memory (the memory of facts) consist of
  
  • Semantic Memory – facts and generalized information, contains knowledge about the world; knowing the meaning and definitions of things
  • Episodic Memory – personal experience (information in stories) – events that occur, whether to you personally or to someone you read about, heard about or watched on televisions
• Non-declarative (unconscious memory)
  • Procedural memory is the long-term memory of skills and procedures, or “how to” knowledge.

How to learn the system (or anything else)

Memory is not a muscle and you cannot understand and get much information into your brain with just mechanical effort. You can spend days digging in the code and still barely understand and remember ideas and concepts behind the system. Recommendations below could help to improve the learning of the system.

Prepare mind

The key for effective learning is open, eager and focused mind. You’ll understand much better if you are highly interested or even love what you are going to learn. I don’t know how could you do it – it is too individual ( meditation? self-hypnosis? finding the job you could love?), but without interest you’ll have much harder time to learn your subject.

There are 3 components for the mind preparation:

• Intentions
• Interest
  
• Attention

Intentions
Any learning should have a reason and you’d better know your goals to focus your mind, keep on track and achieve your goals

• Why do you want to learn the system?
• What and how much do you want to learn?
• How are you going to learn?

Interest
You should enter the right state of mind where interest, emotions and mood support your learning beside rational needs. You want to learn, you are eager and passionate about finding meaning and explanations. Your mind will fly and absorb information at the fullest speed if you can achieve this state.

Attention
Without attention you’ll not get information or it will disappear quickly from your brains. Lack of attention prohibits memory formation and encoding of useful information into your brain.

Attention decreases under

• multitasking and distractions: less attention span and more load on working memory
• stress: interferes with ability to perceive and think
• boredom: your brains are not stimulated enough for active thinking
• poor physical readiness: sleepy, poor diet, lack of exercises and oxygen
• long learning time for the same topic. Switching periodically to other topics helps to sharp and charge your mind again
  

Planning

Even prepared mind needs some tools, methods and support

• Adjust your learning approach to your mental readiness, available time and context

• Decide what strategy to use – learning concepts, tools, memory devices, visualization, or reciting. More is better.
  
• Prepare and tune in advance – review, ask and get more information before actual learning

Immersion

haniamir

When you are prepared and ready, you can immerse into the learning.

Immersion (Flow) – a state of mental focus so intense that awareness of the real world is lost, generally resulting in a feeling of joy and satisfaction. When learning and tasks are too easy, people become bored. If they are too complex they become stressed and frustrated. Immersion happens when perception and understanding are challenged near capacity without being exceeded.
Flow conditions:

• can focus
• clear goals
• immediate feedback
• control over actions, activities and the environment

You should try to direct your learning to match your current knowledge and cognitive abilities with the depth and complexity of the new information. If it is too easy, go deeper and chose more complex topics. If it is too challenging, slow down and return to the more basic level.
The flow will allow you to become most productive, motivated and satisfied with your learning (and work and life).

Create knowledge

iqbalkhatri

Full comprehension is not possible, at least not in one day. Our learning is the series of iterative discoveries and probes, which eventually forms the true knowledge in our heads.
Don’t stumble in one place – switch learning areas often. You will have more comprehensive picture. It is more effective to have short, passionate and focused bursts of learning and letting unconscious mind to sort things out until the next learning session.
And in the same time you should be heuristic and ready for discovering the whole new world of knowledge. Be ready for unexpected insights, turns and paradoxes. They could lead for more valuable discoveries and knowledge than you intended.

The sources of knowledge:

• People – the knowledgeable and articulate people are the best source of the information, others could easily deceive and mislead (often including themselves – as you can see we don’t have perfect memories)
  
• Documents – capture mental effort of other people to explain concepts, ideas and solutions, but documents are often outdated and disconnected from reality.
  
• Code – the ultimate source of the system description. It contains the true knowledge , but with too many details (often distracting), which require serious effort to grasp.
  

The essence of understanding is creation of knowledge by building up your semantic memory with

• Associations
• Organization
• Deep understanding

Associate
Associations make memories much stronger and permanent. Linked memory will create a stronger network in our semantic memory, stay longer and allow easier navigation. It is critical for thinking, problem-solving and creativity. More associations we have, the better we are at thinking.

• Form right connection and links between your memories.
  
• Relate information to what you already know and define information using familiar concepts.

• Compare and contrast for creating deeper associations.
• Use senses and emotions for better encoding memories in brain
• Connect information to your and other stories and situations. Episodic memory is very reliable.
  

If you are learning the new programming language, look for similarities and differences with familiar languages.
If you are studying the system design, connect your findings to known architecture and design patterns.
If you are exploring the system concepts, search for a metaphor or visual picture explaining the system behavior.

Organize
Most of the differences in people’s memory abilities are in how they organize memories and less in how well brains can actually remember. Organizing means putting facts in meaningful categories and align with the way how you want to use them.

• Extract basic concepts instead of the simple memorization of isolated facts
  
• Use Mnemonics
  • imaging – creating a mental picture;
  • method of loci (locations) – ideas or things to be remembered are connected to objects located in a familiar location or along a path or journey;
  • pegword method (number, rhyming schemes) – ideas or things to be remembered are connected to specific words (e.g., one-bun, two-shoe, three-tree, etc.)
  • basic mnemonic like ‘initial letter’ – the first letter of each word in a list is used to make a sentence (the sillier, the better).

• Chunk information – pack informational units together. Working memory could operate with only 4-7 items and chunking helps to increase this capacity.
  

4 types of organization:

• Component (part/whole)–classification by category or concept (e.g., overall system architecture –> major subsystems and layers –> main classes);
• Sequential — chronological; cause/effect; flow (e.g., the system logic, sequence of events and dynamic execution);
• Relevance — central unifying idea or criteria (e.g., business model, architecture principles, design patterns);
• Transitional (connective) — relational words or phrases used to indicate qualitative change over time (e.g., the system state transitions, modes of operation)

Deeply Understand
There are three parts for deep understanding and active high-performance thinking:

• Power of unconscious processing
• Depth of processing
• Top down and bottom up
  

Use unconscious memory processing.
Our automatic, unconscious and ultra fast brain processes are much more powerful than limited conscious controlled thinking. Unconscious processes run in parallel and can handle many tasks at once. They are source of our intuition, insights and most decisions (for which we try to find conscious explanations later).
Unconscious thinking could be trained, enhanced and boosted with experience and active thinking, when we stretch and challenge our knowledge

• question all facts and assumptions
  
• explain and teach these ideas to others in the simple terms (including your grandmother – you’ll quickly find gaps in your understanding)
  
• view from different points of view (including various people interests)
  
• try to find alternative and new ideas
• search what we are missing or cannot trust
  

Depth of Processing - phenomenon of memory in which information that is analyzed deeply is better recalled than information that analyzed superficially. The more deeply learners process information, the better they learn. Therefore, select depth of processing depending on importance of the learning area: dive deep into the most important topics and just barely scan insignificant.

Combine top down and bottom up approaches.
We should use both inductive (example/experience –> concept) and deductive reasoning (concept –> examples) to better understand the software system. Requirement documents and architecture views are not enough and often far from reality. Code provides too much details that sometimes difficult to translate into meaningful concepts and unifying ideas. Only reconciliation of top down and bottom up knowledge could provide you with the true knowledge in reasonable time.

Strengthen learning

Important steps for knowledge retention and becoming better learner:

• Experience
• Review
• Self-correction

Experience your learning as soon as you can. The true knowledge emerges only after you tried to apply it in practice. Convert your theoretical knowledge into practical knowledge tested by reality. It is especially important fore software development, because it is mostly empirical activity.

• Write code and add / change functionality
• Troubleshoot and fix bugs
• Create unit testing – one of the best methods to test reality, understand and experience the system (if the system is testable).

Review

• Rehearse and “over-learn” information that increases level of attention, utilizes more senses, provides immediate feedback
• Review soon in small frequent doses
• Recall main points

Self-correction. Watch yourself. Are you effective? How could you learn better? Criticize your knowledge and learning methods. Praise yourself for achievements. Become more effective at learning.

The learning and exploration of the software system could be the most rewarding intellectual experience or it could be your worst nightmare. You have a choice…
And it is true, sometimes the problem for learning is in the a software system, which is very difficult to understand. The next post will explore how to make the system understandable.


Interesting Resources
The Information Processing Approach to Cognition
Memory and Semantic Memory, Wikipedia
Human Memory Models
Five Tools for Improving Your Memory, by Barry Gordon, M.D., Ph.D.
Improving your Memory