Thought-provoking and interesting author Nicholas Carr has published a new article in Financial Times, where he expands on his ideas about transformation of IT services into utility. He compares IT with electric power generation 100 years ago:

Like data-processing today, power generation was assumed to be an intrinsic part of doing business. But with the invention of the alternating-current electric grid at the turn of the century, that assumption was overturned.

Suddenly, manufacturers did not have to be in the power-generation business. They could run machines with electric current generated in distant power plants by big utilities and fed to their factories over a network of wires. With remarkable speed, the new utilities took over the supply of industrial power. Scores of private power stations were dismantled.

Nicholas Carr applies the same pattern to the transformation of in-house IT services:

One place where the changes will be particularly sweeping is the corporate IT department. As the capacity and capabilities of the computing grid expand, it will continue to displace private systems as the preferred platform for computing. Businesses will gain new flexibility in assembling computing services to perform customized information-processing jobs. They will no longer be constrained by the limits of their own data centres or the dictates of a few big IT vendors.

He predicts unfortunate future for in-house IT:

In the long run, the IT department is unlikely to survive, at least not in its familiar form.

It will have little to do once the bulk of business computing shifts into what Google’s CEO, Eric Schmidt, calls “the computing cloud”. Business units and even individual employees will be able to control the processing of information directly, without the need for legions of technical specialists.

I don’t agree with comparison of power generation to software services. As far as I know, electric power is the product of potential difference and the current. It differs only by watts. Software services are much more complex, variable and diverse. And it gives a hope for survival of in-house IT contrary to Nicholas Carr predictions.

There are several factors that will help survival of in-house IT and especially in-house development teams:

1. Specialization. It is true, many computing services are similar or even identical in many companies: e-mails, word processing, CRM, etc. There is no point to reinvent the wheel, build such applications and even host them. Google or Salesforce.com can host them with one tenth of what it cost to a typical company. But each company has unique business, people and information processes. Software services, which are closer to the core business of the company, will have much higher degree of specialization. You’ll find much less software services specialized for chemical, publishing or brokerage companies and they are really expensive. And it will be very difficult to find the combination of external services which completely matches company needs.
2. Integration. Many computer services are not homogeneous, have incompatible interfaces and couldn’t be integrated easily. They process, represent and interpret information differently. As many programmers know, creation of generic reusable services requires much more effort than creation of narrowly focused for specific use services. Often generic services are complex and still require effort to adopt and integrate into company IT processes. Therefore, “assembling computing services to perform customized information-processing jobs” will be really challenging job.
3. Competitive advantage. For many companies IT processes and services lie at the heart of their business. They are very important for beign effective, profitable and outstanding. Wal Mart, Fedex, Dell, Amazon and other companies built success on proprietary information systems. Outsourcing of traditional IT services will take away benefits of custom tailored information processing.
4. Adaptation. Any business evolves and frequently changes. Relying on outsourced computing services will make company less flexible and responsive to the external and internal changes.
5. Knowledge. Finally, somebody should understand how to apply, use and tune technology for the particular company’s business. People who understand both business and IT will be still needed in-house. Developers who know how to build and integrate effective software solutions will be in demand.

In overall, Nicholas Carr correctly predicts the trend – more and more software services will be outsourced and become utility, especially for smaller companies. How could in-house teams or software companies stay competitive to the big providers of computer services?

I believe they should follow two strategies:

1. Become more effective in translation of business needs, building software systems and integrating outside computer services. Master agile approaches with short feedback cycles, stay lean and avoid over-engineering. Build software that satisfy business needs and easy to use with minimal time and cost.
2. Become technology partners for business (including every developer), understand core business domain and how software could help for the company success. Build software that amplify business capabilities and make company more effective.

Link:
A revolution is taking shape, Nicholas Carr

You can find more in Nicholas books: The Big Switch and Does IT Matter?

Programmer’s Side

Programmers in the software team work well together, trust and support each other. But they don’t talk about their salaries – the most important reason why they work for the company. It is taboo. Tell them that one of their fellow programmers earn 50% more than others. You’ll see smiles disappear, people don’t look at each other and become silent. They will rebound, but these things leave scars on team morale and cohesiveness. Feel of unfairness is not the best companion in the software development.

But why does compensation is so important and sensitive for us?

• We work to live. We spend more than half of our conscious adult life earning money on work. We need money to live. Money enable for us almost everything material in this world. Any non-material incentives cannot compensate insufficient money income.
• Money are social status. The better compensation means you worth more and more attractive from social and even biological perspective. People strive for relative prosperity. They could accept less themselves than see a rival get more.

Company’s Side

The software company is interested in the best productivity, high quality and most optimal software solutions. That usually translates into lower cost, higher profit and happier customers.
How a company can make it possible? People are the most important factor to achieve these goals. Company just need to

• Keep programmers motivated, interested and satisfied. Programmers will be happy if they perceive compensation as fair, work as interesting and environment as comfortable.
• Reward desired behavior for the company success – accountability, creativity, intelligence, self-discipline, learning and may other useful qualities. Compensation should link these qualities to the individual, team and company performance.
• Attract and retain the best contributors and talents. Programmers should have opportunities for earning more for better contribution. Many of them possess entrepreneurial spirit and stiff salary ranges and limits for growth can make them leave.

What is the best way for the company to approach compensation and support these goals?

Compensation Challenges

A company should solve few challenges in order to build an effective compensation system.

• How can the company objectively evaluate individual accountability, creativity, intelligence, self-discipline, learning capabilities, experience, influence, coaching, attentiveness to details, vision, strategic thinking, cooperation, enthusiasm, work ethic, quality, speed, thoughtfulness, inspiring and motivation of other people and many other qualities? What of them are most important for the company success?
• How can compensation encourage collaboration, cement relations and avoid competition between team members?
• How to get great team productivity results, without negative impact on other teams and without sacrificing long-term perspective?
• Finally, what compensation system will be perceived as fair?

Considerations:

1. Objective behavior evaluation is not possible. How can evaluation account for the huge number of behavior variables? It is almost impossible. In addition, some people traits play well only if they are complimented by other people traits: visionary needs detail oriented performer, creative person needs pragmatic opponent, innovator needs patient adapter. That is why all traditional evaluations will never give true picture of contribution and certainly will miss important points.
2. Fairness is in transparent and equal criteria. There are different types of criteria possible: years worked in company, level of education and even produced lines of code :). However these criteria don’t guarantee desired results.
3. The most effective economic criteria – profit. The most effective and simple criteria is found on the market where company operates – profit. It is a bottom line and result of the whole company activity. This is a reason why company hires employees and their compensation should reflect company economic results.
4. Collaborate inside. Free market promote competition which could easily reach all levels in the company. Competition is good in a healthy dose, but too much can damage cooperative nature of the software development inside teams. Also competition between teams and business units is damaging too as it could cause suboptimal decision on the company level.
5. Compete outside. The solution is to make people compete with the same competitors as the whole company facing on market. Link compensation to the whole company economic performance. Measure Up.

Fair Compensation System

There is a wide range of existing compensation systems:

1. Closed vs. Open. Opens salaries will force compensation system to become fair. Secrecy is damaging and corrupting. Open salaries should eliminate personal bias, favoritism and undeserved power from management. People with higher salaries will feel pressure to justify them and show their best performance; otherwise their status will be questioned.
2. Fixed vs. Performance based. Fixed rates promote inefficiency and detachment from the company results. People should have direct dependency between results of their work and the company performance.
3. Driven by management or teams. Team members are in the much better position to determine most valuable contributors and ‘free riders’. Managers don’t have this insight into each team member work. They are not the best people to decide on individual contribution and compensation.
4. Equal salary for levels vs. Individual negotiations. While individual negotiations are more flexible, they could promote bargaining and lack of transparency. People should be more concerned about performance of the team and company and less about individual negotiations. Some compensation parameters could be negotiable, but they should stay within baseline for the same contribution level.
5. Non-money incentives and free perks. They are nice, but not important for all as oppose to compensation packages.

There is no one approach that works for all, but 3 approaches below could increase efficiency of the compensation system. Additional note: effective software team includes all contributors – business experts, designers, programmers, testers, system administrators, etc.

1. Compensation Composition. A company should be flexible in packaging compensation. Some people expect secure and stable income, some are willing to work harder and risk for opportunity to earn more. Use mutual funds approach – allow custom packages (Stability, Income, Growth) with various components. They could include base salary, profit sharing (both depending on teams) and monetary options (partly covered by the company): training, health insurance, pension plans, vacation, stocks, equipment, car expenses, etc.
2. Team Contributors. Team should have minimal number of contributor’s levels: Junior, Full and Senior Contributors. Most people in the team should be Full Contributors and only outstanding members should be Senior Contributors. Each level has fixed ratio of income (e.g. Full Contributor – 25% and Senior – 50% of Junior compensation). The same level makes roughly the same money (depending on packaging). The team makes decision about moving person from one level to another or removing from team at all.
3. Company input. Company provides initial number of roles, budget and profit share structure. Team contribution is measured based on the profit or other criteria if the company on early market expansion (number of users / traffic, share in cost reduction, etc.). Each team receives part of the profit from their activity. Other part comes from the company pool (formed from remaining profit of other teams). Therefore, teams are interested in both their and overall company performance. Product teams have larger share of their profit, infrastructure teams have larger share from the company pool.

Fixed secretive compensation turn a team into strangers or even rivals, concerned only about individual achievements . Open fair and performance-based compensation motivates, retains and attracts programmers eager to create great, useful and profitable software.

Interesting resources:

• Unjust Deserts, Mary Poppendieck
• Measure Up, Mary Poppendieck
• Extreme Programming Group discussion
• Why secret salaries are a baaaaaad idea, Alexander Kjerulf
• Fog Creek Compensation, Joel Spolsky
• Employee Rewards, Accel Team

Software Creation is very dependent on oil. Our computers, keyboards and printers are made from plastic. Our cars use gasoline to get us to work. Furthermore, almost everything in our modern civilization depends on oil. Global oil production will peak any time from now. Will end of oil cause crash of our civilization? I was always worry about this :)
New Scientist assures that governments, technology and biology will rescue our civilization.

• 70% of oil is used for fuel; petrochemicals are only 3.4%. Technology, government regulations and social trends are actively influence reduction of oil usage for energy and fuel. Sweden, for example, plans to be independent of oil exports by 2020, using no oil for heating and halving its transport fuel consumption.
• The new breed of biorefineries is using biomass to produce biofuels and bio-based chemical products. The biomass could be supplied by anything from corn, sugar cane, grasses, wood and soybeans to algae. For example one company is converting turkey guts into oil and fertilizer. By 2025, the US Department of Energy wants 25% of industrial organic chemicals to be delivered from biomass. There are already 120 ethanol refineries in US.
• Biorefineries are more profitable at smaller scales than conventional petrochemical refineries, because they operate at lower temperature and pressure, so they are less complicated to build. However, it is still difficult to compete with some oil-based products.

Therefore, life will continue after oil is over. Proactive thinking, society and government involvement could significantly reduce impact, especially considering that solutions correspond with fighting global warming.

Disconnect between business and IT is increasing and could signal serious change how enterprise technology is run, reports IT World Canada. Michael O’Neil from Info-Tech Research Group thinks that the future of the IT professional no longer lies in acting as an important support system or innovative visionary but as a “utility.” The role where IT functions as a discrete, advisory body will disappear while the majority of IT professionals are assimilated into areas pertaining to business processes and strategy.
There is also an interesting discussion on InfoQ if Agile can rescue this widening gap.

Comments: I don’t think IT should be rescued. The narrow specialization of IT and Software Development professionals and their remoteness from the problem domain and decisions are the serious problems in software creation. I consider disappearance of specialized IT services and departments as a good trend. I hope that combination with another trend – disappearance of centralized and hierarchical corporations – will produce powerful cumulative effect. The companies will become conglomerates of small cross-functional and diverse teams, empowered to make most decisions. Programmers will have deep domain expertise in addition to specialized knowledge how to build programs. Benefits are better decision making, market response and people motivation. See my post The Ideal Software Company.

If you want to build a ship, don’t herd people together to collect wood and don’t assign them tasks and work, but rather teach them to long for the endless immensity of the sea. – Antoine de Saint-Exupery

I have a dream about a company that always makes users happy. The company that quickly builds high quality, simple and usable software that fully meets user needs. The company that reacts rapidly on customer demands, anticipates new needs and immediately use these opportunities. The company where people are happy, motivated and could realize their potential, dreams and life goals. The company that is successful on market and in the hearts of users, investors and employees. The Ideal Software Company.

The Barriers

What are the main problems with building software? What are contributing factors to these problem? What prevents many existing software companies from being ideal?

1. Unclear, incomplete or misinterpreted user needs. The software programs often don’t solve needs on adequate level, have bad usability and contain many unused features.
2. Difficulty to estimate required effort. Missed deadlines, doomed schedules and budget overruns are the common deceases in software development.
3. Inability to adopt to changes. Often it is even impossible to estimate effort at the beginning as many things are changing during the course of the project: customer needs, performance requirements, business context. But for many software companies these changes still bring chaos to their projects.
4. Poor, rigid or low quality solutions. Implementation defines project success. Talent, experience or knowledge are necessary to build good solutions, and they are often missing in the project teams.

The main reasons for these problems

1. Top-down decisions – only few people make decisions with limited information, lack of feedback and independent information
2. Poor aggregation of information from different sources
3. Lack of diversity: homogeneous groups that are responsible for only specific aspect of the project (requirements, design, programming, etc.)
4. Lack of accountability for the end results, mismatch in company and personal employees goals, politics and suboptimal decisions.
5. Low motivation and morale – minimal control over decisions, poor relation of incentives and payout to contribution into the company success.
6. Centralization – authoritative control causes inflexibility, lack of independence and adaptability; missing local tacit knowledge in decision making.

James Surowiecki in his book The Wisdom of Crowds clearly shows that traditional top-down companies are inefficient in making the best decisions.

The Ideal Software Company

A. Company organization

1. Powerful teams – the company represents a conglomerate of powerful teams. Projects teams are small, diverse and empowered to make most project decisions.
2. Egoless management – management carry mostly support, coordination and representation functions. Leadership is moved to the teams level. Top management is a consulting body, which includes highly skilled professionals in finances, technology, sales, etc.
3. Shared and competing infrastructure – company infrastructure could include services as hosting, customer support, recruitment or distribution. These in-house services are shared and compete with outside providers.

B. User needs, product and feature ideas

1. Generate many alternatives – the company uses diverse sources (internal, user community, paid analysts and experts, etc.) for ideas generation
2. Predict most successful ideas – the company establishes prediction markets for deciding forecasting most successful ideas with large group of participants – customers, user community, domain experts, managers, designers, and developers. They make independent guesses based on their private knowledge.
3. Select the best ideas – any team could use this information to sign up for implementation of the product or feature ideas.

C. Team is a business unit

1. Full responsibility – a team is ultimately responsible for all decisions and end-results. The team selects product ideas, come up with solutions an implements them.
2. Diverse teams – teams are diverse, self-sufficient and include people with different background, areas of expertise: marketing, domain experts, UI designers, developers, system administrators, testers, customers and community representatives.
3. Organic composition – the team can choose its composition, hire (or outsource) an additional or remove poor performing member; selects in-house or outsourced services (hosting, customer support, etc.)

D. Team formation

1. Talented people – the company hires talented people with diverse background, skills and experience. The only requirement is high qualification, creativity or expertise.
2. Easy start – Anybody in the company can initiate and assemble the team for implementation. The team applies to the internal VC group (see below) for financing and approval of the project.
3. Easy scalability – the team can contribute to open-source projects (at least for non-core and non-competing features and modules) and easily scale beyond company borders.

E. Sharing and learning

1. Shared best practices – the company promotes the best practices, tools, designs, conventions and approaches from internal and outside sources. Teams are free to select and use any of them.
2. Cross-team training – the company organizes cross-team knowledge sharing and training.
3. Continuous improvement – the company encourages regular reflection and continuous improvement sessions, where the organization, processes, company and teams interactions are scrutinized and changed.

F. Financing, incentives, internal market

1. Internal Venture Capitalist group (VC) – the company has something like Venture Capitalist (VC) group, which includes management, investors representatives and leaders of top teams that approves and finances new projects and teams. Teams are starting with minimal budget that barely covers their first needs.
2. Team value market – The company has internal market for the value of each team. This value is linked to success criteria established by company: (revenue, number of users, cost saving, use inside company, buzz).
3. Pay based on performance – Team budget, payout and incentives are directly corresponding with their value. More successful teams will get more money, resources and influence in the company. VC group disassemble unsuccessful teams. While any employee has guaranteed minimal salary, the people who doesn’t contribute in any team will be fired. People who performs will be very well rewarded.

G. Mobilization

1. The company in time of emergency or especially large projects can force creation of special task teams (as National Guard). Company leaders should approve this temporary mobilization and limit impact on other projects

The Benefits

The Ideal Software Company should overcome most barriers.

• Teams are diverse, independent and make most decisions. Information flow is free and the company ensures that teams receive it most effectively.
• Individual income is directly linked to their team performance (but not individual contribution inside the team). Teams compete with market not between themselves.
• Teams can still uses benefits of centralized services, coordination of activities and shared knowledge. Company organizes prediction markets, internal VC group and provides financing of the projects.
• The system is adaptive, lean and focused on the market and user needs. Bureaucracy is almost absent, the company need only few coordination mechanisms, the approach is compatible with our human nature and self-interest. It is combination of adaptability of decentralized and power of coordinated system.

What if we have only one team? We receive a small, lean and powerful startup. As a startup becomes bigger, it is often turns into inefficient traditional corporation with hierarchical control. Many startups lose their flexibility and adaptability once they become bigger. The Ideal Software Company should avoid this fate with the growth.

Is it possible?

I see many potential problems with creating The Ideal Software Company – political (most managers are not egoless), economical (internal financing schema, appeal to investors), organizational (new ways to employ people). But I believe it is possible.

Agile development processes (Scrum, Extreme Programming, Crystal Clear) are changing the role of people and teams in software companies. I know few companies that are close to the ideas of The Ideal Software Company. Google in software world, 3M and Toyota in manufacturing. Google has free flow of ideas, powerful self-organized teams and attention to user needs. 3M has entrepreneurial cross-functional teams with focus on business goals. Toyota become #1 auto manufacturer because their Toyota Production System gives unprecedented power to the teams and continuous process of improvement and solving root problems.

Is The Ideal Software Company utopia? What do you think?

Who is smarter – a diverse group or the best expert? Can a boss, a project manager or the most experienced specialist make better decisions than a whole team? What are the problems with decision making in traditional corporations?

James Surowiecki in his book The Wisdom of Crowds argues that given the right conditions a group consistently outperforms best experts in decision making. Few people can do better than the group, but each time they will be different people. A group answer will often be at least as good as an answer of the smartest member.

The author provides interesting examples:

1. 800 different people (mostly non-experts) tried to guess weight of an ox at the International Exhibition of 1884 in London. The average guess of the crowd was only one pound off (ox weighed 1,198 pounds) and was better than any estimate of the cattle experts.
2. Stock market knew what company was responsible for the Challenger disaster within a half hour of the shuttle blowing up. Mostly uninformed investors dumped shares of Thiokol almost simultaneously.
3. Team of the men with wide range of knowledge (e.g. mathematician, submarine specialists, salvage men) correctly estimated location of disappeared U.S. submarine Scorpion. The submarine was found 220 yards from collectively estimated spot (original search area was 20 miles circle) and it was better than any individual guess.

Crowd is wise when these criteria are present:

1. Diversity of opinions (each person have some private information)
2. Independence (people’s opinions are not determined by the opinions of others)
3. Decentralization (people are able to specialize and draw on local knowledge)
4. Aggregation (some mechanism exists for turning private judgments into a collective decision)

Groups vs. Experts

• Groups are better at deciding between possible solutions to a problem than coming with alternatives.
• Crowd intelligence is most useful for broad skills as decision making, problem solving or forecasting unknown future.
• Experts judgment is most useful in specialized areas as programming, design, and even management, where experience, skills and talent are important.
• Experts lack “judgment calibration” – they often overestimate their knowledge and performance (except maybe weather forecasters).
• Advices and predictions even from the best experts should be pooled with others to get the best answer.

Diverse groups vs. Homogeneous groups

• Diverse groups add new perspectives, wider set of possible solution and approach problems in novel ways.
• Homogeneous groups, particularly small ones, are often victims of “groupthink”:
  • Information that might represent challenge to the conventional wisdom is either excluded or considered as mistaken
  • Homogeneity fosters pressure toward conformity. A member changes own opinion as it is easier than to challenge the group.

Independence vs. Imitation

• Independence is important to intelligent decision making for 2 reasons:
  1. It keeps the mistakes that people make from being correlated.
  2. Independent individuals are more likely to have new information.

• Lack of independence brings following problems:
  
  • “Social proof” – tendency to assume that if lots of people are doing something or believe something, there must be good reason why. The crowd becomes more influential as it becomes bigger. The governing assumption is when things are uncertain, the best thing to do is just to follow along.
  • “Information cascade” – to supplement their information people will look what others are doing. If the first couple of people get bad information they will make wrong choice. Everyone who follows, even if they have better information, will make the wrong decision too.
  • “Herding” – sticking with the crowd and failing small, rather than trying to innovate and run the risk of failing big.

• However, imitation works much of the time. Each person can’t know everything. With imitations, people, can specialize and the benefits of their investment in uncovering information can be spread widely when others mimic them.
• Imitation also requires little top-down direction. It is a powerful tool for spreading good ideas fast.
• Imitation work when there are:
  1. Initially wide array of options and information
  2. Willingness of at least some people to put their judgment ahead of group’s.

Decentralization vs. Top Down Authority

• In terms of decision making and problem solving these things matter:

1. Specialization - increases the scope and the diversity of the opinions and information in the system.
2. Tacit knowledge – can’t be easily summarized and conveyed to others, because it is specific to a particular place or job or experience, but it is very valuable.

• Decentralization great strength – encourages independence and specialization while still allowing people to coordinate their activities and solve difficult problems on the other.

• Great weakness – no guarantee that valuable information will find its way through the rest of the system.
• Decentralized system can only produce genuinely intelligent results if there is a means of aggregating the information of everyone in the system (e.g. price for free market).

Traditional Corporations

In the traditional corporate model, top management hires the best employees it can, pays them and gives some direction about the problems to work on and hopes for the best. It is great for mobilizing people and get very good at doing things they know how to do. But it is also necessarily limits of the number of possible solutions (company has limited number of workers and their time) and because of reality of organizational and bureaucratic politics.

Coordination and decision making problems when power is concentrated on top

1. Costly in terms of time.
2. Requires too much information
3. Saps initiative of workers and mangers
4. Discourages free flow of information
5. Incentives skewed against dissent and independent analysis
6. Lack of diversity among top managers, insulation of top managers from competition

Solutions:

• Elimination of rigid managerial hierarchies
• Wider distribution of real decision making power

The virtues of decentralization:

• More responsibility people have for their own environments, the more engaged they will be
• Coordination – instead of orders and threats, companies rely on workers to find new, more efficient ways of getting things done. That reduces need for supervision, cuts transaction cost and allow managers to concentrate on other things. Toyota Production System is a supreme example.

Tacit knowledge (emerged only from experience) is crucial to efficiency of markets and as just as important for corporations. Companies should start with the assumption that, just as in the market place, people with local knowledge are often best positioned to come up with a workable and efficient solutions. The benefits of specialization and local knowledge often outweigh managerial experience in decision making.

How these ideas could be applied to the Software Development? See the next post.

Secrets of Creativity

http://www.sciammind.com/article.cfm?articleID=00028EE8-4369-123A-822283414B7F4945
Why some people are more creative than others? The Scientific American article provides some insights:

• Intelligence is not a crucial ingredient. A crucial variable is the difference between “convergent” and “divergent” thinking.
• Convergent thinking aims for a single, logical and correct solution to a problem. Divergent thinking proceeds from different starting points and changes direction as required
• Creative people can free themselves from conventional thought patterns and follow new pathways to unusual or distantly associated answers, which leads to multiple solutions, all of which could be correct and appropriate.
• These two processes took place in different brain regions – left hemisphere is responsible for convergent thinking and the right hemisphere for divergent thinking.
  • The left side examines details and processes them logically and analytically but lacks a sense of overriding, abstract connections.
  • The right side is more imaginative and intuitive and tends to work holistically, integrating pieces of an informational puzzle into a whole.
• Our creative talent is gradually repressed from childhood. Schools place emphasis on teaching children to solve problems correctly, not creatively.
• The brain is a creature of habit; using well-established neural pathways is more economical than elaborating new or unusual ones, which wither with time if not used.
• Creative people are generally very knowledgeable about a given discipline, it is very improbable to come up with a grand idea without deep involvement in this area.
• Creative solutions result from examining challenge from all sides, disassembling and reassembling the building blocks in an infinite number of ways. The problem solver must thoroughly understand the blocks.
• Too much specialized knowledge can stand in the way of creative thinking. Experts often often internalize “accepted” thought processes, so that they become automatic. Intellectual flexibility is lost.
• Creative revelations come to most people when their minds are involved in an unrelated activity. A little relaxation and distance changes the mind’s perspective on the problem – without us being aware of it. The brain clear away thought barriers by itself and at some point, newly combined associations break into consciousness, and we experience sudden, intuitive enlightenment.
• The neural processes that take place during creativity remain hidden from consciousness and we cannot actively influence or accelerate them.

Comments:

1. Creativity plays a very important role in the software creation. IQ is not enough for developing the best programs as it will lead for technically complex, standard and expensive solutions. Solving changing business need with limited resources, dealing with inconsistent people and complex systems requires unorthodox solutions.
2. Companies should start hiring people based on “Creativity Quotient” in addition to IQ and EQ. The article has interesting examples of the potential interview tests.
3. We should specifically train our creativity and always try to look for solutions in novel ways, considering unusual alternatives, combinations and associations; breaking traditional approaches. We should strive to come with more than one solution.
4. Creativity needs knowledge, serious preparation and understanding what challenges to solve. The brain will not work on the problem until it has enough food for thoughts :). And brain can work out solutions without our conscious involvement, automatically. We just need to learn how to prepare our brains properly.

Material Software Companies

http://technology.guardian.co.uk/opinion/story/0,,2112850,00.html
Software companies are coming to earth and turning from nonmaterial to very physical companies. Nicholas Carr explains why.

• Google has data centres around the globe, reportedly holding as many as 2m or 3m computers altogether, and it continues to spend billions of dollars a year to put up new ones. Its arch-rival, Microsoft, is building a 140,000 sq m six-building data centre on a former bean field. Also rushing to build or lease data centres are Yahoo!, Ask.com, Intuit, Salesforce.com and Deutsche Telekom’s T-Systems unit, among many others.
• Nature of the software business is changing. In the past, software companies only had to concern themselves with writing code, copying their programs on to discs and selling them. It was up to the buyers of the software to maintain the computers, storage drives and all the other hardware needed to run the programs. Now companies are beginning to rent programs over the net for a monthly fee. Even sophisticated applications for maintaining customer accounts, tracking finances, managing workers and performing other complicated tasks are now being offered as web services. The burden of running software is shifting from the buyer to the seller.

Therefore software companies are finding that they have to compete not just on the elegance of their programs, but on their ingenuity and efficiency in buying and deploying physical assets – land, buildings, computers, and other gear – as well as managing the huge amounts of energy required to keep all the machines running.

Comment: I feel we will miss this nonmaterial world, where we should care only about own computer, a table and a chair to write programs :) . But I still believe that the essential and core business of the software companies will be programming, not management of all these physical assets. What do you think?

Discouraging Incentives

http://www.nytimes.com/2007/07/02/opinion/02schwartz.html
Do you think more incentives are better? Not always argues Barry Schwartz.

Economists assumption: the more motives the better. Give people two reasons to do something and they will be more likely to do it. Unfortunately this assumption is false. There are circumstances in which adding an incentive competes with other motives and diminishes their impact.
When you pay people for doing things they like, they come to like these activities less and will no longer participate in them without a financial incentive. The intrinsic satisfaction of the activities gets “crowded out” by the extrinsic payoff.

Comment: Money is the most powerful motivator. However we should carefully offer additional incentives for something that people like doing. It could leave a bitter taste and kill passion, motivation and dedication even if they continue doing this. But not giving money when people expect them could be even worse. So, probably, it is a good idea to discuss / understand person’s internal motives and expectations before finding a way to encourage desirable behavior. Maybe just public and unreserved admiration will be the best reward. We, humans, are so complicated :)

Computers carry 70% of foreign currency trades

http://www.newscientisttech.com/channel/tech/mg19426061.500-gordon-gekko-makes-way-for-trading-software.html (need subscription)
Computers make one third of all trading decisions in US markets. Experts predict that more than 50% will be done by 2010. Machines can make multiple trades and monitor thousands of stocks at breakneck speed. They can use tactics that makes trades indistinct and hide their intent; for example, by spreading the deal over many small trades. There are big profits to be made before market realizes these opportunities. Companies are moving their servers as close as possible to stock exchange systems to reduce order time: milliseconds matter in competition of computers. There is a new arm race between trading “algos” (algorithms) – they now try to guess and sneak signs of other algos trading. David Cliff designed one of the first commercially successful algos in mid 90s. Human traders still have place on less understood and illiquid markets where instinct and experience are still important; software helps them with simulations and test of ideas. People are still much better in predicting market trends. How much longer?

Comment: First, people, who try competing with these machines for day trading, have less and less chances. Second, the most important that another intellectual human area is given up to computers along with chess and drug creation. What is next? Computers can became smarter than people in the next 20-30 years. They just need to learn how to write programs for themselves.

Human-Aided Computing

http://www.technologyreview.com/Infotech/18962/
Researchers at Microsoft are trying to tap into some of the specialized–and often subconscious–computing power in the human brain, and use it to solve problems that have so far been intractable for machines. Today it takes relatively large supercomputers many hours to recognize faces–something a human can do almost instantly. One application for this face-recognition technique could be to use it for quickly sorting snapshots from surveillance videos to find frames with faces and those without. This strategy could be useful for identifying other types of objects, such as dogs or cats, and different types of words. Subconscious brain power could therefore improve automated image search by preclassifying objects to help a computer more accurately identify pictures.

Comment: Now Microsoft evil guys :) teach computers how to use our brains. If it will go further computers will not only become smarter (see previous comment), but also hungry for our brain power.

Forgetting helps you remember the important stuff

http://news-service.stanford.edu/news/2007/june6/memory-060607.html
Bruce Kuhl and Anthony Wagner at Standford University have discovered that the brain’s ability to suppress irrelevant memories makes it easier for humans to remember what’s really important. This function, which is carried out in the prefrontal cortex region behind the forehead, helps the brain; it doesn’t have to work as hard in the future when it tries to remember an important memory because the competing but irrelevant memories have been weakened.
Memory allows humans to be predictive about what’s likely to be relevant to them as they go through life, Wagner explained. “What forgetting does is allow the act of prediction to occur much more automatically, because you’ve gotten rid of competing but irrelevant predictions,” he said. “That’s very beneficial for a neural information processing system.”

Comment: I knew this before – brain automatically forget irrelevant memories! You can read and learn as much as you want, but you’ll forget most of it. It doesn’t make sense for us to learn all the language constructs syntax, all the classes in framework and read so many books. We’ll forget if it is irrelevant. Now the most important question is how to make this information relevant. I believe the best way to do this – create abstractions, actively aggregate in your head conceptual pictures or just practically use it.

Mind Control: Unwanted Thoughts

http://www.psychologytoday.com/rss/pto-20040204-000001.html
Now if you want to forget something :)
Trying hard not to think about something almost guarantees that it will pop up in your consciousness. When you are actively avoiding a thought, one part of your brain is busily working to keep the upsetting thought at bay. It’s searching out distractors—something else to focus on that will protect you from the idea you’re trying to avoid.

At the same time, another part of the mental machinery has to keep checking to make sure that the job’s being done properly. Inadvertently, this monitoring process calls attention to the unwanted thought, and makes you more vulnerable to the very ideas you’re fleeing from. Article gives 2 advices: make secret preoccupation open or do the opposite of the thing you want to do.

Comment: These facts show again power of our automatic brain systems over our conscious mind and importance of cooperating with unconscious mind. You can try 2 additional things when you build program and you have unwanted thoughts:

• don’t program at all – quality of your work will be low
• or find most interesting problem in the program and you’ll quickly forget your thoughts

New approaches to software development

http://www.scottberkun.com/blog/2007/asshole-driven-development/
Scott Berkun provides information about the new (or reinvented) systems for software project management: Asshole Driven development (ADD) , Cognitive Dissonance development (CDD), Cover Your Ass Engineering (CYAE) and many others.

No comments.

“Somebody has to pay for all this.” – Kent Beck

Why do software projects exist?

A rational self-interest is a foundation of economics – maximizing expected benefits and minimizing expected cost. Software is one of the important instruments in hands of businesses, organizations and individuals that helps to achieve both goals. We could make money developing software and we could become more efficient and reduce cost. Benefits are not only limited to profit and money as with government, science or open source volunteers.

Economic forces start, drive and end software projects. These forces are complex, unpredictable and demanding. And they are the main reason for existence of the software.

What are economic forces?

Economic forces operate on 3 levels – internal, market and macro. You can find below description and examples of these forces.

Internal

• Vision, strategy and capabilities define what software should be built. Microsoft main business is Windows and Office, but Ray Ozzie replaced Bill Gates as chief software architect and are quietly changing the vision. He probably realizes that Microsoft strategy should change to stay competitive. Otherwise it will be dead.
• People expertise, skills and experience define what is possible to build. Google raise, power and innovations come from talented people. It recognizes it and became #1 company to work for in 2007. Google has best recruiting machine, great perks and developer centric culture. We all know results – now it is one of the most innovative and growing software companies in the world.
• Resources, time and finances define what could be afforded in reality. Smaller companies can only dream about Google (157B market value) and Microsoft (292B) resources and large-scale projects. However, software development is not capital intensive and it is possible to grow start-up with few people to become a big corporation. Google did it in 10 years, Facebook is doing it now.

Market

• Company market positioning, growth dynamic and perspectives define what it could sell. Operation systems is an interesting example – Windows XP has 82%. Who will win new race Microsoft Vista, Apple Leopard or Linux for better market positioning? Don’t forget about possible Google OS.
• Customers, market trends and composition define what is possible to sell. Traditional packaged software started to loose share to hosted services / SaaS (software-as-a-service) like Google and Salesforce. It is expected that 25% of software revenues by 2011 will be generated from SaaS from 8% in 2005. However, we are still too far from pure network computer (if we don’t consider mainframe terminals). The main share of today SaaS (50%) belongs to CRM and ERP hosted systems.
• Competitors, prices and alternative markets define what will be sold in reality. Software can attack other markets – consider VoIP vs traditional telephony. 20% of US business already use VoIP. On other front, open-source software could reach at least 22% market share in the next five years and disrupt traditional business models.

Macro

• Politics and regulations – taxes, laws and licensing. Microsoft claims Google bolsters piracy. Google is claiming that Microsoft desktop search violates anti-trust settlement. Microsoft claims open source software like Linux violates 235 of its patents. And so on.
• Economic – growth, interest rates, inflation, labor. Economy is the king and affects any market participant. Only 50% of Internet companies survived dot-com bubble burst.
• Society – norms, interests and preferences. More than 50% of teenagers are using Facebook and Myspace and it seriously impacts their lifestyle.
• International – globalization, cultures, business climate. Offshore IT services change landscape for software development – the forecast is $29B in worldwide customer spending.
• Technology – innovation, application, trends. Internet, mobile devices and other new technologies radically change software application, business models and lifecycle. Internet can be considered #1 technology innovation for changing software and world in general. Only creation of computers is more important event for software.

What are economic paradigms to create and deliver software?

• Retail - packaged software done by centralized effort of the single company (Microsoft, Adobe)
• In-House – own programmers (many brave companies who develop software themselves)
• Contract – contracted outside company (IBM, Accenture)
• Open Source – multiple participants (companies, individuals, organizations) develop product together (Linux, Firefox)
• Online Service - online hosted software available as a service over Internet (Google, Yahoo)

Are relation between business and software development happy?

Unfortunately, these relations are not happy in the many cases. These two worlds have contrary expectations from each other.

Software development is one of the most troublesome instruments in hands of business. Even excellent business decisions are worth nothing without good implementation. When project is started, two other groups of forces start to play – Human and System. And they add much more complexity and challenges to the rational nature of economic forces.

Question: Do you agree that economic forces are primary in the software life?

Software is everywhere: inside our computers, cars, phones and even toasters. Software tells these devices what to do.

Everybody can develop software. Hundreds of millions do. We use similar skills as in writing a cooking recipe or telling a friend how to find a shopping mall – we just need to come up with the set of instructions. Basic logic and knowledge of instruction language is enough. You don’t need to have a computer science degree or even finish courses to become a good programmer.

Does it sound simple? Creation of a program should be a routine job now as growing potatoes or building a bridge. And we have at least 2 good reasons to hope for this:

• We had more than 60 years to learn how to properly instruct computers what to do. The first computer ENIAC was built in 1946.
• Today 19 millions of professional programmers in the world and multi-billion industry are very serious about effectively building good software .

And still, software creation is unpredictable, unreliable and often fails.

Why? We should understand better what is software development.

Software development is the translation of a user need or marketing goal into a software product. – Wikipedia

We can add more to this definition.

1) Is it engineering?

The application of scientific and mathematical principles to practical ends – The American Heritage Dictionary

Software development is close to engineering. As in engineering we have

• Problem - user needs and requirements
• Solution - software system that solves these needs
• Principles - accepted rules for building solutions
• Constraints - limitations that our solution should meet
• Trade-offs – finding a compromise between different options
• Methods - how do we practically translate user needs into solution applying principles, meeting constraints and making trade-offs

IEEE defines software engineering as application of a systematic, disciplined, quantifiable approach to development, operation, and maintenance of software; that is, the application of engineering to software.

Many prominent people in software development don’t agree that engineering is the only approach to create software.

Pete McBreen: “The biggest problem with software engineering is the assumption that a systematic, disciplined, and quantifiable approach is the only possible approach. …software development is not a defined process. At best, all we can hope to achieve is an empirical process. The reason is that the sources of all software requirements are people. There is no way to automate requirements elicitation; rather, people must talk to each other so that the software team can learn what the users really need the application to do. At the same time, the users are learning the technical constraints and costs and then use that knowledge to adjust the feature set that they ask for.”

Martin Fowler: “Engineering methodologies have been around for a long time. They’ve not been noticeable for being terribly successful. They are even less noted for being popular. The most frequent criticism of these methodologies is that they are bureaucratic. There’s so much stuff to do to follow the methodology that the whole pace of development slows down.”

Software development is similar to engineering as it deals with complex problems, constraints and trade-offs. The difference is that engineering relies on proven laws of physical world, well defined principles and processes, while software development relies on uncertain mental concepts, shaky people needs and empirical principles and processes (guided by experience). Directly applying engineering approach to software development could introduce more problems than solve.

2) Is it business?

The social science of managing people to organize and maintain collective productivity toward accomplishing particular creative and productive goals, usually to generate profit. – Wikipedia

Software development became essential part of most businesses for generating profit and reducing cost. The relation is often unhappy. Only 35% of software projects started in 2006 could be considered as successful and this is an enormous improvement with 1994 figures – 16%. And still, there is a long way ahead to make software development economically successful and effective.

Kent Beck: “We need to make our software development economically more valuable by spending money more slowly, earning revenue more quickly, and increasing the probable productive lifespan of our project. But most of all we need to increase the options for business decisions.”

Business is chaotic and depends on unpredictable factors as economy, market trends and customer behavior. Business drives software projects, destabilizes them with frequent changes of directions and still expects predictable results. Finally, most of the software projects doesn’t make sense without business and economic demands.

3) Is it art and craft?

a. A system of principles and methods employed in the performance of a set of activities
b. Skill that is attained by study, practice, or observation
c. High quality of conception or execution, as found in works of beauty; aesthetic value
- The American Heritage Dictionary

Andrew Hunt and David Thomas [from Pragmatic Programmer]: “Programming is a craft. At its simplest, it comes down to getting a computer to do what you want it to do (or what your user wants it to do). As a programmer, you are part listener, part advisor, part interpreter, and part dictator. You try to capture elusive requirements and find a way of expressing them so that a mere machine can do them justice. You try to document your work so that others can understand it, and you try to engineer your work so that others can build on it. What’s more, you try to do all this against the relentless ticking of the project clock. You work small miracles every day.”

Pete McBreen [from Software Craftsmanship ]: “We must insist that developers really know their craft before we trust them to create systems for us or with us. Whether you are a user, manager, or developer, it doesn’t make sense to trust an inexperienced software developer. This is why we focus on software craftsmanship rather than on software engineering. The best processes in the world will not save a project from failure if the people involved do not have the necessary skills to execute the process; conversely, really good developers can make any process work.”

Donal Knuth: “We have seen that computer programming is an art, because it applies accumulated knowledge to the world, because it requires skill and ingenuity, and especially because it produces objects of beauty… A scientific approach is generally characterized by the words logical, systematic, impersonal, calm, rational, while an artistic approach is characterized by the words aesthetic, creative, humanitarian, anxious, irrational. It seems to me that both of these apparently contradictory approaches have great value with respect to computer programming.”

Successful software solutions for complex problems are difficult to achieve by standard scientific and engineering approaches alone. Complex problems require high level of mastery, creativity and intuition. And these traits are hard to learn, acquire and apply consistently.

4) Is it psychology?

The study of mental processes such as perception, cognition, emotion, personality, as well as environmental influences, such as social and cultural influences, and interpersonal relationships, in order to devise theories of human behavior – Wikipedia

Alistair Cockburn [from Characterizing people as non-linear, first-order components in software development ]: “In the title, I refer to people as “components”. That is how people are treated in the process / methodology design literature. The mistake in this approach is that “people” are highly variable and non-linear, with unique success and failure modes. Those factors are first-order, not negligible factors. Failure of process and methodology designers to account for them contributes to the sorts of unplanned project trajectories we so often see.”

Tom DeMarco and Timothy Lister [from Peopleware]: “The major problems of our work are not so much technological as sociological in nature.”

Gerald M. Weinberg [from Psychology of Computer Programming]: “Because of the complex nature of the programming task, the programmer’s personality – his individuality and identity – are far more important factors in his success than is usually recognized.”

Software is built by humans and in most cases used by humans. The way how individuals and teams of software professionals think, feel and interact has significant affect on creation of the software. And humans are difficult to predict, rationalize and quantify.

Software Creation Forces.

Now we can better understand what is software development, why it has problems and how it is different from established practices of growing potatoes and building bridges. Software teams are engaged in following activities to build software.

1. Trying to solve complex problems that are driven by unpredictable businesses and even more unpredictable users.
2. Translating these needs into software based on uncertain world of mental concepts and empirical principles. We don’t have luxury of firm physical laws, sound scientific principles and established engineering methods.
3. Relying on humans to do this translation. Humans have varying psychology, talents and creativity; they expose complex social behavior. These factors are playing important role for individual and team productivity and quality of work.

As a result, many strong and contrary forces affect software development. However, we can identify 3 main group of forces:

1. Economic forces – profit, efficiency and customer needs.
2. Human forces – psychology, productivity, motivation; talent, creativity, learning; group behavior, interactions and social complexity.
3. System forces – organization and evolution of the complex systems.

The goal of this blog is to understand and harness these forces to improve software creation – one of the most useful, intellectual and creative human activities.

Question for readers: Is software development more engineering, business, art, psychology or something else?