Survey of case studies of the use of knowledge management in software engineering

September 24, 2002 14:26 WSPC/117-ijseke 00096 International Journal of Software Engineering and Knowledge Engineering Vol. 12, No. 4 (2002) 391{414 c© World Scientic Publishing Company A SURVEY OF CASE STUDIES OF THE USE OF KNOWLEDGE MANAGEMENT IN SOFTWARE ENGINEERING TORGEIR DINGSYR and REIDAR CONRADIy *Sintef Telecom and Informatics, NO-7465 Trondheim, Norway Torgeir.Dingsoyr@sintef.no yDepartment of Computer and Information Science, Norwegian University of Science and Technology, NO-7491 Trondheim, Norway Reidar.Conradi@idi.ntnu.no Submitted 1 September 2001 Revised 25 May 2002 Accepted 20 June 2002 This article examines the literature on case studies of knowledge management systems in use in organisations that develop software. We investigate knowledge management approaches in eight case studies, and what the reported benets are. Surprisingly, very few organisations claim to have lowered software production costs or increased the qual- ity of the software. But many claim to have improved the work situation for software developers and managers. Keywords: Knowledge management; software engineering; learning software organisa- tions; experience factory. 1. Introduction This article is a survey of case studies of knowledge management systems in use in companies that develop computer software. We nd many descriptions of such knowledge management systems in the research literature, but most of them deal with technical issues, and few are dealing with how these systems actually work in the organisations where they are deployed. This is an attempt to systematically present published case studies of knowledge management systems that can be found in the research literature, and to analyse (1) What systems are in use, and (2) What is the impact of such systems on work in an organisation? We have written this article for people who are either skilled in knowledge management, and are eager to know how this is interpreted and used in software engineering, or for people in the software engineering eld, who are interested in knowing more about what knowledge management has had to oer them. This article is partially based on [1]. Corresponding author. 391 September 24, 2002 14:26 WSPC/117-ijseke 00096 392 T. Dingsyr & R. Conradi First, we will briefly motivate the use of knowledge management systems in software development by discussing the use of software, common problems in de- velopment and suggested improvement actions. We then go on to dene what we mean by \knowledge" and \knowledge management", before we state more precise research questions for this survey. Next, we present dierent technology innovations for knowledge management in software engineering as context, and then present and discuss eight case studies found in the literature. 1.1. Software development; problems and remedies To develop and maintain software is often referred to as \software engineering". One denition is that software engineering \is concerned with theories, methods and tools which are needed to develop software . . . for computers", and it diers from engineering in other disciplines because it is \not constrained by materials governed by physical laws or by manufacturing processes" [2] (quoted in [3]). 1.1.1. Problems with software development Software development can often be challenging. There are many examples of soft- ware projects that have failed. The much-cited Standish report on software projects [4] \shows a staggering 31.1% of projects will be cancelled before they ever get com- pleted. Further results indicate 52.7% of projects will cost 189% of their original estimates. The cost of these failures and overruns are just the tip of the proverbial iceberg. The lost opportunity costs are not measurable, but could easily be in the trillions of dollars . . . ". The view that the software systems we use today are not very mature is also supported by the American \President’s Information Technol- ogy Advisory Committee", that writes: \The Nations needs robust systems, but the software our systems depend on is often fragile. Software fragility is its tendency not to work properly | or at all. Fragility is manifested as unreliability, lack of security, performance lapses, errors and diculty in upgrading" [5]. So why does there seem to be so many problems related to software develop- ment projects? Software is an immaterial product, and it can be dicult to get an overview of a total program system, which can be millions of lines of code, to identify all possible error sources. Also, a very small defect might have a lot of influence in safety-critical systems, like the European Space Agency’s Ariane 5 satellite launcher, that ended in failure in 1996. About 40 seconds after initiation, the launcher \veered o its flight path, broke up and exploded" according to the re- port by the inquiry board [6]. The error was \caused by an internal variable related to the horizontal velocity of the launcher exceeding a limit which existed in the software". Thus, just a few lines of code that was lacking, had severe consequences | a loss of around 500 million pounds. Other problems can be that the communication between the end-users and the software developers is lacking, or that project management is dicult in an environment where a small bug can take a very long time to correct, and where it is often dicult to determine how much work is left to do on a software module. September 24, 2002 14:26 WSPC/117-ijseke 00096 Use of Knowledge Management in Software Engineering 393 Numerous examples of problems in software development projects can be found in popular books like Crash | Learning from the World’s worst Computer Disasters [7] and Software Runaways [8]. After listing all these problems that exist in software, you may ask: are all software systems that bad? That is not so, there are a lot of software projects which deliver software that is highly usable and working. Robert Glass has argued that the software failures are the exception rather than the trend [9] | \we tend to focus on the unusual things that go wrong because they’re more interesting or important than the run-of-the-mill things that go right". We should not use a word like \crisis" to describe the software development eld when we know of so many well-working systems. The main reason for this argument is that problems in software is used to motivate a lot of research; which should be able to stand on its own feet. We acknowledge that there have been more writings about the failures than the successes in software engineering projects, and that the situation might not be as bad as it looks. But as the reports we have cited earlier shows, there are at least quite a lot of projects that could improve, although it is not right to use a word like \crisis". 1.1.2. Suggested remedies There has been a lot of discussion in the software engineering community about nding a \silver bullet" to end the problems, or at least reduce the impact of them. Several solutions have been tried to improve the way software is developed, like changes in the way software is produced, the \process", introduction of new pro- gramming languages, and supporting tools to assist in development. The goal is usually to increase productivity and quality of the developed software. The out- come of several of these improvement initiatives was summed up in an article in Communications of the ACM [10]. Claims of \order of magnitude" improvements were evaluated, on dierent \technologies", such as:  Structured techniques | using structured analysis, design and programming.  Fourth generation programming languages (4GL).  Computer Aided Software Engineering | tools to support software engineering, mainly in analysis and design.  Formal methods | formal specication and verication of software.  Cleanroom methodologies | a method for removing defects from software.  Process models | descriptions of appropriate processes in software engineering.  Object-oriented technology | to nd \objects" in the problem to be solved, and use those in generating software solutions. Many of the technologies show promising results, but there are relatively few scien- tic articles that evaluate how the dierent methods work. Also, in some studies that claim improvement, the improvement technology is confused with other changes, September 24, 2002 14:26 WSPC/117-ijseke 00096 394 T. Dingsyr & R. Conradi like changes in the programming language. So there is still a need for more research on how these technologies really work. 1.2. What is knowledge management? Recently, much focus has been placed on \managing knowledge" better in what we can call knowledge-intensive companies. This has been applied in many other domains than software development, but we focus mainly on what has been achieved there, although we draw on general knowledge management theory to discuss what has happened in this domain. But rst, we discuss what we mean by \knowledge" before going on to discuss \knowledge management". 1.2.1. What is knowledge? The term \knowledge" is dened in the Oxford Dictionary and Thesaurus [11] as: \awareness or familiarity gained by experience (of a person, fact, or thing)", \persons range of information", \specic information; facts or intelligence about something", or \a theoretical or practical understanding of a subject". A more philosophical (and positivist) view of knowledge is to see it as \justied true belief". We often divide knowledge into two types, tacit and explicit knowledge [12]. By tacit knowledge we mean knowledge that a human is not able to express explicitly, but is guiding the behaviour of the human. For example, how to ride a bike is something that is dicult to express, which you have to learn by trial and error. Another example of tacit knowledge is the struggle of Japanese engineers to make a machine that bakes bread. According to Nonaka and Takeuchi [13], there were several trials to construct such a machine, but the bread simply did not taste as good as bread made by human bakers. The company NEC then decided to send people to a local baker to see how the process of making bread was carried out. The researchers returned with new insight on the kneading process, and later were able to replicate this in their machine. This is an example of tacit knowledge that is dicult to transfer by other means than looking at someone who are actually baking bread. Explicit knowledge is knowledge that we can represent, for example, in reports, books, talks, or other formal or informal communication. So when we later talk about computer systems for knowledge management, it is only the explicit knowl- edge that can be managed in these kinds of systems; the tacit knowledge remains in the people! Some claim that tacit knowledge can be converted to explicit through externalisation [13], and from explicit to tacit through internalisation. We also nd conversions from tacit to tacit | socialisation, and explicit to explicit | combination. Some terms related to knowledge are experience and information. In normal English, experience means \actual observation of or practical acquaintance with facts or events", or \knowledge or skill resulting from this" [11]. Most people see experience as a type of knowledge that you have gained from practise. Information is seen as \something told; knowledge", \items of knowledge; news". In normal September 24, 2002 14:26 WSPC/117-ijseke 00096 Use of Knowledge Management in Software Engineering 395 English, it is dicult to distinguish the terms information and knowledge. Within articial intelligence, information is often referred to as \data with meaning". The characters \4m" does not say much in itself, but if we know that \m" stands for \meters", it can be useful information. Knowledge is then often dened as information that is used (in an articial intelligence-sense: in a computer system). For an interesting discussion about the terms data, information and knowledge in articial intelligence, see [14]. This use of the term knowledge in articial intelligence is however greatly dis- puted by Dreyfus [15], who claims that knowledge requires other processes than those in a computer system. To sum up this discussion, it is clearly out of scope to land the discussion on knowledge in this article, but we will use a pragmatic denition of knowledge, what Taylor [16] who has been working with \information use environments" would call \instrumental information" | information that is used so that individuals know how to do something, or \factual information" | information that is used to determine facts. We will refer to this type of \operational information" as explicit knowledge, and we will also use the term tacit knowledge. 1.2.2. What is knowledge management? There are many interpretations of what knowledge management is, and many terms that describe computer systems to support managing knowledge in companies. In 1974, the book The Corporate Memory was published [17], arguing on the ben- et of collecting information from dierent sources in a company and making it \searchable". At this time, the information was gathered on paper, and \search" would mean to submit a form to a department who would manually search through their les. The term corporate memory is still in use, but now meaning a comput- erised database for storing documents from many people in a company. The term \corporate brain" is also used to describe such a database. Another related term is \organisational memory", which does not really have a clear denition, but \intu- itively, organisations should be able to retrieve traces of their past activities, but the form of this memory is unclear in research literature. Early eorts assume one could consider memory as though it were a single, monolithic repository of some sort for the entire organisation" [18]. Many see this term as meaning both a process of collecting and using information as well as a repository. In Software Engineering, to reuse life cycle experience, processes and products for software development is often referred to as having an \Experience Factory" [19]. In this framework, experience is collected from software development projects, and is packaged and stored in an experience base. By packing, we mean generalising, tailoring and formalising experience so that it is easy to reuse. This will be further elaborated in the next subsection. So what do we mean by knowledge management? We think that this term in- cludes issues from all the terms discussed. Some goals of knowledge management can September 24, 2002 14:26 WSPC/117-ijseke 00096 396 T. Dingsyr & R. Conradi be [20]: \To make the enterprise act as intelligently as possible to secure its viability and overall success". Thomas Davenport has dened it as \a method that simpli- es the process of sharing, distributing, creating, capturing and understanding of a company’s knowledge" [21]. If we look a bit more into knowledge management, we nd that some important aspects are [22]:  Survey, develop, maintain and secure the intellectual and knowledge resources of the enterprise.  Determine the knowledge and expertise required to perform work tasks, organise it, make the requisite knowledge available, \package it", and distribute it to the relevant points of action.  Provide (. . . ) knowledge architecture so that the enterprise’s facilities, proce- dures, guidelines, standards, examples, and practices facilitate and support active Knowledge Management as part of the organisation’s practices and culture. This seems to be pretty in line with what people from two software companies see as knowledge management. We interviewed 13 managers and developers about what they meant by \knowledge management" and got answers like \manage, plan, deploy, collect and spread knowledge in an organisation, and do it in a planned manner", and \to create, store, survey, use and revise knowledge". We can divide between two dierent usages, or strategies for knowledge man- agement [23]:  Codication | to systematise and store information that represents the knowl- edge of the company, and make this available for the people in the company.  Personalisation | to support the flow of information in a company by storing information about knowledge sources, like a \yellow pages" of who knows what in a company. We should add here that the codication strategy does not t all types of knowledge. In situations where knowledge is very context-dependent, and where the context is dicult to transfer, it can be directly dangerous to reuse knowledge without analysing it critically. For some more examples of problems with this strategy, see [24]. Another strategy apart from the two mentioned above could be to support the growth of knowledge | the creation of new knowledge by arranging for innovation through special learning environments or expert networks, but it is beyond the scope of this article. When we go on to discuss computer systems that support knowledge management, we will restrict the scope to systems supporting the rst two strategies. The Experience Factory One way to manage knowledge is by giving the responsibility for capturing and reusing experience to a separate part of the development organisation. This is the September 24, 2002 14:26 WSPC/117-ijseke 00096 Use of Knowledge Management in Software Engineering 397 Software Development Project Sp on so rin g O rg an isa tio n St ra te gi c Im pr ov em en t M an ag em en t Project Support Experience Base Ex pe rie nc e Pa ck ag e En gi ne er in g Experience Factory Fig. 1. The Experience Factory (taken from [25]). idea behind the \Experience Factory"; a technical and social knowledge manage- ment infrastructure to reuse life cycle experience, processes and products, which has been very much referred to in the software engineering eld [19]. Experience is collected from software development projects, and are packaged and stored in an experience base. The Experience Factory is a part of the quality improvement paradigm [26], which is inspired by work in Total Quality Management. It involves a feedback loop for improvement initiatives which include: (1) characterise the environment, (2) set goals, (3) choose process, (4) execute, (5) analyse, and (6) package. So, what we learn from these improvement cycles should be made available for the organisation. Examples of experience packages are:  Product Packages | information about the life cycle of a product, information on how to reuse it and lessons learned from reuse.  Process Packages | information on how to execute a life cycle process, and how to reuse it.  Relationship Packages | used for analysis and forecasts. Can be cost and defect models, resource models.  Tool Packages | instructions for use of a tool and experience with it.  Management Packages | reference information for project managers.  Data Packages | data relevant for a software project or its activities. Can be project databases or quality records. The Experience Factory organisation will then help new software develop projects with earlier experience, and can also suggest improvements in processes based on collected experience (we call this \strategic improvement management" in Fig. 1). September 24, 2002 14:26 WSPC/117-ijseke 00096 398 T. Dingsyr & R. Conradi Strategy Goals and a way to achieve them Processes Methods to manage tacit and explicit knowledge Tools
Infrastructure for explicit knowledge + + Fig. 2. A model of the components of a knowledge management \System" or \Program". The interaction between the Experienc