Preface

This document is the final report for Task 1 of SRI Project 11459, Architectural Frameworks for Composable Survivability and Security, under DARPA Contract No. N66001-01-C-8040 as part of DARPA's Composable High-Assurance Trustworthy Systems (CHATS) program. Douglas Maughan was the DARPA Program Manager through the first two years of the project. He has been succeeded by Tim Gibson. 

Acknowledgments are given at the end of the body of this report. However, the author would like to give special mention to the significant contributions of Drew Dean and Virgil Gligor. 

This report contains no proprietary or sensitive information. Its contents may be freely disseminated. All product and company names mentioned in this report are trademarks of their respective holders.

Abstract

This report presents the results of our DARPA CHATS project. We characterize problems in and approaches to attaining computer system and network architectures. The overall goal is to be better able to develop and more rapidly configure highly trustworthy systems and networks able to satisfy critical requirements (including security, reliability, survivability, performance, and other vital characteristics). We consider ways to enable effective systems to be predictably composed out of interoperable subsystems, to provide the required trustworthiness -- with reasonably high assurance that the critical requirements will be met under the specified operational conditions, and (we hope) that do something sensible outside of that range of operational conditions. This work thus spans the entire set of goals of the DARPA CHATS program -- trustworthiness, composability, and assurance -- and much more.

By trustworthiness, we mean simply worthy of being trusted to fulfill whatever critical requirements may be needed for a particular component, subsystem, system, network, application, mission, enterprise, or other entity. Trustworthiness requirements might typically involve (for example) attributes of security, reliability, performance, and survivability under a wide range of potential adversities. Measures of trustworthiness are meaningful only to the extent that (a) the requirements are sufficiently complete and well defined, and (b) can be accurately evaluated.

This report should be particularly valuable to system developers who have the need and/or the desire to build systems and networks that are significantly better than today's conventional mass-market and custom software. The conclusions of the report can also be useful to government organizations that fund research and development efforts, and to procurers of systems that must be trustworthy.

Executive Summary

Anyone will renovate his science who will steadily look after the irregular phenomena. And when the science is renewed, its new formulas often have more of the voice of the exceptions in them than of what were supposed to be the rules. William James 

In this report, we confront an extremely difficult problem -- namely, how to attain demonstrably trustworthy systems and networks that must operate under stringent requirements for security, reliability, survivability, and other critical attributes, and that must be able to evolve gracefully and predictably over time -- despite changes in requirements, hardware, communications technologies, and radically new applications. In particular, we seek to establish a sound basis for the creation of trustworthy systems and networks that can be easily composed out of subsystems and components, with predictably high assurance, and also do something sensible when forced to operate predictably outside of the expected normal range of operational conditions. Toward this end, we examine a set of principles for achieving trustworthiness, consider constraints that might enhance composability, pursue architectures and trustworthy subsystems that are inherently likely to result in trustworthy systems and networks, define constraints on administrative practices that reduce operational risks, and seek approaches that can significantly increase assurance. The approach is intended to be theoretically sound as well as practical and realistic. We also outline directions for new research and development that could significantly improve the future for high-assurance trustworthy systems.

With respect to the future of trustworthy systems and networks, perhaps the most important recommendations involve the urgent establishment and use of soundly based, highly disciplined, and principle-driven architectures, as well as development practices that systematically encompass trustworthiness and assurance as integral parts of what must become coherent development processes and sound subsequent operational practices. Only then can we have any realistic assurances that our computer-communication infrastructures -- and indeed our critical national infrastructures -- will be able to behave as needed, in times of crisis as well as in normal operation. The challenges do not have easy turn-the-crank solutions. Addressing them requires considerable skills, understanding, experience, education, and enlightened management. Success can be greatly increased in many ways, including the availability of reliable hardware components, robust and resilient network architectures and systems, consistent use of good software engineering practices, careful attention to human-oriented interface design, well-conceived and sensibly used programming languages, compilers that are capable of enhancing the trustworthiness of source code, techniques for increasing interoperability among heterogeneous distributed systems and subsystems, methods and tools for analysis and assurance, design and development of systems that are inherently easier to administer and that provide better support for operational personnel, and many other factors. The absence or relative inadequacy with respect to each of these factors today represents a potential weak link in a process that is currently riddled with too many weak links. On the other hand, much greater emphasis on these factors can result in substantially greater trustworthiness, with predictable results. 

The approach taken here is strongly motivated by historical perspectives of promising research efforts and extensive development experience (both positive and negative) relating to the development of trustworthy systems. It is also motivated by the practical needs and limitations of commercial developments as well as some initial successes in inserting significantly greater discipline into the open-source world. It provides useful guidelines for disciplined system developments and future research.

This report cannot be everything for everyone, although it should have some appeal to a relatively broad range of readers. As a consequence of the inherent complexity associated with the challenges of developing and operating trustworthy systems and networks, we urge readers with experience in software development to read this report thoroughly, to see what resonates nicely with their experience. However, to the inexperienced developer or to the experienced developer who believes in seat-of-the-pants software creation, we offer a few words of caution. Many of the individual concepts should be well known to many of you. However, if you are looking for easy answers, you may be disappointed; indeed, each chapter should in turn convince you that there are no easy answers. On the other hand, if you are looking for some practical advice on how to develop systems that are substantially more trustworthy than what is commercially available today, you may find many encouraging directions to pursue.

Although there are some novel concepts in this report, our main thrust involves various approaches that can make better use of what we have learned over the past many years in the research community and that can be used to better advantage in production systems. Many of the lessons relating to serious trustworthiness can be drawn from past research and prototype development. However, those lessons have been largely ignored in commercial development communities, and perhaps have also been insufficiently observed by the developers of source-available software. There are many directions herein -- both new and old -- for fruitful research and development that can help to fill in the gaps.

We believe that observance of the approaches described here would greatly improve the present situation. The opportunities for this within the open-source community are considerable, although they are also applicable to closed-source proprietary systems (despite various caveats).

Roadmap of This Report

The outline of this report is as follows.

• Chapter 1 presents some of the terminology and foundations on which this report is based.
• Chapter 2 considers the roles of principles in the conceptualization, design, implementation, operation, and use of information systems and networks having critical requirements for security, reliability, and survivability. On one hand, some of the principles under discussion are well established in the literature and in certain educational curricula, and some are even intuitively appealing to experienced developers. On the other hand, very few of these principles are seriously observed in conventional commercial programming practice, and therefore are sorely missing in many system architectures, system implementations, programming languages, compilers, software engineering disciplines, and software development tools. Indeed, although these principles are sometimes knee-jerkingly pooh-poohed as being impractical, they are also potentially very valuable.
• Chapter 3 outlines some of the obstacles to achieving facile composability and interoperability, and considers approaches that can contribute to the development of significantly greater composability in systems with critical requirements. In some cases, compositions of seemingly compliant subsystems can actually compromise the ability of the resulting system to satisfy its requirements. Of considerable interest is the concept of combining subsystems in ways that actually increase the resulting trustworthiness, or at least do not diminish it. Also relevant are the concepts of software engineering discipline, programming-language constructs, execution compatibility and interoperability, and development tools.
• Chapter 4 considers characteristics of architectures that are likely to predictably satisfy the CHATS goals, based on the discussion of principles and the analysis of composability. Although various different architectures are needed for different classes of applications, they can share many common principles and attributes.
• Chapter 5 considers further characteristics of architectures for which the interfaces at various layers are relatively perspicuous -- that is, understandable because of the way in which they have been designed and implemented.
• Chapter 6 examines techniques for achieving the required trustworthiness with some significant measure of assurance. It stresses the importance of incorporating the principled use of assurance techniques throughout development and operation.
• Chapter 7 reexamines the collected wisdom of the earlier chapters in the light of experience, and seeks to provide practical guidelines for applying that wisdom to the system development process.
• Chapter 8 considers some potentially significant areas for future research and development.
• Chapter 9 provides a summary of the report, the needs for future research and prototype development, and conclusions.
• Appendix A summarizes the work on formally based static code analysis carried out by Hao Chen and David Wagner, under a project subcontract to the University of California at Berkeley. It also notes subsequent work carried out by Hao Chen.
• Appendix B provides some useful background for the material in Chapters 2, 3, and 4, and particularly Chapter 5, relating to system structural and correctness properties associated with modularity. Appendix B is based on material written by Virgil Gligor, in connection with joint work he did with Drew Dean and Peter Neumann for Lee Badger's Visibly Controllable Computing initiative.

1 The Foundations of This Report

We essay a difficult task; but there is no merit save in difficult tasks.
Ovid 

In the context of this report, the term "trustworthy" is used in a broad sense that is meaningful with respect to any given set of requirements, policies, properties, or other definitional entities. It represents the extent to which those requirements are likely to be satisfied, under specified conditions. That is, trustworthiness means worthy of being trusted to satisfy the given expectations. For example, typical requirements might relate to attributes of security, reliability, performance, and survivability under a wide range of potential adversities. Each of these attributes has expectations that are specific to each layer of abstraction (and differing from one layer to another) -- for example, with respect to hardware, operating systems, applications, systems, networks, and enterprise layers.

• Security has been defined in many different ways -- some of which are rather general, and some of which are meaningful only within their own specific contexts. In its strictest sense, security might be thought of as the absence of some set of vulnerabilities. In a more operational sense, it might be thought of as the absence of opportunities for misuse, despite certain vulnerabilities. However, in the realities of expected use, essentially nothing is perfectly secure -- particularly in the face of insider misuse. Consequently, security is more commonly thought of as the result of a collection of attempts to prevent bad things from happening and the integrated anticipation of being able to recover suitably when bad things do happen. Security requirements typically specify properties relating to system integrity, data integrity, data confidentiality, and the ability to withstand systemic denial-of-service attacks, along with other more detailed properties such as user authentication, system authentication, user access control, accountability, monitoring, real-time misuse detection, and appropriate responses to detected security violations -- among others. The list of detailed requirements may be very inclusive. Security is a comprehensive notion, relevant throughout the processes of system conceptualization, requirements formulation, design, development, use, and operation; it applies to systems in the large as well as subsystems applications, and entire enterprises. Measures of security often tend to be of questionable value -- for example, probabilistic measures of how secure something is, which may be meaningful in the small, but almost useless in the large. In any event, there is a need for useful metrics.
• Reliability requirements might include properties relating to the ability to tolerate hardware failures and software flaws, the characterization of acceptable degradation in the face of untolerated faults, probabilities of success, expected mean times between failures, and so on. Measures of reliability typically represent the extent to which flaws, failures, and errors [26, 309] can be avoided or tolerated.
• Performance requirements might include aggregate throughput measures, processing speeds, storage capacities, and guaranteed real-time response (for example).
• Survivability requirements typically address overall system or enterprise availability despite numerous adversities that could compromise the intended goals, and thus (for example) necessarily encompass relevant aspects of security, reliability, performance, and other critical requirements (e.g., [264]). The range of anticipated adversities may be extensive in extremely critical applications.
• Trustworthiness can then be thought of as the well-founded assessment of the extent to which a given system, network, or component will satisfy its specified requirements, and particularly those requirements that are critical to an enterprise, mission, system, network, or other entity. Trustworthiness is meaningful only with respect to those expectations.
• Assurance provides some sort of measure or indication of the likelihood that the desired trustworthiness is actually well founded, typically through a combination of well-specified requirements, structured architectures, formal and nonformal evaluations, and operational practices. A system can be said to be trustworthy only with respect to its stated requirements, with some level of assurance that it will behave as expected relative to those requirements. If the requirements are fundamentally incomplete, the concept of trustworthiness is similarly inherently incomplete.
• Risk is sometimes thought of as a mathematical product of aggregate measures of threats, vulnerabilities, and potential quantifiable losses or asset values. However, reducing risk to a single quantifiable number is almost always overly simplistic, because the space being considered is highly multidimensional. More intuitively, risks are events that in one sense or another are undesirable. Clearly, some risks are more important than others, and efforts should be made to ensure that the most serious risks are avoided wherever practicable.

Note that these concepts are sometimes interrelated. Achieving survivability in turn requires security, reliability, and some measures of guaranteed performance (among other requirements). Human safety typically does as well. Many of these properties are meaningful in different ways at various layers of abstraction. At the highest layers, they tend to be emergent properties of systems in the large, or indeed entire enterprises -- that is, they are meaningful only in terms of the entire system complex rather than as lower-layer properties.

The concept of trustworthiness is essentially indistinguishable from what is termed dependability [24, 25, 26, 202, 309], particularly within the IEEE and European communities. In its very early days, dependability was focused primarily on hardware faults, subsequently extended to software faults, and then generalized to a notion of faults that includes security threats. In that framework, dependability's generalized notions of fault prevention, fault tolerance, fault removal, and fault forecasting (the last of which has some of the elements of assurance) seem to encompass everything that trustworthiness does, albeit with occasionally different terminology. However, a recent paper, Basic Concepts and Taxonomy of Dependable and Secure Computing, by Avizienis, Laprie, Randell, Landwehr [26] (which is a gold mine for serious researchers) attempts to distinguish security as a specifically identifiable subset of dependability, rather than more generally treating it as one of various sets of application-relevant requirements subsumed under trustworthiness, as we do in this report. (Their new reformulation of security encompasses primarily confidentiality, integrity, and availability -- which in this report are only part of the necessary trustworthiness aspects that are required for security -- although it also alludes to other attributes of security. However, any differences between their paper and this report are largely academic -- we are each approaching the same basic problems.)

We make a careful distinction throughout this report between trust and trustworthiness. Trustworthiness implies that something is worthy of being trusted. Trust merely implies that you trust something whether it is trustworthy or not, perhaps because you have no alternative, or because you are naive, or perhaps because you do not even realize that trustworthiness is necessary, or because of some other reason. We generally eschew the terms trust and trusted unless we specifically mean trust rather than trustworthiness. (The slogan on an old T-shirt worn around the National Security Agency was "In trust we trust.")

A prophetic definition long ago due to Leslie Lamport can be paraphrased in the context of this report as follows: a distributed system is a system in which you have to trust components whose mere existence may be unknown to you. This is increasingly a problem on the World Wide Web, which is today's ultimate distributed system.

There are many R&D directions that we believe are important for the short- and long-term futures -- for the computer and network communities at large, for DARPA developers, and for system and network developers generally. (We outline some recommendations for future R&D in Chapter 9.) The basis of our project is the exploration and exploitation of a few of the potentially most timely and significant research and development directions.

• Principles. We revisit fundamental principles of trustworthy system development, cull out those likely to be most effective, explore their practical limitations, and seek a basis for principled architectures, principled development, and principled operation.
• Composability. We explore existing obstacles to achieving seamless composability and techniques for attaining practical composability in the future. Composability is meaningful at many layers of abstraction, for components, subsystems, networked systems, and networks of networks. It is also applicable to policies, protocols, specifications, formal representations, and proofs. Subsystem composability takes on a variety of forms, including sequential (with or without feedback, with or without recursion, etc.) and parallel execution.
• Reusability. One of the side benefits of the effort to ensure composability is the possibility of easier reusability of modules, subsystems, and other components, and particular high-assurance components -- in contexts other than their initially conceived applicability.
• Compatibility and interoperability. In this report, compatibility implies that different entities can coexist without adverse side effects, perhaps without even knowing of each other's existence. Interoperability further implies that different entities are actually able to work constructively with one another.
• Architecture. By the term architecture, we specifically encompass the structure of systems and networks at various layers of abstraction (in terms of composable components, subsystems, etc.), the design of the functional interfaces particularly at the visible layers of abstraction, and the interdependencies between, among, and within those components and layers of abstraction. Architectures can be defined with increasingly refined degrees of specificity -- from conceptual coarse-grain representations down to extremely explicit schematics that are sufficiently detailed to being transformed into precise design specifications from which implementations may be carried out relatively unambiguously. In effect, successive degrees of architectural specificity, design specifications, and implementations represent the iterative development process of refinement, instantiation, and necessary revision.
• Trustworthy principled composable architectures. We seek to establish principle-based composable distributed-system network-oriented open architectures inherently capable of fulfilling critical security, reliability, survivability, and performance requirements, while being readily evolvable over time to accommodate widely differing applications, different hardware and software providers, and changing technologies.
• Trustworthy system development and its foundations. In addition to principled architectures, we seek to provide a sound basis for system requirements, specifications, system development, implementation, trustworthiness, and assurance of that trustworthiness for composable interoperable components, with predictable behavior when composed.  
• Trustworthy protocols. We need to develop new protocols and/or extend existing protocols that effectively mask the peculiarities of various networking technologies wherever possible, but are able to accommodate a wide range of technologies (e.g., wireless and wired, optical, and electronic), and capable of addressing all relevant critical requirements. Existing protocols (e.g., IPv4, IPSEC, IPv6, secure routing, secure name service) are not adequate for critical-system and network trustworthiness. This is a very difficult challenge, and necessarily needs the involvement of the IETF, NIST standards efforts, and development communities.
• Principled operational practice. We need to bring the above concepts into the realm of operational practice, which is seriously in need of greater trustworthiness and controllability. Many of the concepts considered here have considerable potential toward that end, particularly for system and network management.

Throughout the history of efforts to develop trustworthy systems and networks, there is an unfortunate shortage of observable long-term progress relating specifically to the multitude of requirements for security. (See, for example, an interview with Richard Clarke [149] in the IEEE Security and Privacy.) Blame can be widely distributed among governments, industries, and users -- both personal and corporate. Significant research and development results are typically soon forgotten or else deprecated in practice. Systems have come and gone, programming languages have come and (sometimes) gone, and certain specific systemic vulnerabilities have come and gone. However, many generic classes of vulnerabilities seem to persist forever -- such as buffer overflows, race conditions, off-by-one errors, mismatched types, divide-by-zero crashes, and unchecked procedure-call arguments, to name just a few. Overall, it is primarily only the principles that have remained inviolable -- at least in principle -- despite their having been widely ignored in practice. It is time to change that unfortunate situation, and honor the principles.

There is an unfortunate shortage of fundamental books that provide useful background for the material discussed in this report. Two recent books by Matt Bishop,  Computer Security: Art and Science [47] and Introduction to Computer Security [48], are worthy of particular note -- the former for its rather comprehensive and somewhat rigorous computer-science-based treatment of security, and the latter for its less formal approach that should be more accessible to others who are not computer scientists. Chuck Pfleeger's  Security in Computing [303], Ross Anderson's  Security Engineering [14], and Morrie Gasser's  Building a Secure Computer System [127] are also worthy sources. A recent book by Ian Sommerville [359] provides extensive background on software engineering.

A paper [266] summarizing our conclusions as of early 2003 is part of the DISCEX3 proceedings, from the April 2003 DARPA Information Survivability Conference and Exposition.

2 Fundamental Principles of Trustworthiness

Synopsis

Enormous benefits can result from basing requirements, architectures, implementations, and operational practices on well-defined and well-understood generally accepted principles.

In this chapter, we itemize, review, and interpret various design and development principles that if properly observed can advance composability, trustworthiness, assurance, and other attributes of systems and networks, within the context of the CHATS effort. We consider the relative applicability of those principles, as well as some of the problems they may introduce.

2.1 Introduction

Everything should be made as simple as possible -- but no simpler.
Albert Einstein 

A fundamental hypothesis motivating this report is that achieving assurable trustworthiness requires much greater observance of certain underlying principles. We assert that careful attention to such principles can greatly facilitate the following efforts.

• Principled architectures. Establishment of composable open distributed-system network-oriented architectures capable of fulfilling critical security, reliability, survivability, and performance requirements, while being readily adaptable to widely differing applications, different hardware, heterogeneous software providers, and changing technologies. As noted in Chapter 1, architecture here specifically implies both the structure of systems and networks and the design of their functional interfaces and interconnections, at various layers of abstraction.
• Principled system development. Development of specifications, implementation, trustworthiness, and assurance of that trustworthiness for composable interoperable components, with predictable behavior when those components are composed. 
• Principled assurance. Attainment of assuredly trustworthy systems and networks, capable of addressing all relevant critical requirements, with new or extended protocols that mask the peculiarities of various networking technologies wherever advantageous.

The benefits of disciplined and principled system development cannot be overestimated, especially in the early stages of the development cycle. Principled design and software development can stave off many problems later on in implementation, maintenance, and operation. Huge potential cost savings can result from diligently observing relevant principles throughout the development cycle and maintaining discipline. But the primary concept involved is that of disciplined development; there are many methodologies that provide some kind of discipline, and all of those can be useful in some cases.

In concept, most of the principles discussed here are fairly well known and understood by system cognoscenti. However, their relevance is often not generally appreciated by people with little development or operational experience. Not wishing to preach to the choir, we do not dwell on elaborating the principles themselves, which have been extensively covered elsewhere (see Section 2.3). Instead, we concentrate on the importance and applicability of these principles in the development of systems with critical requirements -- and especially secure systems and networks. The clear implication is that disciplined understanding and observance of the most effective of these principles can have enormous benefits to developers and system administrators, and also can aid user communities. However, we also explore various potential conflicts within and among these principles, and emphasize that those conflicts must be thoroughly understood and respected. System development is intrinsically complicated in the face of critical requirements. For example, it is important to find ways to manage that complexity, rather than to mistakenly believe that intrinsic complexity is avoidable by pretending to practice "simplicity". 

2.2 Risks Resulting from Untrustworthiness

As noted above, trustworthiness is a concept that encompasses being worthy of trust with respect to whatever critical requirements are in effect, often relating to security, reliability, guarantees of real-time performance and resource availability, survivability in spite of a wide range of adversities, and so on. Trustworthiness depends on hardware, software, communications media, power supplies, physical environments, and ultimately people in many capacities -- requirements specifiers, designers, implementers, users, operators, maintenance personnel, administrators, and so on.

There are numerous examples of untrustworthy systems, networks, computer-related applications, and people. We indicate the extensive diversity of cases reported in the past with just a few tidbits relevant to each of various categories. See Computer-Related Risks [260] and the Illustrative Risks index [267] for numerous further examples and references involving many different types of system applications. (In the Illustrative Risks document, descriptors indicate relevance to loss of life, system survivability, various aspects of security, privacy, development problems, human interface confusions, and so on.) Some of these examples are revisited in Section 6.9, in considering how principled architectures and assurance-based risk reduction might have avoided the particular problems.

• Safety  
  • Aviation disasters, attributable to problems with airframes, avionics computer hardware and software, badly designed human interfaces, pilots, air-traffic control systems, air-traffic controllers, maintenance crews, airport security lapses, and so on: KAL 007 (flying on erroneous autopilot course), Air New Zealand crash into Mount Erebus (erroneous course data), Lauda Air (thrust reverser accidentally deployed in flight), Iranian Airbus shootdown (bad operational interfaces); Black Hawk helicopter problems, to name just a few.
  • Medical disasters, attributable to hardware flaws, malfunctions, software bugs, and confusing human interfaces: Therac 25 (nonatomic transition from high-intensity to low-intensity mode); database errors resulting in operation failures; electromagnetic interference (pacemakers, defibrillators).
• Reliability and availability  
  • Failures in defense systems, control systems, telecommunications systems, space, financial systems, etc.: Patriot missiles missing Scuds (excessive clock drift); Yorktown Aegis missile cruiser disabled (Windows NT crashed by divide by zero in an application); ARPANET collapse (1980); AT&T long-distance collapse (1990); first Shuttle launch (Columbia backup computer synchronization problem); massive electrical power outages, with large-scale propagation (e.g., 13-hour northeast power outage of 9 November 1965; lower New York state outage on 13 July 1977; blackout of 10 western U.S. states on 2 October 1984; western U.S. power blackouts on 2 July 1996; North American west-coast power outages on 10 Aug 1996, including eight states, Canada and Baja; massive northeast U.S. power-grid overload and blackout on 14 August 2003.
• Security 
  • Unintentional security flaws in delivered software, including many whose origins have been well understood for many years but that keep recurring
  • Intentionally installed trapdoors, Trojan horses, other malware
  • Insider and outsider exploitations involving loss of confidentiality, loss of integrity, denials of service, viruses, worms, spam, malware introduced by Web browsing, financial frauds and misuse
  • Operational problems, such as configuration errors, administrator oversights
  • Many other problems. Overall, the situation here is truly deplorable and the risks very diverse. The Illustrative Risks index [267] includes many cases of reported security problems.
• Survivability 
  • Survivability despite diverse adversities ultimately depends on system and network reliability, security, computational adequacy and bandwidth, database availability, and various other attributes. (For example, see [264].) Some of the problems noted above involve failures of system and network survivability, as a result of hardware and software malfunctions, exploitations of security vulnerabilities, accidents, malice, electromagnetic interference and other environmental events, etc.
• Privacy 
  • Privacy is often relegated to a second-order consideration. Privacy can in some cases be aided by appropriate technology, but many of the misues are the result of misuse by trusted insiders or are extrinsic -- involving indirect misuse external to computer systems. Identity theft is an increasingly pervasive example. (For example, see [11, 104, 342].)
• Maintainability 
  • Design flaws and software bugs are ubiquitous, either lingering or emerging in upgrades. Overrreliance on patch management is just one consequence. Inadequate architectural approaches to managing complexity exacerbate the problems (such as bloatware with inordinate interdependence on untrustworthy components), as does inattention to human interfaces -- particularly system and network administrators.

Many systems actually have critical requirements that span multiple areas such as security, reliability, safety, and survivability. Although the cases listed above generally result from a problem in primarily one of these areas, there are many cases in which a maliciously induced security problem could alternatively have resulted from an accidentally triggered reliability problem, or -- similarly -- where a reliability/availability failure could also have been triggered intentionally. (For example, see Chapter 4 of [260].)

One such application area with critical multidisciplinary requirements has become of particular interest since the 2000 November election, resulting from the emerging desire for completely electronic voting systems that ideally should have stringent requirements for system integrity, voter privacy, and accountability, and -- perhaps most important -- the impossibility of uncontrolled human intervention during elections. Some of today's major existing all-electronic systems permit unmonitored human intervention (to recover from election-day glitches and to "fix" problems -- including during the voting and vote-counting procedures!), with no meaningful accountability. Some systems even routinely undergo code changes after the software has been certified! Thus, we are confronted with all-electronic paperless voting systems that have no independent audit record of what has happened in the system internals, with no real assurance that your vote was correctly recorded and counted, with no alternative recount, no systemic way of determining the presence of internal errors and fraud, and no evidence in case of protests. The design specs and code are almost always proprietary, and the system has typically been certified against very weak voluntary standards that do not adequately detect fraud and internal accidents, with evaluations that are commissioned and paid for by the vendors. In contrast, gambling machines are regulated with extreme care (for example, by the Nevada Gaming Commission), and held to extremely high standards.

For a partial enumeration of recorded cases of voting-system irregularities over the past more than twenty years, see the online html version of [267], clicking on Election Problems, or see the corresponding section in the .pdf and .ps versions.

Section 5.2.2 reconsiders some of the above cases as well as others in which problems arose specifically because of problems involving the human interfaces.

2.3 Trustworthiness Principles

Willpower is always more efficient than mechanical enforcement, when it works. But there is always a size of system beyond which willpower will be inadequate.
Butler Lampson  

Developing and operating complex systems and networks with critical requirements demands a different kind of thinking from that used in routine programming. We begin here by considering various sets of principles, their applicability, and their limitations.

We first consider the historically significant Saltzer-Schroeder principles, followed by several other approaches.

2.3.1 Saltzer-Schroeder Security Principles, 1975

The ten basic security principles formulated by Saltzer and Schroeder [334] in 1975 are all still relevant today, in a wide range of circumstances. In essence, these principles are summarized with a CHATS-relevant paraphrased explanation, as follows:

* Economy of mechanism: Seek design simplicity (wherever and to whatever extent it is effective).
* Fail-safe defaults: Deny accesses unless explicitly authorized (rather than permitting accesses unless explicitly denied).
* Complete mediation: Check every access, without exception.
* Open design: Do not assume that design secrecy will enhance security.
* Separation of privileges: Use separate privileges or even multiparty authorization (e.g., two keys) to reduce misplaced trust.
* Least privilege: Allocate minimal (separate) privileges according to need-to-know, need-to-modify, need-to-delete, need-to-use, and so on. The existence of overly powerful mechanisms such as superuser is inherently dangerous.
* Least common mechanism: Minimize the amount of mechanism common to more than one user and depended on by all users. Avoid sharing of trusted multipurpose mechanisms, including executables and data -- in particular, minimizing the need for and use of overly powerful mechanisms such as superuser and FORTRAN common. As one example of the flaunting of this principle, exhaustion of shared resources provides a huge source of covert storage channels, whereas the natural sharing of real calendar-clock time provides a source of covert timing channels. 
* Psychological acceptability: Strive for ease of use and operation -- for example, with easily understandable and forgiving interfaces.
* Work factor: Make cost-to-protect commensurate with threats and expected risks.
* Recording of compromises: Provide nonbypassable tamperproof trails of evidence.

Remember that these are principles, not hard-and-fast rules. By no means should they be interpreted as ironclad, especially in light of some of their potential mutual contradictions that require development tradeoffs. (See Section 2.6.)

The Saltzer-Schroeder principles grew directly out of the Multics experience (e.g., [277]), discussed further at the end of this section. Each of these principles has taken on almost mythic proportions among the security elite, and to some extent buzzword cult status among many fringe parties. Therefore, perhaps it is not necessary to explain each principle in detail -- although there is considerable depth of discussion underlying each principle. Careful reading of the Saltzer-Schroeder paper [334] is recommended if it is not already a part of your library. Matt Bishop's security books [47, 48] are also useful in this regard, placing the principles in a more general context. In addition, Chapter 6 of Matt Curtin's book [89] on "developing trust" -- by which he might really hope to be "developing trustworthiness" -- provides some useful further discussion of these principles.

There are two fundamental caveats regarding these principles. First, each principle by itself may be useful in some cases and not in others. The second is that when taken in combinations, groups of principles are not necessarily all reinforcing; indeed, they may seem to be mutually in conflict. Consequently, any sensible development must consider appropriate use of each principle in the context of the overall effort. Examples of a principle being both good and bad -- as well as examples of interprinciple interference -- are scattered through the following discussion. Various caveats are considered in the penultimate section.

Table 1 examines the applicability of each of the Saltzer-Schroeder principles to the CHATS goals of composability, trustworthiness, and assurance (particularly with respect to security, reliability, and other survivability-relevant requirements).

In particular, complete mediation, separation of privileges, and allocation of least privilege are enormously helpful to composability and trustworthiness. Open design can contribute significantly to composability, when subjected to internal review and external criticism. However, there is considerable debate about the importance of open design with respect to trustworthiness, with some people still clinging tenaciously to the notion that security by obscurity is sensible -- despite risks of many flaws being so obvious as to be easily detected externally, even without reverse engineering. Indeed, the recent emergence of very good decompilers for C and Java, along with the likelihood of similar reverse engineering tools for other languages, both suggest that such attacks are becoming steadily more practical. Overall, the assumption of design secrecy and the supposed unavailability of source code is often not a deterrent, especially with ever-increasing skills among black-box system analysts. However, there are of course cases in which security by obscurity is unavoidable -- as in the hiding of private and secret cryptographic keys, even where the cryptographic algorithms and implementations are public.

Fundamental to trustworthiness is the extent to which systems and networks can avoid being compromised by malicious or accidental human behavior and by events such as hardware malfunctions and so-called acts of God. In [264], we consider compromise from outside, compromise from within, and compromise from below, with fairly intuitive meanings. These notions appear throughout this report.

There are other cases in theory where weak links can be avoided (e.g., zero-knowledge protocols that can establish a shared key without any part of the protocol requiring secrecy), although in practice they may be undermined by compromises from below (e.g., involving trusted and supposedly trustworthy insiders subverting the underlying operating systems) or from outside (e.g., involving penetrations of the operating systems and masquerading as legitimate users). For a fascinating collection of papers on vulnerabilities and ways to exploit weak links, see Ross Anderson's website:  
http://www.cl.cam.ac.uk/users/rja14/

From its beginning, the Multics development was strongly motivated by a set of principles -- some of which were originally stated by Ted Glaser and Peter Neumann in the first section of the very first edition of the Multics Programmers' Manual in 1965. (See http://multicians.org.) It was also driven by extremely disciplined development. For example, with almost no exceptions, no coding effort was begun until a written specification had been approved by the Multics advisory board; also with almost no exceptions, all of the code was written in a subset of PL/I just sufficient for the initial needs of Multics, for which the first compiler (early PL, or EPL) had been developed by Doug McIlroy and Bob Morris.

In addition to the Saltzer-Schroeder principles, further insights on principles and discipline relating to Multics can be found in a paper by Fernando Corbató, Saltzer, and Charlie Clingen [85] and in Corbató's Turing lecture [84].

2.3.2 Related Principles, 1969 and Later

Another view of principled system development was given by Neumann in 1969 [255], relating to what is often dismissed as merely "motherhood" -- but which in reality is both very profound and difficult to observe in practice. The motherhood principles under consideration in that paper (alternatively, you might consider them just as desirable system attributes) included automatedness, availability, convenience, debuggability, documentedness, efficiency, evolvability, flexibility, forgivingness, generality, maintainability, modularity, monitorability, portability, reliability, simplicity, and uniformity. Some of those attributes indirectly affect security and trustworthiness, whereas others affect the acceptability, utility, and future life of the systems in question. Considerable discussion in [255] was also devoted to (1) the risks of local optimization and the need for a more global awareness of less obvious downstream costs of development (e.g., writing code for bad -- or nonexistent -- specifications, and having to debug really bad code), operation, and maintenance (see Section 7.1 of this report); and (2) the benefits of higher-level implementation languages (which prior to Multics were rarely used for the development of operating systems [84, 85]).  

In later work and more recently in [264], Neumann considered some extensions of the Saltzer-Schroeder principles. Although most of those principles might seem more or less obvious, they are of course full of interpretations and hidden issues. We summarize an extended set of principles here, particularly as they might be interpreted in the CHATS context.

• Sound architecture. Recognizing that it is much better to avoid design errors early than to attempt to fix them later, the importance of architectures inherently capable of evolvable, maintainable, robust implementations is enormous -- even in an open-source environment. The value of a well-thought-out architecture is considerable in open-source systems. The value in closed-source proprietary systems could also be significant, if it were thought through early on, although architectural foresight is often impeded by legacy compatibility requirements that tend to lock system evolution into inflexible architectures. Good interface design is as fundamental to good architectures as is their structure. Both the architectural structure and the architectural interfaces (particularly the visible interfaces, but also some of the internal interfaces that must be interoperable) benefit from careful early specification.
• Minimization of what must be trustworthy. Appropriate trustworthiness should be situated where it is most needed, suitable to overall system requirements, rather than required uniformly across widely distributed components (with potentially many weak links) or totally centralized (with creation of a single weak link and forgetting other vulnerabilities). Trustworthiness is expensive to implement and to ensure, and as a consequence significant benefits can result from minimizing what has to be trustworthy. This principle can contribute notably to sound architectures. In combination with economy of mechanism, this suggests avoidance of bloatware and unfortunate dependence on less trustworthy components.
• Abstraction. The primitives at any given logical or physical layer should be relevant to the functions and properties of the objects at that layer, and should mask lower-layer detail where possible. Ideally, the specification of a given abstraction should be in terms of objects meaningful at that layer, rather than requiring lower-layer (e.g., machine dependent) concepts. Abstractions at one layer can be related to the abstractions at other layers in a variety of ways, thus simplifying the abstractions at each layer rather than collapsing different abstractions into a more complex single layer. Horizontal and vertical abstractions are considered in Section 3.3. Six types of abstraction are discussed in Section B.4.1.
• Encapsulation. Details that are relevant to a particular abstraction should be local to that abstraction and subsequently isolated within the implementation of that abstraction and the lower layers on which the implementation depends. One example of encapsulation involves information hiding -- for example, keeping internal state information hidden from the visible interfaces. Another example involves masking the idiosyncrasies of physical devices from higher-layer system interfaces. and of course from the user interfaces as well.
• Modularity. Modularity relates to the characteristic of system structures in which different entities (modules) can be relatively loosely coupled and combined to satisfy overall system requirements, whereby a module could be modified or replaced as long as the new version satisfies the given interface specification. In general, modularity is most effective when the modules reflect specific abstractions and provide encapsulation within each module. See Section B.2 for an extensive discussion of modularity.
• Layered and distributed protection. Protection (and generally defensive design for security, reliability, and so on) should be distributed to where it is most needed, and should reflect the semantics of the objects being protected. With respect to the reality of implementations that transit entities of different trustworthiness, layers of protection are vastly preferable to flat concepts such as single signon (that is, where only a single authentication is required). With respect to psychological acceptability, single signon has enormous appeal; however, it can leave enormous security vulnerabilities as a result of compromise from outside, from within, or from below, in both distributed and layered environments. Thus, with respect to the apparent user simplicity provided by single signon, psychological acceptability conflicts with other principles, such as complete mediation, separation of privileges, and least common privilege. (Of particular interest here are work in distributed system protection and digital certificates such as begun by SDSI/SPKI, and continuing through recent work on digital rights management -- e.g., [70, 71, 72, 151, 374] -- all which are relevant in Chapter 4.)
• Constrained dependency. Improperly guarded dependencies (see Section 3.4) on less trustworthy entities should be avoided. However, it is possible in some cases to surmount the relative untrustworthiness of mechanisms on which certain functionality depends -- as in the types of trustworthiness-enhancing mechanisms enumerated in Section 3.5. In essence, do not trust anything on which you must depend -- unless you are seriously satisfied with demonstrations of its trustworthiness.
• Object orientation. The OO paradigm bundles together abstraction, encapsulation, modularity of state information, inheritance (subclasses inheriting the attributes of their parent classes -- e.g., for functionality and for protection), and subtype polymorphism (subtype safety despite the possibility of application to objects of different types). This paradigm facilitates programming generality and software reusability, and if properly used can enhance software development. This is a contentious topic, in that most of the OO methodologies and languages are somewhat sloppy with respect to inheritance. (Jim Horning notes that the only OO language he knows that takes inheritance of specifications seriously was the DEC/ESL OWL/Trellis, which was a descendant of CLU.)
• Separation of policy and mechanism. Statements of policy should avoid inclusion of implementation-specific details. Furthermore, mechanisms should be policy-neutral where that is advantageous in achieving functional generality. However, this principle must never be used in the absence of understanding about the range of policies that might be usefully implemented. There is a temptation to avoid defining meaningful policies, deferring them until later in the development -- and then discovering that the desired policies cannot be realized with the given mechanisms. This is a characteristic chicken-and-egg problem with abstraction.
• Separation of duties. In relation to separation of privileges, separate classes of duties of users and computational entities should be identified, so that distinct system roles can be assigned accordingly. Distinct duties should be treated distinctly, as in system administrators, system programmers, and unprivileged users.
• Separation of roles. Concerning separation of privileges, the roles recognized by protection mechanisms should correspond in some readily understandable way to the various duties. For example, a single all-powerful superuser role is intrinsically in violation of separation of duties, separation of roles, separation of privilege, and separation of domains. The separation of would-be superuser functions into separate roles as in Trusted Xenix is a good example of desirable separation. Once again (as with single signon, noted above), there is a conflict between principles: the monolithic superuser mechanism provides economy of mechanism, but violates other principles. In practice, all-powerful mechanisms are sometimes unavoidable, and sometimes even desirable despite the negative consequences (particularly if confined to a secure subenvironment). However, they should be avoided wherever possible.
• Separation of domains. Concerning separation of privileges, domains should be able to enforce separate roles. For example, a single all-powerful superuser mechanism is inherently unwise, and is in conflict with the notion of separation of privileges. However, separation of privileges is difficult to implement if there is inadequate separation of domains. Separation of domains can help enforce separation of privilege, but can also provide functional separation as in the Multics ring structure, a kernelized operating system with a carefully designed kernel, or a capability-based architecture.  
• Sound authentication. Authentication is a pervasive problem. Nonbypassable authentication should be applicable to users, processes, procedures, and in general to any active entity or object. Authentication relates to evidence that the identity of an entity is genuine, that procedure arguments are legitimate, that types are properly matched when strong typing is to be invoked, and other similar aspects.
• Sound authorization and access control. Authorizations must be correctly and appropriately allocated, and nonsubvertible (although they are likely to assume that the identities of all entities and objects involved have been properly authenticated -- see sound authentication, above). Crude all-or-nothing authorizations are often riskful (particularly with respect to insider misuse and programming flaws). In applications for which user-group-world authorizations are inadequate, access-control lists and role-based authorizations may be preferable. Finer-grained access controls may be desirable in some cases, such as capability-based addressing and field-based database protection. However, knowing who has access to what at any given time should be relatively easy to determine.
• Administrative controllability. The facilities by which systems and networks are administered must be well designed, understandable, well documented, and sufficiently easy to use without inordinate risks.
• Comprehensive accountability. Well-designed and carefully implemented facilities are essential for comprehensive monitoring, auditing, interpretation, and automated response (as appropriate). Serious security and privacy issues must be addressed relating to the overall accountability processes and audit data.

Table 2 summarizes the utility of the extended-set principles with respect to the three goals of the CHATS program acronym, as in Table 1.

For example, authorization is of limited use without authentication,  whenever identity is important. Similarly, authentication may be of questionable use without authorization. In some cases, authorization requires fine-grained access controls. Least privilege requires some sort of separation of roles, duties, and domains. Separation of duties is difficult to achieve if there is no separation of roles. Separation of roles, duties, and domains each must rely on a supporting architecture.

The comprehensive accountability principle is particularly intricate, as it depends critically on many other principles being invoked. For example, accountability is inherently incomplete without authentication and authorization. In many cases, monitoring may be in conflict with privacy requirements and other social considerations [101], unless extremely stringent controls are enforceable. Separation of duties and least privilege are particularly important here. All accountability procedures are subject to security attacks, and are typically prone to covert channels as well. Furthermore, the procedures themselves must be carefully monitored. Who monitors the monitors? (Quis auditiet ipsos audites?)

2.3.3 Principles of Secure Design (NSA, 1993)

Also of interest here is the 1993 set of principles (or perhaps metaprinciples?) of secure design [56], which emerged from an NSA ISSO INFOSEC Systems Engineering study on rules of system composition. The study was presented not as a finished effort, but rather as something that needed to stand the test of practice. Although there is some overlap with the previously noted principles, the NSA principles are enumerated here as they were originally documented. Some of these principles are equivalent to "the system should satisfy certain security requirements" -- but they are nevertheless relevant. Others might sound like motherhood. Overall, they represent some collective wisdom -- even if they are fairly abstract and incompletely defined.

• The security engineering of a system must not be done independently from the total engineering of the system.
• A system without requirements cannot fail; it merely presents surprises [386].
• The system is for the users and not the system designers.
• A systems seldom fully satisfies all of its requirements.
• Many failures of a system to meet its overall requirements are often obvious. However, failures to meet security requirements are often not obvious.
• In an operational system, it is the users' mission and information that are at risk, not the developers' or evaluators' information. The accreditor accepts those risks when deciding to use a system operationally.
• It is only in the context of a system and a security policy that the "security characteristics" of a component can be defined and evaluated.
• Every component in a system must operate in an environment that is a subset of its specified environment; [in particular,] every component in a system must operate in a security environment that is a subset of its specified security environment. (A component should not be asked to respond to events for which it was not designed -- and evaluated.) [This is a gross oversimplification, particularly for systems relying on other components on the Internet.]
• Security is a system problem.
• Keep it simple to make it secure.
• There is no security in uncertainty. [This needs to be reconsidered. Random keys -- or at least keys that have the appearance of randomness -- are fundamental to cryptographic security. Furthermore, random sampling -- testing, inspections, etc. -- can be effective.]
• A system should be evaluatable and evaluated.
• Architectural analysis should not be treated lightly.
• A system is only as strong as its weakest link; the fortress walls of security should all be high enough. (Note that weak links are often not obvious.) [A system may be even weaker than its weakest link!]
• A component should protect itself from other components by adhering to the principle of mutual suspicion.
• A system should be manageable and managed.
• A system should be able to come up in a recognizably secure state.
• A system should recognize error conditions.
• Pay special attention to information flow.
• Secure systems should protect the confidentiality of user data.
• Secure systems should protect the integrity of user data.
• Secure systems should protect the reliability of user processes.

Considerable discussion of these metaprinciples is warranted. For example, "Every component in a system must operate in a security environment that is a subset of its specified environment" implies iteratively that maximum trust is required throughout design and implementation of the other components, which is a gross violation of our notion of minimization of what must be trustworthy. It would be preferable to require that each component check that the environment in which it executes is a subset of its specified environment -- which is closely related to Schroeder's notion of mutual suspicion [343], noted further down the list.

"A system is only as strong as its weakest link" is generally a meaningful statement. However, some weak links may be more devastating than others, so this statement is overly simplistic. In combination with least privilege, separation of domains, and some of the other principles noted previously, the effects of a particular weak link might be contained or controlled. But then, you might say, the weak link was not really a weak link. However, to a first approximation, as we noted above, weak links should be avoided where possible, and restricted in their effects otherwise, through sound architecture and sound implementation practice.

2.3.4 Generally Accepted Systems Security Principles (I²F, 1997)

The 1990 report of the National Research Council study group that produced Computers at Risk [83] included a recommendation that a serious effort be made to develop and promulgate a set of Generally Accepted Systems Security Principles (GASSP). That led to the creation of the International Information Security Foundation (I²SF). A draft of its GASSP document [279] is available online. A successor effort is under way, after a long pause.

The proposed GASSP consists of three layers of abstraction, nine Pervasive Principles (relating to confidentiality, integrity, and availability), a set of 14 Broad Functional Principles, and a set of Detailed Principles (yet to be developed, because the largely volunteer project ran out of steam, in what Jim Horning refers to as a last gassp!). The GASSP effort thus far actually represents a very worthy beginning, and one more approach for those interested in future efforts. The top two layers of the GASSP principle hierarchy are summarized here as follows.

Pervasive Principles
* PP-1. Accountability
* PP-2. Awareness
* PP-3. Ethics
* PP-4. Multidisciplinary
* PP-5. Proportionality
* PP-6. Integration
* PP-7. Timeliness
* PP-8. Assessment
* PP-9. Equity
Broad Functional Principles
* BFP-1. Information Security
* BFP-2. Education and Awareness
* BFP-3. Accountability
* BFP-4. Information Management
* BFP-5. Environmental Management
* BFP-6. Personnel Qualifications
* BFP-7. System Integrity
* BFP-8. Information Systems Life Cycle
* BFP-9. Access Control
* BFP-10. Operational Continuity and Contingency Planning
* BFP-11. Information Risk Management
* BFP-12. Network and Infrastructure Security
* BFP-13. Legal, Regulatory, and Contractual Requirements of Info Security
* BFP-14. Ethical Practices

The GASSP document gives a table showing the relationships between the 14 Broad Functional Principles and the 9 Pervasive Principles. That table is reproduced here as Table 3.

2.3.5 TCSEC, ITSEC, CTCPEC, and the Common Criteria (1985 to date)

Any enumeration of relevant principles must note the historical evolution of evaluation criteria over the past decades -- from the 1985 DoD Trusted Computer System Evaluation Criteria (TCSEC, a.k.a. The Orange Book [249]) and the ensuing Rainbow Books, to the 1990 Canadian Trusted Computer Product Evaluation Criteria (CTCPEC, [64]), and the 1991 Information Technology Security Evaluation Criteria (ITSEC, [116]). These efforts have resulted in an international effort to produce the Common Criteria framework (ISO 15408 [172]), which represents the current state of the art in that particular evolutionary process. (Applicability to multilevel security is also addressed within the Common Criteria framework, although it is much more deeply embedded in the higher-assurance levels of the TCSEC.)

2.3.6 Extreme Programming, 1999

A seemingly radical approach to software development is found in the Extreme Programming (XP)  movement [33]. (Its use of "XP" considerably predates Microsoft's.) Although XP appears to run counter to most conventional programming practices, it is indeed highly disciplined. XP might be thought of as very small chief programmer teams somewhat in the spirit of a Harlan Mills'  Clean-Room approach, although it has no traces of formalism and is termed a lightweight methodology. It involves considerable emphasis on disciplined planning throughout (documented user stories, scheduling of relatively frequent small releases, extensive iteration planning, and quickly fixing XP whenever necessary), designing and redesigning throughout (with simplicity as a driving force, the selection of a system metaphor, and continual iteration), coding and recoding as needed (paired programmers working closely together, continual close coordination with the customer, adherence to agreed-upon standards, only one programmer pair may integrate at one time, frequent integration, deferred optimization, and no overtime pay), and testing repeatedly throughout (code must pass unit tests before release, tests must be created for each bug found, acceptance tests are run often, and the results are published).

In essence, Extreme Programming seeks to have something running at the end of each period (e.g., each week) by deferring less important concepts until later. There is a stated desire to let business folks decide which features to implement, based on the experience with the ongoing development.

Questions of how to address architecture in the large seem not to be adequately addressed within Extreme Programming (although these questions are absolutely fundamental to the approach that we are taking in this report, but perhaps are considered extraneous to XP). The concept of deferring architectural design until later in the process may work well in small systems (where dynamic changes tend to be relatively local), but can seriously complicate development of highly complex systems. Perhaps if coupled with principled architectures recommended here, Extreme Programming could be effective for larger development efforts. See the Web site noted in [33] for considerable background on the XP movement, including a remarkably lucid Frequently Asked Questions document contrasting XP with several other approaches (UML, RUP, CMM, Scrum, and FDD -- although this is a little like comparing apples and oranges). Wikipedia also has a useful analysis of socalled agile or lightweight methodologies, with relevant references (http://en.wikipedia.org/wiki/Agile_software_development ).

2.3.7 Other Approaches to Principled Development

There are too many other design and development methodologies to enumerate here, ranging from very simple to quite elaborate. In some sense, it does not matter which methodology is adopted, as long as it provides some structure and discipline, and is relatively compatible with the abilities of the particular design and development team. For example, Dick Karpinski hands out a business card containing his favorite, Tom Gilb's Project Management Rules: (1) Manage critical goals by defining direct measures and specific targets; (2) Assure accuracy and quality with systematic project document inspections; (3) Control major risks by limiting the size of each testable delivery. These are nice goals, but depend on the skills and experience of the developers -- with only subjective evaluation criteria. Harlan Mills' "Clean-Room" technology has some elements of formalism that are of interest with respect to increasing assurance, although not specifically oriented toward security. In general, good development practice is a necessary prerequisite for trustworthy systems, as are means for evaluating that practice.

2.4 Design and Implementation Flaws, and Their Avoidance

Nothing is as simple as we hope it will be. Jim Horning  

Some characteristic sources of security flaws in system design and implementation are noted in [260], elaborating on earlier formulations and refinements (e.g., [5, 271]). There are various techniques for avoiding those flaws, including sound architectures, defensively oriented programming languages, defensively oriented compilers, better runtime environments, and generally better software engineering practice.

• Identification and authentication. The lack of nonspoofable identities and inter-subsystem authentication within user systems and network infrastructures presents some major obstacles to the robust networking of systems. In addition to permitting denial-of-service attacks and penetrations, it also makes it very difficult if not impossible for traceback abilities to identify the source of misuse -- assuming that the misuse can be detected. The pervasive use of fixed/reusable passwords (especially those that traverse networks unencrypted or are otherwise exposed) is also a high-risk problem. Elaborate schemes for managing these passwords (such as avoiding dictionary words) ignore many of the risks. An enormous improvement can be achieved by using one-time authenticators such as cryptographic tokens, and -- in certain constrained user environments -- biometrics, at least within supposedly trustworthy subsystems and subnetworks. The pervasive use of unauthenticated IP addresses that are easily spoofed is another area of risk. Remote sites and remote users are frequently not properly identified and authenticated. Meaningful authentication is a precursor to the avoidance or restriction of many kinds of misuse.
• Authorization. Our systems and networks suffer from a serious lack of context-sensitive authorization. Monolithic access controls tend to grant all-or-nothing or extremely coarse permissions. The development and consistent use of finer-grained authorization techniques would be very helpful in enforcing separation of privilege and least privilege. In the classified world, gross levels (e.g., Top Secret, Secret, Confidential, and Unclassified) are clearly too inclusive, which is why finer-grained compartments are invoked. An excellent article on the principle of least privilege is written by David A. Wheeler [380], with specific application to the FreeBSD jail(), Linux Security Modules (LSM), Security-Enhanced Linux (SELinux), and Wietse Venema's Postfix mail transfer agent. This article is mandatory reading. (See also Wheeler's free online book on secure programming for Linux and Unix [379].) 
• Initialization and allocation. Failures in the initialization of procedures, processes, and indeed stable system and network configuration management represent a large class of system flaws. Consistency checking on entry, determination of suitable availability of appropriate resources, and deletion of possible residues are examples of techniques that can provide improved initialization and allocation.
• Finalization. In many programming languages, the lack of graceful termination and complete deallocation is inadequately recognized as a source of flaws. For example, deletion of leftover residues from previous executions is often ignored or relegated to an initialization problem, rather than treated systematically on termination (perhaps on the grounds that it might be avoided altogether in some circumstances). In general, finalization should be symmetrically matched with initialization. Whatever is done in initialization may need to be explicitly undone or at least checked for consistent status at finalization. Programming languages that incorporate garbage collection (GC) attempt to do this implicitly, although not always perfectly. For example, note that Java's finalizers based on pointer unreachability are inherently imprecise. Various other GC-based languages have subtle finalization problems, as do non-GC-based programming languages. Also, note that WinXP zeroizes free pages in an idle loop, instead of waiting until reuse. Overall, the need for secure and robust finalization remains a research topic.
• Runtime validation. A large class of flaws results from inadequate runtime validation. Careful attention to techniques such as argument validation and bounds checks (especially to prevent insertion of Trojan horses such as executables added to arguments, causing buffer overflows), divide-by-zero checks, and strong typing of arguments can have enormous benefits. Brian Randell long ago suggested the benefits of moving checking closer to the operations being performed (whether in space, in time, or in layer of abstraction), to reduce the intervening infrastructure that must be trustworthy. This is also applicable to end-to-end checks and end-to-end security.
• Consistent naming. Aliases, pointers, links, caches, and dynamic changes without relinking, and various redundant representations all represent common sources of security vulnerabilities. Symmetric treatment of aliases, symbolic naming and dynamic linking, strong type checking, use of globally unique names, and recognition of stale caches and cache invalidation are examples of beneficial techniques.
• Encapsulation. Exposure of procedure and process internals may allow leakage of supposedly protected information or externally induced interference. Proper encapsulation requires a combination of system architecture, programming language design, software engineering, static checking, and dynamic checking.
• Synchronization consistency. Many vulnerabilities arise as a result of timing and sequencing, such as dependence on relative ordering, race conditions, synchronization, and deadlocks -- in both synchronous and asynchronous contexts. Note that many of these problems arise because of sharing of state information (particularly in real time or in sequential ordering) across abstractions that otherwise seem disjoint. Atomic transactions, multiphase commits, and hierarchical locking strategies noted in Section 3.4 are examples of constructive design techniques. A classical kind of vulnerability is time-of-check to time-of-use (TOCTTOU) flaws, which result from a lack of atomicity, to which inadequate encapsulation can also be a contributing factor.
• Adverse dependencies. Dependence on untrustworthy programs or subsystems is another considerable source of vulnerabilities. It can emerge as a result of flawed compilers and flawed runtime library programs, as well as program bugs -- including those resulting from improper program changes and upgrades, but also from Trojan horses.
• Other logic errors. There are also many common logic errors (such as off-by-one counting and omitted negations) that need to be avoided. Many of these arise in the design process, but some involve bad implementation.

Useful techniques for detecting some of these vulnerabilities include defensive programming language design, compiler checks, and formal methods analyzing consistency of programs with specifications. Of particular interest is the use of static checking. Such an approach may be formally based, as in the use of model checking by Hao Chen, Dave Wagner, and Drew Dean (in the MOPS system, developed in part under our CHATS project). (See Appendix A.) Alternatively, there are numerous approaches that do not use formal methods, ranging in sophistication from lint to LCLint (Evans) to Extended Static Checking (Nelson, Reino, et al., DEC/Compaq SRC). Note that ESC is completely formally based, including use of a theorem prover; indeed, it is a formal method that has some utility even in the absence of formal software specifications.

Jim Horning notes that even partial specifications increase the power of the latter two, and provide a relatively gentle way to incorporate additional formalism into development. Strong type checking and model checking tend to expose various flaws, some of which are likely to be consequential to security and reliability. For example, purify and similar tools are useful in catching memory leaks, array-bound violations, and related memory problems. These and other analytic techniques can be very helpful in improving design soundness and code quality -- as long as they are not relied on by themselves as silver bullets.

All of the principles have some bearing on avoiding these classes of vulnerabilities. Several of these concepts in combination -- notably modularity,  abstraction,  encapsulation, device independence where advantageous, information hiding, complete mediation, separation of policy and mechanism, separation of privilege, least privilege, and least common mechanism -- are relevant to the notion of virtual interfaces and virtual machines. The basic notion of virtualization is that it can mask many of the underlying details, and makes it possible to change the implementation without changing the interface. In this respect, several of these attributes are found in the object-oriented paradigm.

Several examples of virtual mechanisms and virtualized interfaces are worth noting. Virtual memory masks physical memory locations and paging. A virtual machine masks the representation of process state information and processor multiplexing. Virtualized input-output masks device multiplexing, device dependence, formatting, and timing. Virtual multiprocessing masks the scheduling of tasks within a collection of seemingly simultaneous processes. The Multics operating system [277] provides early illustrations of virtual memory and virtual secondary storage management (with demand paging hidden from the programs), virtualized input-output (with symbolic stream names and device independence where commonalities exist), and virtual multiprogramming (with scheduling typically hidden from the programming interfaces). The GLU environment [177] is an elegant illustration of virtual multiprocessing. GLU allows programs to be distributed dynamically among different processing resources without explicitly programmed processor allocation, based on precompiling of embedded guidance in the programs.

2.5 Roles of Assurance and Formalism

In principle, everything should be simple.
In reality, things are typically not so simple.
(Note: The SRI CSL Principal Scientist is evidently both a Principle Scientist and a Principled Scientist, as well as Principal Scientist. PGN) 

In general, the task of providing some meaningful assurances that a system is likely to do what is expected of it can be enhanced by any techniques that simplify or narrow the analysis -- for example, by increasing the discipline applied to system architecture, software design, specifications, code style, and configuration management. Most of the cited principles tend to do exactly that -- if they are applied wisely. Techniques for increasing assurance are considered in greater detail in Chapter 6, including the potential roles of formal methods.

2.6 Caveats on Applying the Principles

For every complex problem, there is a simple solution. And it's always wrong.
H.L. Mencken 

As we noted above, the principles referred to here may be in conflict with one another if each is applied independently; in certain cases, the principles are not composable. In general, each principle must be applied in the context of the overall development. Ideally, greater effort might be useful to reformulate the principles to make them more readily composable, or at least to make their potential tradeoffs or incompatibilities more explicit.

There are also various potentially harmful considerations that must be considered -- for example, overuse, underuse, or misapplication of these principles, and certain limitations inherent in the principles themselves. Merely paying lipservice to a principle is clearly a bad idea; principles must be sensibly applied to the extent that they are appropriate to the given purpose. Similarly, all of the criteria-based methodologies have many systemic limitations (e.g., [257, 372]); for example, formulaic application of evaluation criteria is always subject to incompleteness and misinterpretation of requirements, oversimplification in analysis, and sloppy evaluations. However, when carefully applied, such methodologies can be useful and add discipline to the development process. Thus, we stress here the importance of fully understanding the given requirements and of creating an overall architecture that is appropriate for realizing those requirements, before trying to conduct any assessments of compliance with principles or criteria. And then, the assessments must be taken for what they are worth -- just one piece of the puzzle -- rather than overendowed as definitive results out of context. Overall, there is absolutely no substitute for human intelligence, experience, and foresight.

The Saltzer-Schroeder principle of keeping things simple is one of the most popular and commonly cited. However, it can be extremely misleading when espoused (as it commonly is) in reference to systems with critical requirements for security, reliability, survivability, real-time performance, and high assurance -- especially when all of these requirements are necessary within the same system environment. Simplicity is a very important concept in principle (in the small), but complexity is often unavoidable in practice (in the large). For example, serious attempts to achieve fault-tolerant behavior often result in roughly doubling the size of the overall subsystem or even the entire system. As a result, the principle of simplicity should really be one of managing complexity rather than trying to eliminate it, particularly where complexity is in fact inherent in the combination of requirements. Keeping things simple is indeed a conceptually wonderful principle, but often not achievable in reality. Nevertheless, unnecessary complexity should of course be avoided. The back-side of the Einstein quote at the beginning of Section 2.1 is indeed both profound and relevant, yet often overlooked in the overzealous quest for perceived simplicity. 

An extremely effective approach to dealing with intrinsic complexity is through a combination of the principles discussed here, particularly abstraction,  modularity,  encapsulation, and careful hierarchical separation that architecturally does not result in serious performance penalties, well conceived virtualized interfaces that greatly facilitate implementation evolution without requiring changes to the interfaces or that enable design evolution with minimal disruption, and far-sighted optimization. In particular, hierarchical abstraction can result in relative simplicity at the interfaces of each abstraction and each layer, in relative simplicity of the interconnections, and perhaps even relative simplicity in the implementation of each module. By keeping the components and their interconnections conceptually simple, it is possible to achieve conceptual simplicity of the overall system or networks of systems despite inherent complexity. Furthermore, simplicity can sometimes be achieved through design generality, recognizing that several seemingly different problems can be solved symmetrically at the same time, rather than creating different (and perhaps incompatible) solutions. Such approaches are considered further in Chapter 4.

Note that such solutions might appear to be a violation of the principle of least common mechanism, but not when the common mechanism is fundamental -- as in the use of a single uniform naming convention or the use of a uniform addressing mode that transcends different subtypes of typed objects. In general, it is riskful to have multiple procedures managing the same data structure for the same purposes. However, it can be very beneficial to separate reading from writing -- as in the case of one process that updates and another process that uses the data. It can also be beneficial to reuse the same code on different data structures, although strong typing is then important.

Of considerable interest here is David Musser's notion of Generic Programming, or programming with concepts. His Web site defines a concept as "a family of abstractions that are all related by a common set of requirements. A large part of the activity of generic programming, particularly in the design of generic software components, consists of concept development -- identifying sets of requirements that are general enough to be met by a large family of abstractions but still restrictive enough that programs can be written that work efficiently with all members of the family. The importance of the C++ Standard Template Library, STL, lies more in its concepts than in the actual code or the details of its interfaces." (http://www.cs.rpi.edu/ musser/gp/)

One of our primary goals in this project is to make system interfaces conceptually simple while masking complexity so that the complexities of the design process and the implementation itself can be hidden by the interfaces. This may in fact increase the complexity of the des