The analysis of cryptographic APIs using the theorem prover Otter

Abstract

In 2000, Bond and Anderson exposed a new family of attacks on application programming interfaces (APIs) of security modules. These attacks elicit compromising behaviors using an unexpected sequence of legal calls to the module, uncovering severe security flaws even in widely-deployed cryptographic hardware. Because these attacks do not depend on the underlying ryptographic mechanisms, they often succeed even under the assumption of ideal cryptographic primitives. This thesis presents a methodology for the automatic detection of API attacks. Taking a cue from previous work on the formal analysis of security protocols and noting these attacks' independence from precise cryptographic mechanisms, we model APIs opaquely, purely according to specifications. We use a theorem prover tool and adapt it to the security API context. Several specifications of Cryptographic APIs are implemented for analysis using a theorem prover known as OTTER. These implementations successfully found known attacks, and provide evidence that OTTER will also be able to find new attacks, and perhaps eventually verify security in arbitrary Cryptographic APIs. Based on these implementations, various strategies, potential problems, and solutions are discussed that can be applied towards the formal analysis of Cryptographic APIs. We detail how, using these formalization and automation techniques, we have confirmed a number of known attacks and exposed an undocumented behavior of the IBM 4758 CCA, a hardware add-on crucial to a large portion of banking transactions worldwide. We show how the confirmed attacks' complexity and unintuitiveness make a very strong case for continued focus on automated formal verification of cryptographic APIs.