A new class of functions for describing logical structures in text
Name
57509348-MIT.pdf
Description
Full printable version
Size
2.85 MB
Format
Adobe PDF
Checksum (MD5)
123447fe905f1cd73df5cb4ecc159911
Author(s)
Dao, Ngon D. (Ngon Dong), 1974-
Advisor(s)
C. Forbes Dewey, Jr.
Date Issued
2004
Publisher
Massachusetts Institute of Technology
Abstract
Text documents generally contain two forms of structures, logical structures and physical structures. Loosely speaking, logical structures are sections of text that are both visually and semantically distinct. For example, a document may have an "introduction", a "body", and a "conclusion" as its logical structures. These structures are so named because each section has a distinct purpose in conveying the document's logical arguments or intentions. Perfect machine recognition of logical structures in large collections of documents is an unsolved problem in computational linguistics. This thesis presents evidence that a new family of functions on text segments carries information that is useful for differentiating document logical structures. For any given text segment, a function of this form is referred to as the cadence, and it is based on a new interpretation of the vector space representation that Gerard Salton introduced in 1975. Cadence also differs from the original Salton representation in that it relies on three heuristic transformations based on authorship, location, and term coherence. To test the hypothesis that the cadence of a text segment carries information helpful to differentiating logical structures, a corpus was built containing 2800 documents with manually-annotated logical structures. Structures representing abstracts, introductions, bodies, and conclusions from this corpus were clustered with a k-means algorithm using cadence data. Precision and recall performances were computed for the results, and a chi-squared cross-tabulation test was used to determine the statistical significance of the clustering results. Precision and recall were highest for abstracts (P = 0.931 [plus-minus] 0.025, R = 0.992
(cont.) [plus-minus] 0.026), followed by introductions (P = 0.747 [plus-minus] 0.025, R = 0.802 [plus-minus] 0.026) and conclusions (P = 0.737 [plus-minus] 0.025, R = 0.813 [plus-minus] 0.026), and lowest for bodies (P = 0.876 [plus-minus] 0.03, R = 0.663 [plus-minus] 0.026). These results suggest that cadence may have substantial promise for finding logical structures in un-annotated documents.
Description
Thesis (Ph. D.)--Harvard-MIT Division of Health Sciences and Technology, 2004.
Includes bibliographical references (p. 49-51).
Subjects
Harvard University--MIT Division of Health Sciences and Technology.
MIT Department
Harvard University--MIT Division of Health Sciences and Technology
Terms of Use
M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission.
Persistent DSpace Link
