Deduplication in a massive clinical note dataset
Abstract
Duplication, whether exact or partial, is a common issue in many datasets. In clinical notes data, duplication (and near duplication) can arise for many reasons, such as the pervasive use of templates, copy-pasting, or notes being generated by automated procedures. A key challenge in removing such near duplicates is the size of such datasets; our own dataset consists of more than 10 million notes. To detect and correct such duplicates requires algorithms that both accurate and highly scalable. We describe a solution based on Minhashing with Locality Sensitive Hashing. In this paper, we present the theory behind this method and present a database-inspired approach to make the method scalable. We also present a clustering technique using disjoint sets to produce dense clusters, which speeds up our algorithm.
- Publication:
-
arXiv e-prints
- Pub Date:
- April 2017
- DOI:
- 10.48550/arXiv.1704.05617
- arXiv:
- arXiv:1704.05617
- Bibcode:
- 2017arXiv170405617S
- Keywords:
-
- Computer Science - Databases;
- Computer Science - Information Retrieval
- E-Print:
- Extended from the Master project report of Sanjeev Shenoy, Department of Computer Science and Engineering, University of California, San Diego. June 2016