Detecting Potential Topics In News Using BERT, CRF and Wikipedia
Abstract
For a news content distribution platform like Dailyhunt, Named Entity Recognition is a pivotal task for building better user recommendation and notification algorithms. Apart from identifying names, locations, organisations from the news for 13+ Indian languages and use them in algorithms, we also need to identify n-grams which do not necessarily fit in the definition of Named-Entity, yet they are important. For example, "me too movement", "beef ban", "alwar mob lynching". In this exercise, given an English language text, we are trying to detect case-less n-grams which convey important information and can be used as topics and/or hashtags for a news. Model is built using Wikipedia titles data, private English news corpus and BERT-Multilingual pre-trained model, Bi-GRU and CRF architecture. It shows promising results when compared with industry best Flair, Spacy and Stanford-caseless-NER in terms of F1 and especially Recall.
- Publication:
-
arXiv e-prints
- Pub Date:
- February 2020
- DOI:
- 10.48550/arXiv.2002.11402
- arXiv:
- arXiv:2002.11402
- Bibcode:
- 2020arXiv200211402A
- Keywords:
-
- Computer Science - Computation and Language
- E-Print:
- 6 pages, 5 tables, 1 figure, 2 examples. This is a report based on applied research work conducted at Dailyhunt