Term frequency with average term occurrences for textual information retrieval

Ibrahim, O. and Landa-Silva, Dario (2016) Term frequency with average term occurrences for textual information retrieval. Soft Computing, 20 (8). pp. 3045-3061. ISSN 1433-7479

PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Download (594kB) | Preview


In the context of Information Retrieval (IR) from text documents, the term-weighting scheme (TWS) is a key component of the matching mechanism when using the vector space model (VSM). In this paper we propose a new TWS that is based on computing the average term occurrences of terms in documents and it also uses a discriminative approach based on the document centroid vector to remove less significant weights from the documents. We call our approach Term Frequency With Average Term Occurrence (TF-ATO). An analysis of commonly used document collections shows that test collections are not fully judged as achieving that is expensive and may be infeasible for large collections. A document collection being fully judged means that every document in the collection acts as a relevant document to a specific query or a group of queries. The discriminative approach used in our proposed approach is a heuristic method for improving the IR effectiveness and performance, and it has the advantage of not requiring previous knowledge about relevance judgements. We compare the performance of the proposed TF-ATO to the well-known TF-IDF approach and show that using TF-ATO results in better effectiveness in both static and dynamic document collections. In addition, this paper investigates the impact that stop-words removal and our discriminative approach have on TFIDF and TF-ATO. The results show that both, stopwords removal and the discriminative approach, have a positive effect on both term-weighting schemes. More importantly, it is shown that using the proposed discriminative approach is beneficial for improving IR effectiveness and performance with no information in the relevance judgement for the collection.

Item Type: Article
Additional Information: The final publication is available at Springer via http://dx.doi.org/10.1007/s00500-015-1935-7
Keywords: Heuristic term-weighting scheme, Random term weights, Textual information retrieval, Discriminative approach, Stop-words removal
Schools/Departments: University of Nottingham, UK > Faculty of Science > School of Computer Science
Identification Number: https://doi.org/10.1007/s00500-015-1935-7
Depositing User: Landa-Silva, Dario
Date Deposited: 21 Jan 2016 10:00
Last Modified: 08 Dec 2016 19:45
URI: http://eprints.nottingham.ac.uk/id/eprint/31296

Actions (Archive Staff Only)

Edit View Edit View