- Text mining
Text mining, sometimes alternately referred to as "text
data mining", roughly equivalent to " text analytics", refers generally to the process of deriving high quality informationfrom text. High quality information is typically derived through the dividing of patterns and trends through means such as statistical pattern learning. Text mining usually involves the process of structuring the input text (usually parsing, along with the addition of some derived linguistic features and the removal of others, and subsequent insertion into a database), deriving patterns within the structured data, and finally evaluation and interpretation of the output. 'High quality' in text mining usually refers to some combination of relevance, novelty, and interestingness. Typical text mining tasks include text categorization, text clustering, concept/entity extraction, production of granular taxonomies, sentiment analysis, document summarization, and entity relation modeling ("i.e.", learning relations between named entities).
Labour-intensive manual text-mining approaches first surfaced in the mid-1980s, but technological advances have enabled the field to advance swiftly during the past decade. Text mining is an
interdisciplinaryfield which draws on information retrieval, data mining, machine learning, statistics, and computational linguistics. As most information (over 80%) is currently stored as text, text mining is believed to have a high commercial potential value.Increasing interest is being paid to multilingual data mining: the ability to gain information across languages and cluster similar items from different linguistic sources according to their meaning.
Sentiment analysismay, for example, involve analysis of movie reviews for estimating how favorably a review is for a movie. [Cite conference
author = Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan
title = Thumbs up? Sentiment Classification using Machine Learning Techniques
booktitle = Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)
pages = 79–86
url = http://www.cs.cornell.edu/home/llee/papers/sentiment.pdf] Such an analysis may require a labeled data set or labeling of the affectivity of words. A resource for affectivity of words has been made for
WordNet. [Cite journal
author = Alessandro Valitutti, Carlo Strapparava, Oliviero Stock
title = Developing Affective Lexical Resources
issue = 1
pages = 61–83
url = http://www.psychnology.org/File/PSYCHNOLOGY_JOURNAL_2_1_VALITUTTI.pdf]
Recently, text mining has received attention in many areas.
One of the largest text mining applications that exists is probably the classified
ECHELONsurveillance system. Additionally, many text mining software packages such as AeroText, Attensity, SPSSand Expert Systemare marketed towards security applications, particularly analysis of plain text sources such as Internet news.
Europol's Serious Crime division developed an analysis system in order to track transnational organized crime. This Overall Analysis System for Intelligence Support (OASIS) integrates among the most advanced text analytics and text mining technologies available on today's market. This system led Europol to make the most significant progress to support law enforcement objectives at the international level. [cite web| url=http://www.ialeia.org/awards| title="IALEIA-LEIU Annual Conference in Boston on April 9, 2008"]
A range of text mining applications in the biomedical literature has been described. [Cite journal
author = K. Bretonnel Cohen & Lawrence Hunter
title = Getting Started in Text Mining
PLoS Computational Biology
month = January
volume = 4
issue = 1
pages = e20
doi = 10.1371/journal.pcbi.0040020
url = http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.0040020] One example is
PubGenethat combines biomedical text mining with network visualization as an Internet service. [Cite journal
author = Tor-Kristian Jenssen, Astrid Lægreid, Jan Komorowski1 & Eivind Hovig
title = A literature network of human genes for high-throughput analysis of gene expression
volume = 28
pages = 21–28
doi = 10.1038/ng0501-21
pmid = 11326270
url = http://www.nature.com/ng/journal/v28/n1/abs/ng0501_21.html
doi_brokendate = 2008-06-20
* Summary: Cite journal
author = Daniel R. Masys
title = Linking microarray data to the literature
volume = 28
pages = 9–10
pmid = 11326264
doi = 10.1038/ng0501-9
doi_brokendate = 2008-06-20] Another example, which uses ontologies with textmining is [http://www.gopubmed.org GoPubMed.org] . [Cite journal
author = Andreas Doms, Michael Schroeder
title = GoPubMed: exploring PubMed with the Gene Ontology
Nucleic Acids Research
volume = 33
pages = W783–W786
doi = 10.1093/nar/gki470
url = http://www.nature.com/ng/journal/v28/n1/abs/ng0501_21.html
pmid = 15980585]
oftware and applications
Research and development departments of major companies, including
IBMand Microsoft, are researching text mining techniques and developing programs to further automate the mining and analysis processes.Text mining software is also being researched by different companies working in the area of search and indexing in general as a way to improve their results.
Text mining is starting to be used in marketing as well, more specifically in analytical
Customer relationship management. [http://www.textmining.UGent.be Coussement and Van den Poel] (2008) apply it to improve predictive analyticsmodels for customer churn ( Customer attrition). [Cite journal
author = Kristof Coussement, and Dirk Van den Poel
title = Integrating the Voice of Customers through Call Center Emails into a Decision Support System for Churn Prediction
journal = Information and Management
month = forthcoming
url = http://www.textmining.ugent.be] .
The issue of text mining is of importance to publishers who hold large
databasesof information requiring indexing for retrieval. This is particularly true in scientific disciplines, in which highly specific information is often contained within written text. Therefore, initiatives have been taken such as Nature's proposal for an Open Text Mining Interface (OTMI) and NIH's common Journal Publishing Document Type Definition(DTD) that would provide semantic cues to machines to answer specific queries contained within text without removing publisher barriers to public access.
Academic institutions have also become involved in the text mining initiative:
National Centre for Text Mining, a collaborative effort between the Universities of Manchester and Liverpool, provides customised tools, research facilities and offers advice to the academic community.They are funded by the Joint Information Systems Committee(JISC) and two of the UK Research Councils.With an initial focus on text mining in the biological and biomedicalsciences, research has since expanded into the areas of Social Science.
In the United States, the School of Information at
University of California, Berkeleyis developing a program called BioText to assist bioscience researchers in text mining and analysis.
Software and applications
Research and development departments of major companies, including
IBMand Microsoft, are researching text mining techniques and developing programs to further automate the mining and analysis processes.Text mining software is also being researched by different companies working in the area of search and indexing in general as a way to improve their results.There is a large number of companies that provide commercial computer programs:
AeroText- provides a suite of text mining applications for content analysis. Content used can be in multiple languages.
* Autonomy - suite of text mining, clustering and categorization solutions for a variety of industries.
* [http://www.eaagle.com/index.php?go=FTM Eaagle Full Text Mapper (FTM)] - a text mining software solution that uses knowledge discovery and data visualization as a basis for analyzing unstructured text.
Endeca Technologies- provides software to analyze and cluster unstructured text.
Expert System S.p.A.- suite of semantic technologies and products for developers and knowledge managers.
Fair Isaac- leading provider of decision management solutions powered by advanced analytics (includes text analytics).
Inxight- provider of text analytics, search, and unstructured visualization technologies. (Inxight was bought by Business Objects that was bought by SAP AGin 2008)
Pervasive Data Integrator- includes Extract Schema Designer that allows the user to point and click identify structure patterns in reports, html, emails, etc. for extraction into any database
* RapidMiner/YALE - open-source data and text mining software for scientific and commercial use.
* SAS - solutions including SAS Text Miner and Teragram - commercial text analytics, natural language processing, and taxonomy softwares leveraged for
Information Management. [http://www.sas.com/technologies/analytics/datamining/textminer/]
SPSS- provider of SPSS Text Analysis for Surveys, Text Mining for Clementine, LexiQuest Mine and LexiQuest Categorize, commercial text analytics software that can be used in conjunction with SPSS Predictive Analytics Solutions.
Thomson Data Analyzer- Enables complex analysis on patent information, scientific publications and news.
LexisNexis- LexisNexis is a provider of business intelligence solutions based on an extensive news and company information content set. Through the recent acquisition of Datops LexisNexis is leveraging its search and retrieval expertise to become a player in the text and data mining field. [http://www.lexisnexisanalytics.com]
LanguageWare- Text Analysis libraries and customization tooling from IBM [http://www.alphaworks.ibm.com/tech/lrw]
Open-source software and applications
* GATE - natural language processing and language engineering tool.
* YALE/RapidMiner with its Word Vector Tool plugin - data and text mining software.
* tm [http://cran.r-project.org/web/packages/tm/index.html] [http://www.jstatsoft.org/v25/i05] - text mining in the
R programming language
* UIMA - architecture for integrating natural language processing capabilities [http://incubator.apache.org/uima/]
Until recently websites most often used text-based lexical searches; in other words, users could find documents only by the words that happened to occur in the documents. Text mining may allow searches to be directly answered by the
semantic web; users may be able to search for content based on its meaning and context, rather than just by a specific word.
Additionally, text mining software can be used to build large dossiers of information about specific people and events. For example, by using software that extracts specifics facts about businesses and individuals from news reports, large datasets can be built to facilitate
social networks analysisor counter-intelligence. In effect, the text mining software may act in a capacity similar to an intelligence analystor research librarian, albeit with a more limited scope of analysis.
Text mining is also used in some email
spam filters as a way of determining the characteristics of messages that are likely to be advertisements or other unwanted material.
* Ronen Feldman and James Sanger, "The Text Mining Handbook", Cambridge University Press, ISBN 9780521836579
* Kao Anne, Poteet, Steve R. (Editors), Natural Language Processing and Text Mining, Springer, ISBN-10: 184628175X
* Konchady Manu "Text Mining Application Programming (Programming Series)" by Manu Konchady, Charles River Media, ISBN 1584504609
* M. Ikonomakis, S. Kotsiantis, V. Tampakas, Text Classification Using Machine Learning Techniques, WSEAS Transactions on Computers, Issue 8, Volume 4, August 2005, pp. 966-974 (http://www.math.upatras.gr/~esdlab/en/members/kotsiantis/Text%20Classification%20final%20journal.pdf)
Approximate nonnegative matrix factorization, an algorithm used for text mining
BioCreativetext mining evaluation in biomedical literature
Natural language processing
Text classificationsometimes is considered a (sub)task of text mining.
UIMAUnstructured Information Management Architecture from IBM.
Web mining, a task that may involve text mining (e.g. first find appropriate web pages by classifying crawled web pages, then extract the desired information from the text content of these pages considered relevant).
* http://www.itl.nist.gov/iaui/894.02/related_projects/muc/ MUC
* http://projects.ldc.upenn.edu/ace/ ACE (LDC)
* http://www.itl.nist.gov/iad/894.01/tests/ace/ ACE (NIST)
* http://www.arts-humanities.net/text_mining (Discussion group text mining)
* [http://portal.tapor.ca Text Analysis Portal for Research (TAPoR)]
* http://textanalytics.wikidot.com/ Text Analytics Wiki
* [http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.0040020;jsessionid=7C485EC9A7B5B0B48AB12894E268DB7A Getting started in text mining]
* [http://erabaki.ehu.es/jjga/pimiento Pimiento] A Text-Mining Application Framework written in Java.
Wikimedia Foundation. 2010.
Look at other dictionaries:
Text Mining — Text Mining, seltener auch Textmining, Text Data Mining oder Textual Data Mining, ist ein Bündel von Analyseverfahren, die die algorithmusassistierte Entdeckung von Bedeutungsstrukturen aus un oder schwachstrukturierten Textdaten ermöglichen soll … Deutsch Wikipedia
text mining — noun (computing) The application of the principles of data mining (qv) to large amounts of written text • • • Main Entry: ↑text … Useful english dictionary
Text-Mining — Der Begriff Text Mining (engl. für „Textschürfen“) bezeichnet die automatisierte Entdeckung relevanter Informationen aus Textdaten. Mit statistischen und linguistischen Mitteln erschließt Text Mining Software aus Texten Informationen, die die… … Deutsch Wikipedia
Text mining — Fouille de textes La fouille de textes ou l extraction de connaissances dans les textes est une spécialisation de la fouille de données et fait partie du domaine de l intelligence artificielle. Cette technique est souvent désignée sous l… … Wikipédia en Français
text mining — /ˈtɛkst maɪnɪŋ/ (say tekst muyning) noun the extraction of information from a large text database by computer searches designed to uncover new information which is not easily retrieved by conventional means … Australian English dictionary
Biomedical text mining — (also known as BioNLP) refers to text mining applied to texts and literature of the biomedical and molecular biology domain. Itis a rather recent research field on the edge of natural language processing, bioinformatics, medical informatics and… … Wikipedia
National Centre for Text Mining — The National Centre for Text Mining (NaCTeM) was the world’s first publicly funded text mining (TM) centre. It was established to provide support, advice, and information on TM technologies and to disseminate information from the … Wikipedia
Text analytics — The term text analytics describes a set of linguistic, lexical, pattern recognition,extraction, tagging/structuring, visualization, and predictive techniques. The termalso describes processes that apply these techniques, whether independently or… … Wikipedia
Mining industry of Angola — Mining in Angola is an activity with great economic potential since the country has one of the largest and most diversified mining resources of Africa. Angola is the third largest producer of diamonds in Africa and has only explored 40% of the… … Wikipedia
Mining industry of Ghana — accounts for 5% of the country s GDP and minerals make up 37% of total exports, of which gold contributes over 90% of the total mineral exports. Thus, the main focus of Ghana s mining and minerals development industry remains focused on gold.… … Wikipedia