By Elizabeth Thede, Special for The Time USA
Many people have heard of Boolean searching, which relies on and/or/not connectors. For example, apple and (banana or pear) and not broccoli is a sample Boolean expression as it combines and/or/not with the keywords apple, banana, pear and broccoli. The purpose of this article is to start with a simple Boolean search, and show how today’s text retrieval moves way beyond that.
Take dtSearch, for example. dtSearch can instantly search across terabytes of data including a wide variety of different data formats, like Microsoft “Office” files, compressed file types, emails along with the full-text of nested email attachments, and PDFs, including the brand new PDF 2.0 format. Beyond files and emails, dtSearch can also search databases and certain other offline secure Intranet as well as online public Internet content.
dtSearch instantly searches terabytes of data by building an index that holds each unique word in the data, and the location of that word in the data. To build the index, dtSearch just needs to know what folders or drives you want to index, and dtSearch will do the rest, including figuring out what type of files, emails and the like are in those folders or drives.
Once the index is done, dtSearch lets you instantly search any combination of indexed data using more than 25 different search types. Federated searching can search across multiple data sources. And with concurrent multithreaded searching, multiple parties can all search across the same index or indexes at once, again using any combination of over 25 search options.
One such search type is Boolean. For example, with apple and (banana or pear) and not broccoli, dtSearch would find any documents, emails, attachments and the like that contain the word apple and either (banana or pear) and absolutely no mention of broccoli. After a search, dtSearch can highlight hits in all one color or using multiple colors, such as banana in yellow and apple in green.
But you can also extend Boolean searching with phrase searching, such as: green apple and (yellow banana or red pear) and not broccoli. And you can also mix in proximity searching, like green apple within 23 words of yellow banana, or directed proximity searching like green apple within 23 words preceding yellow banana.
Stemming can find different endings on a word. So, for yellow banana, it would pick up yellowing bananas. You can take all the same search requests and turn on fuzzy searching, so if someone slightly mistypes banana in an email and the resulting misspelling is banona, dtSearch will still find that with a low level of fuzzy searching. With a higher level of fuzzy searching, dtSearch would pick up banana, banona and bandanna.
dtSearch also supports concept searching based on a built-in thesaurus or “synonym rings” that the end-user adds on the fly. Phonic searching finds English sound-alikes. And with wildcard searching, a ? can stand in for any single character and an * for any number of characters.
Then there is a whole other type of searching called natural language. For a natural language search, dtSearch takes just they keywords, ignoring any connectors, to find documents, emails, etc. that are the best match based on vector space relevancy ranking. So if apple is in 10 million documents and banana is only in 3, banana would get a much higher weight than apple in returning files containing that keyword.
And dtSearch lets you add custom variable term weighting on top of that, so apple and banana could receive a positive weight of six and you can add on broccoli with a negative weight of nine. You can combine term weighting with Boolean search as well. The advantage there would be that you could downgrade any document or email that contains the word broccoli without eliminating those entirely from search results.
While dtSearch can perform any of the above type of searches across all content in emails, documents and the like, dtSearch can also limit searching of particular terms to certain metadata fields. You could look for emails that contain the word apple in the subject line and (yellow banana or red pear) and not broccoli in the document as a whole. Or you could rank banana higher but only if it appears as subject line metadata. Beyond classic word searching, dtSearch also supports regular expression searching, numeric range searching, identification of credit card numbers, and even searches for a file’s unique hash value.
With the dtSearch Engine package, dtSearch adds faceted searching for leveraging metadata in documents or in a backend SQL or NoSQL database (which the dtSearch Engine can also search). This type of search lets the end-user “drill down” through different metadata levels prior to performing a full-text search. And granular data classification lets everyone perform the same searches across the same index or indexes, while ensuring that the retrieved data for each end-user will only match what the enterprise authorizes that end-user to see.
In conclusion, very large enterprises like government agencies and Fortune 100 companies use dtSearch enterprise and developer products to instantly search (with well over 25 search options) terabytes of online and offline data. However, even if you just want to search your own PC, you can download a fully-functional 30-day evaluation version of dtSearch Desktop anytime at dtSearch.com
