By the Price of Business Show, Hosted by Kevin Price. The Price of Business is a media partner of this site
A significant development in the world of file formats is not an everyday occurrence. But more than 25 years after Adobe created the PDF file format in 1993, there is a major new version of PDF: PDF 2.0. File format advancements are big news for a search engine like dtSearch.
The underlying technology that recognizes PDF and other file types is called “document filters.” If you looked at a PDF file or Word document in its raw binary format as a search engine needs to look at it, it would appear as gibberish. You might be able to find a word here and there but reading it as text would be nearly impossible. What you’d need as an initial step is to review or parse the binary format to find the text and metadata inside.
The first step for the document filters is to recognize the file format. If you are interested in this process from a forensics-oriented perspective, dtSearch figures out the file type from the contents of the file itself, not the file extension. So if a OneNote file or PDF file is mislabeled with a .docx extension, dtSearch will still work with that correctly.
In parsing files such as PDF 2.0, the document filters need to follow the text and the metadata inside wherever it may lead. In the same PDF file, you can have English text followed by Chinese text followed by Russian text. The dtSearch document filters need to identify the correct data format and then identify the correct Unicode encodings within that data format.
Document filters also need to work with compound files. Suppose you have an email with a ZIP or RAR attachment containing OneNote files and PDFs. Or suppose you have a PowerPoint embedded inside a Word document. The dtSearch documents filters must parse all of these multilevel nested structures to “read” the text correctly.
Once dtSearch has parsed file formats like emails plus attachments, PDFs, and Microsoft Office documents, dtSearch can instantly search terabytes of files, along with databases and web-based data. dtSearch does this by first building an index that holds each unique word in the data, and the location of that word in the data. To build the index, just point dtSearch to a folder or a folder tree or even a whole disk drive. As noted above, there is no need to tell dtSearch what file formats are in that folder tree or disk drive, as the document filters figure that out from the documents themselves.
Beyond PDF 1 and 2, there are a couple of categories of PDFs that also require attention for a search engine and its document filters. One is PDFs that are “image only” and accordingly require Optical Character Recognition or OCR from a product like Adobe Acrobat to turn that image into text. If you are looking at a PDF from inside Adobe Reader and you see text there, but when you try to cut and paste that text you can’t do it, that is typically an image-only PDF. dtSearch can flag those so that you can run them through Adobe Acrobat, for example, and turn them into searchable PDFs.
The second category is encrypted files. While dtSearch can index and search many encrypted PDFs, there are other files that can be encrypted so that a third-party product is required to unencrypt them before dtSearch can index and search them. dtSearch can also flag those files for you.
Enterprises with extremely large data sets like government agencies and 4 out of 5 of the Fortune 500’s largest Aerospace and Defense companies use dtSearch enterprise and developer products to instantly search terabytes of data. However, if you just want to search your own PC, you can download a fully-functional 30-day evaluation version of dtSearch Desktop anytime at dtSearch.com
