Enterprise Search is still Elusive and Ineffective

Search is all about looking among a collection of files and web sites for content that meets some criterion. For this to happen there must be (a) access to that content store; (b) the means to find a match with the search criterion if a match exists, (c) access or authorization to view the search results. All search engines do (b) well - to one extent or another - and none confront (c) or they claim to do it, by doing it the wrong way.

Discovery is all about finding content that is relevant without having explicitly set out to search for it. Discovery requires that the search system understands the user’s present context and perspective.

When we “google”, we are searching through hundreds of billions of web pages and documents in the world wide web – content that people and organizations have published with no intention of keeping private. The moment we click on a search result, we are actually going to the site where that web page or content resides and if subsequent to the original indexing, the site has been secure we may occasionally be prevented from accessing that site. But even in those cases the search engine usually will provide a cached page of results showing what it found when it did the indexing sometime in the past. That, and a lot of technology, is how finding stuff has become so easy in the open internet.

Very secure organizations such as the defense organizations will use a rigorous method of classification that ensures that secret, top secret, and “eyes only” information is accessible only by a select few who have been vetted and classified to a sufficient security level, and then only if they have a need-to-know. The bar for “need-to-know” is just as it sounds – does the person with the right classification have a need to know in order to execute his or her mission. The pecking order, curiosity, or “I am the boss” doesn’t mean anything when it comes to need to know. This author has had people with a lower security clearance than him, who reported to him, assigned to projects where he did not have to directly supervise or comment on their work and therefore had no need-to-know. There was strictly no access to that information.

Commercial enterprises fall somewhere in between open search and security control. The value of the knowledge in their content is useful when others in the company can find it, learn from it, and re-use it – encouraging businesses to share the information with others in the company. On the other hand, the proprietary nature of the information (such as intellectual property, or competition sensitive content) will cause them to hide it from free access. A third dimension that companies are required to follow is compliance – compliance with their own policies, governmental regulations at the local, state, and federal level of each country in which they operate.

Compliance levies certain requirements related to record keeping with integrity of data, document, and other content that reflect the decisions made and the process by which the decisions were arrived at. Nothing escapes this wide net – emails, documents, analysis and spreadsheets, data and technical files, meeting notes, calendar items that indicate meetings and phone calls, Instant messaging, and even hand written notes. They must be retained for a stipulated period of time, and they must be available for audit.

Goto Top