examine Wiki & Documentation Rss Feed

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 22:03:54 GMT

What is Examine?

Examine allows you to index and search data easily and wraps the Lucene.Net indexing/searching engine. Lucene is super fast and allows for very fast searching even on very large amounts of data. Examine is provider based so it is very extensible and allows you to configure as many indexes as you like and each may be configured individually. Out of the box Examine gives you abstract implementations of Lucene based indexers and searchers as well as a Fluent API that can be used to search for your data.

Documentation links

Terminology
Naming Conventions
Creating an Indexer
Creating a Searcher
Creating an Index Set
ExamineManager
Fluent Search API
Support for faceted search
Examine with Azure
For UmbracoExamine documentation see http://our.umbraco.org/documentation/Reference/Searching/Examine/

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 22:03:37 GMT

What is Examine?

Documentation links

For UmbracoExamine documentation see http://our.umbraco.org/documentation/Reference/Searching/Examine/

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 22:03:14 GMT

What is Examine?

Documentation links

For UmbracoExamine documentation see http://our.umbraco.org/documentation/Reference/Searching/Examine/

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 22:03:05 GMT

What is Examine?

Documentation links

_For UmbracoExamine documentation see http://our.umbraco.org/documentation/Reference/Searching/Examine/

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:48:02 GMT

What is Examine?

Documentation links

Examine Implementations

UmbracoExamine

Updated Wiki: IndexSet

Shandem — Thu, 04 Apr 2013 21:47:30 GMT

Creating an Index Set

For each Indexer and Searcher specified you will require an index set. Creating an index set is simply done via configuration and normally is used to just supply the location of where the Lucene index will be stored.

Index sets can also specify what data to store. This is done by filters by including/excluding certain data types, and including only the specified fields. If none of this is specified then all data found in the data source will be indexed. When specifying fields you can also configure what the the data type for a given field is (i.e. date, number, etc...) which affects how it is stored and how it can be searched.

Config

  <ExamineLuceneIndexSets>
    <IndexSet SetName="myIndexSet" IndexPath="~/App_Data/myIndexSet" />
  </ExamineLuceneIndexSets>

Updated Wiki: IndexSet

Shandem — Thu, 04 Apr 2013 21:45:15 GMT

Creating an Index Set

Config

  <ExamineLuceneIndexSets>
    <IndexSet SetName="myIndexSet" IndexPath="~/App_Data/myIndexSet" />
  </ExamineLuceneIndexSets>

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:43:20 GMT

What is Examine?

Documentation links

Examine Implementations

UmbracoExamine

Updated Wiki: ExamineWithAzure

Shandem — Thu, 04 Apr 2013 21:42:19 GMT

Using Examine on Azure

First, in order to support Lucene on Azure, you must have a Azure blob storage account and then you'll need to get the Azure Examine binaries.

Now its time to start configuring your app to run on Azure. First, you'll need to add an appSetting to your web.config:

<add key="blobStorage" value="DefaultEndpointsProtocol=http;AccountName=YOURACCOUNTNAME;AccountKey=YOURACCOUNTKEY"/>

If you don't want to register this value in your web.config, you should be able to add this setting with this same key to your Azure settings. Also note that DefaultEndpointsProtocol above can be either 'http' or 'https'

This next configuration change is highly recommended since when testing in Azure once in a while Lucene file locks would occur during rebuild so it is recommended to disable rebuilding the indexes on start-up if they don't exist:

<Examine RebuildOnAppStart="false">

Next we need to configure your index and search providers. This involves changing the 'type' attribute of your Indexers and Searchers to use the correct Azure Examine provider. Use the conversion below to determine the type of provider you'll need:

Non-Azure provider	Azure provider
Examine.LuceneEngine.Providers.SimpleDataIndexer	Examine.Azure.SimpleAzureLuceneIndexer
UmbracoExamine.UmbracoContentIndexer	UmbracoExamine.Azure.AzureContentIndexer
UmbracoExamine.UmbracoMemberIndexer	UmbracoExamine.Azure.AzureMemberIndexer
UmbracoExamine.PDF.PDFIndexer	UmbracoExamine.PDF.Azure.AzurePDFIndexer
UmbracoExamine.UmbracoExamineSearcher	UmbracoExamine.Azure.AzureSearcher

Lastly, you'll need to update your Examine Index config to change the location of your Index Sets. Change your 'IndexPath' to just be an alphanumeric string name such as "MyIndex", this will create a blog storage catalogue with this name to store you index in.

If you've created your own custom provider and want it to work with Azure, you will need to create a new Azure aware custom provider. This doesn't require much code and utilizes an AzureDirectory object for the LuceneDirectory used to index and search. For examples on how to do this, please get the source code for Examine and see how the above Azure providers have been created.

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:42:07 GMT

What is Examine?

Documentation links

Examine Implementations

UmbracoExamine

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:41:29 GMT

What is Examine?

Documentation links

Examine Implementations

UmbracoExamine

Examine with Azure

Updated Wiki: Fluent Search API

Shandem — Thu, 04 Apr 2013 21:40:00 GMT

Fluent Search API

Examine offers a fluent (chainable) search API which aims to make constructing complex searches simple. The underlying API is determined by the provider implementation, with Examine just exposing the appropriate methods.

An example of the fluent API in action is as follows:

ISearchCriteria sc = ExamineManager.Instance.CreateSearchCriteria(100 /* max record count */, IndexType.Content /* type of index to query */);
IBooleanOperation query = sc.NodeName("Examine").And().Range("createdDate", new DateTime(2010, 03, 21), new DateTime(2010, 03, 31); // will match all nodes which have the name Examine created between the 21st and the 31st March 2010).
IEnumerable<SearchResult> results = ExamineManager.Instance.Search(query.Compile()); // prepares the query to be handled by the searcher

ISearchCriteria

The ISearchCriteria interface is the real workhorse of the API, it’s the first interface you start with, and it’s the last interface you deal with. In fact, ISearchCriteria implements IQuery, meaning that all the query operations start here.

In addition to query operations there are several additional properties for such as the maximum number of results and the type of data being searched.

Because ISearchCriteria is tightly coupled with the BaseSearchProvider implementation it is actually created via a factory pattern, like so:

ISearchCriteria searchCriteria = ExamineManager.Instance.SearchProviderCollection["MySearcher"].CreateSearchCriteria(100, IndexType.Content);

What we’re doing here is requesting that our BaseSearchProvider creates an instance of an ISearchCriteria. It takes two parameters:

int maxResults
Examine.IndexType indexType

This data can/ should be then used by the search method to return what’s required.

IQuery

The IQuery interface is really the heart of the fluent API, it’s what you use to construct the search for your site. Since Examine is designed to be technology agnostic the methods which are exposed via IQuery are fairly generic. A lot of the concepts are borrowed from Lucene.Net, but they are fairly generic and should be viable for any searcher.

The IQuery API exposes the following methods:

IBooleanOperation Id(int id);
IBooleanOperation NodeName(string nodeName);
IBooleanOperation NodeName(IExamineValue nodeName);
IBooleanOperation NodeTypeAlias(string nodeTypeAlias);
IBooleanOperation NodeTypeAlias(IExamineValue nodeTypeAlias);
IBooleanOperation ParentId(int id);
IBooleanOperation Field(string fieldName, string fieldValue);
IBooleanOperation Field(string fieldName, IExamineValue fieldValue);
IBooleanOperation MultipleFields(IEnumerable<string> fieldNames, string fieldValue);
IBooleanOperation MultipleFields(IEnumerable<string> fieldNames, IExamineValue fieldValue);
IBooleanOperation Range(string fieldName, DateTime start, DateTime end);
IBooleanOperation Range(string fieldName, DateTime start, DateTime end, bool includeLower, bool includeUpper);
IBooleanOperation Range(string fieldName, int start, int end);
IBooleanOperation Range(string fieldName, int start, int end, bool includeLower, bool includeUpper);
IBooleanOperation Range(string fieldName, string start, string end);
IBooleanOperation Range(string fieldName, string start, string end, bool includeLower, bool includeUpper);

As you can see all the methods within the IQuery interface return an IBooleanOperator, this is how the fluent API works!

Hopefully it’s fairly obvious what each of the methods are, but the one you’re most likely to use is Field. Field allows you to specify any field in your index, and then provide a word to lookup within that field.

IExamineValue

You’ve probably noticed the IExamineValue parameter which is passable to a lot of the different methods, methods which take a string, but what is IExamineValue?
Well obviously it’s some-what provider dependant, so I’ll talk about it as part of Umbraco Examine, as that’s what I think most initial uptakers will want.

Because Lucene supports several different term modifiers for text we decided it would be great to have those exposed in the API for people to leverage. For this we’ve got a series of string extension methods which reside in the namespace

UmbracoExamine.SearchCriteria

So once you add a using statement for that you’ll have the following extension methods:

public static IExamineValue SingleCharacterWildcard(this string s)
public static IExamineValue MultipleCharacterWildcard(this string s)
public static IExamineValue Fuzzy(this string s)
public static IExamineValue Fuzzy(this string s, double fuzzieness)
public static IExamineValue Boost(this string s, double boost)
public static IExamineValue Proximity(this string s, double proximity)
public static IExamineValue Excape(this string s)

All of these return an IExamineValue (which UmbracoExamine internally handles), and it tells Lucene.Net how to handle the term modifier you required.

I wont repeat what is said within the Lucene documentation, I suggest you read that to get an idea of what to use and when.
The only exceptions are Escape.

Escape

If you’re wanting to search on multiple works together then Lucene requires them to be ‘escaped’, otherwise it’ll (generally) treat the space character as a break in the query. So if you wanted to search for Umbraco Rocks and didn’t escape it you’d match on both Umbraco and Rocks, where as when it’s escaped you’ll then match on the two words in sequence.

IBooleanOpeation

IBooleanOperation allows your to join multiple IQuery methods together using:

IQuery And()
IQuery Or()
IQuery Not()

These are then translated into the underlying searcher so it can determine how to deal with your chaining. At the time of writing we don’t support nested conditionals (grouped OR’s operating like an And).
Each operator will then return an IQuery which will allow you to chain another query method into it.

There’s another method on IBooleanOperation which doesn’t fall into the above, but it’s very critical to the overall idea:

ISearchCriteria Compile()

The Compile method will then return an ISearchCriteria which you then pass into your searcher. It’s expected that this is the last method which is called and it’s meant to prepare all search queries for execution.
The reason we’re going with this rather than passing the IQuery into the Searcher is that it means we don’t have to have the max results/ etc into every IQuery instance, it’s not something that is relevant in that scope, so it’d just introduce code smell, and no one wants that.

Example:

var sc = ExamineManager.Instance.CreateSearchCriteria();
var query = sc.NodeName("umbraco").And().Field("bodyText", "is awesome".Escape()).Or().Field("bodyText", "rock".Fuzzy());
var results = ExamineManager.Instance.Search(query.Compile());

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:39:38 GMT

What is Examine?

Documentation links

Examine Implementations

Examine with Azure

Examine Implementations

UmbracoExamine

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:38:46 GMT

What is Examine?

Documentation links

Examine Implementations

Examine with Azure

Examine Implementations

UmbracoExamine

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:38:20 GMT

What is Examine?

Documentation links

Examine Implementations

Examine with Azure

Examine Implementations

UmbracoExamine

Updated Wiki: NamingConventions

Shandem — Thu, 04 Apr 2013 21:37:51 GMT

Naming conventions

It is important to note the naming conventions above. Your Indexer, Searcher and associated Index Set must all be named according to convention so that they match. With the above examples the naming conventions are:

ExternalIndexer
ExternalSearcher
ExternalIndexSet

Notice that the prefix is all the same, this is a requirement. The suffixes must also match so that the indexer name is suffixed with Indexer, the searcher is suffixed with Searcher and the index set is suffixed with IndexSet.

Updated Wiki: NamingConventions

Shandem — Thu, 04 Apr 2013 21:37:45 GMT

Naming conventions

Updated Wiki: NamingConventions

Shandem — Thu, 04 Apr 2013 21:37:39 GMT

!!Naming conventions

It is important to note the naming conventions above. Your Indexer, Searcher and associated Index Set must all be named according to convention so that they match. With the above examples the naming conventions are:

ExternalIndexer
ExternalSearcher
ExternalIndexSet

Notice that the prefix is all the same, this is a requirement. The suffixes must also match so that the indexer name is suffixed with Indexer, the searcher is suffixed with Searcher and the index set is suffixed with IndexSet.

Updated Wiki: Terminology

Shandem — Thu, 04 Apr 2013 21:37:28 GMT

Examine Terminology

Describes the different terms/names used throughout Examine

Index

An Index is the repository that stores searchable data, generally Examine uses Lucene as it's index repository but if you really wanted it is possible to use any repository, though this would require writing a lot of code.

The 3 basic parts

Examine is made up of 3 basic parts:Indexers, Searchers and Index Sets. Each one of these components requires exactly 1 of the other components, there is a 1 to 1 to 1 ratio between these objects. For example, each Index Set will have an associated Indexer and Searcher.

Indexer

An indexer in Examine is the object that performs the storing data into the index. The most basic interface that defines an indexer is `Examine.IIndexer`. An indexer's functionality is to: Add or update index data for a data entity, delete index data for a data entity, re-index all data based on a specified data source.

Searcher

A searcher in Examine is the object that performs the searching of data that is stored in the index.The most basic interface that defines a searcher is `Examine.ISearcher`. Examine has the capability to search based on it's own Fluent Api, or by querying directly to Lucene using raw lucene search syntax.

Index Set

An index set is what defines an index, where the index is saved and how the information is stored in the index.

Updated Wiki: Documentation

Shandem — Thu, 04 Apr 2013 21:37:19 GMT

What is Examine?

Documentation links

Examine Implementations

Examine with Azure

Fluent Search API

ISearchCriteria sc = ExamineManager.Instance.CreateSearchCriteria(100 /* max record count */, IndexType.Content /* type of index to query */);
IBooleanOperation query = sc.NodeName("Examine").And().Range("createdDate", new DateTime(2010, 03, 21), new DateTime(2010, 03, 31); // will match all nodes which have the name Examine created between the 21st and the 31st March 2010).
IEnumerable<SearchResult> results = ExamineManager.Instance.Search(query.Compile()); // prepares the query to be handled by the searcher

Full documentation on the Fluent Search API can be found here

Examine Implementations

UmbracoExamine