Connecting your ASP.NET Core application to Elasticsearch using NEST 5.x (Part 1/4)

Hello! My name is Dobromir Nikolov and in this tutorial I’m going to teach you how you can integrate ASP.NET Core together with Elastic and NEST 5.X to build a simple, yet powerful recipes search engine (full code on github). The tutorial is going to be split in 4 parts:

1. Installing the NEST package and using it to connect your application to your Elastic instance.
2. Implementing functionality for reading and indexing the data into Elastic through NEST.
3. Implementing the Search functionality.
4. Implementing Autocomplete, MoreLikeThis and GetById functionality.

What you’ll learn

By the end of part 4 you’ll have learned:

1. How to enable your .NET application to "talk" to your Elastic instance through NEST’s ElasticClient.
2. How to provide an ElasticClient instance throughout your application.
3. How to use NEST AttributeMapping.
4. How to use the ElasticClient object to index data into Elastic.
5. How to build an Elastic query using C# and fluent DSL.
6. How to make use of the Completion and MoreLikeThis queries.

In this part we’re going to cover the first 2 steps, but first, let’s take a look at…

---

The application we’re building

The application is simple, but extremely flexible in terms of searching data.

Behind the scenes, the user inputted query is converted to a query_string query that is sent to Elastic. We’re taking advantage of query_string‘s native support for wildcards, boolean operators, phrase matching and boosting.

The user inputted string is proccessed in the following way:
> * Exact phrases to look for are marked with quotes ("an example phrase").
> * Each word/phrase that isn’t marked in any way (example) will be present in the result set’s ingredients.
> * Each word/phrase that is marked with a minus (-example) will not be present in the result set’s ingredients.
> * Each word/phrase can be boosted to become more relevant (score higher) in the search results (example^2).

So, if we want to look for recipes containing garlic, tomatoes, chopped onions and not containing eggs and red-pepper flakes, we’ll have to input the query string:

> garlic^2 tomatoes -egg* "chopped onions" -"red-pepper flakes" // Order doesn’t matter

As you can see, we’ve boosted the garlic. This means that recipes where garlic appears more frequently will score higher.

The asterisk symbol following the egg is called a wildcard query. It means that it’s going to look for words that start with egg and continue with anything else, no matter the length (for example, egg*** is going to match **eggs, eggplant, eggbeater, etc.), very similar to SQL’s like.

You can substitute the asterisk with a "?" sign, which will indicate that Elastic should look for just a single missing letter.

This spits out the result:

---

Set up Elastic on your local machine

If you haven’t done so already, there is a very nice official guide on setting up Elastic on your local machine. You can check it out here. I advise you to also set up Kibana, since it’s a really useful tool for directly interacting with Elastic. We won’t be relying on Kibana in this tutorial, but I don’t think you can escape it in the long run.

---

Using NEST to connect to your local cluster

By now, you’ve installed and ran Elastic, but in order to make use of its amazing functionality, we have to be able to communicate with it.

In order to strap up .NET to our Elastic cluster, we’ll be using the NEST client, which is the official .NET client that is supported by Elastic. It’s open-source and you can take a look at the code or contribute on Github. You can learn more about NEST by reading the docs.

Step 1. Create an empty ASP.NET Core project

Just fire up Visual Studio and go to:

> File -> New -> Project -> ASP.NET Core Web Application -> Web API -> No Authentication -> OK

Note that if you are using a version different than 2015, there could be a slight difference.

Step 2. Install NEST

NEST is distributed as a Nuget package, so in order to install it, just open your package.json file and under the "dependencies" section, insert a new line. (in VS 2017, you’ll have to edit the .csproj file)

"NEST": "5.4.0" // Or any 5.x version

Alternatively, you can just run the following command in your package manager console.

>Install-Package NEST

Step 3. Provide an ElasticClient throughout your application

In order to "talk" to an Elastic instance, NEST uses an object called ElasticClient.

ElasticClient is responsible for translating our C# queries to plain JSON requests that are sent to and processed by the Elastic cluster we’re connected to. The client is thread-safe and we’ll have just one instance of it throughout our application.

These are the steps we’re going to take:

1. Add our Elastic settings to our appsettings.json file.
2. Create an ElasticConnectionSettings class, which will serve as a container for the aforementioned settings.
3. Create an ElasticClientProvider class, which will be responsible for instantiating and containing the actual ElasticClient we’ll be using. This is the class we’re going to inject when we need an ElasticClient.
4. Register the ElasticClientProvider class in the DI.

Step 1. Add our Elastic settings to appsettings.json

>Note: You can just go to step 3 and hardcode your cluster’s url and index. I don’t like hardcoding things, that’s why I’m moving my settings to appsettings.json.

Open appsettings.json and insert the following section.

"ElasticConnectionSettings": {
    "ClusterUrl": "http://localhost:9200",
    "DefaultIndex": "recipes"
  }

If you want to connect to another url/use another index, just change the variables.

Step 2. Create the ElasticConnectionSettings class

In the root of your project, create a folder called Elastic. In this folder we’ll be containing all the Elastic specific objects. Now create a class called ElasticConnectionSettings with the following implementation:

public class ElasticConnectionSettings
{
    public string ClusterUrl { get; set; }

    public string DefaultIndex
    {
        get
        {
            return this.defaultIndex;
        }
        set
        {
            this.defaultIndex = value.ToLower();
        }
    }

    private string defaultIndex;
}

Its properties will be matched against the ones in the appsettings.json file. The DefaultIndex property has a setter that will always set the lowercase value of the index specified in appsettings.json. This is because indices in Elastic must always be lowercase.

Step 3. Create the ElasticClientProvider class

In the Elastic folder we created, add a new class called ElasticClientProvider with the following implementation:

using Microsoft.Extensions.Options;
using Nest;

public class ElasticClientProvider
{
    public ElasticClientProvider(IOptions<ElasticConnectionSettings> settings)
    {
        // Create the connection settings
        ConnectionSettings connectionSettings =
            // Get the cluster URL from appsettings.json and pass it in
            new ConnectionSettings(new System.Uri(settings.Value.ClusterUrl));
        // This is going to enable us to see the raw queries sent to elastic when debugging (really useful)
        connectionSettings.EnableDebugMode();

        if (settings.Value.DefaultIndex != null)
        {
         // Get the index name from appsettings.json and pass it in
            connectionSettings.DefaultIndex(settings.Value.DefaultIndex);
        }
        // Create the actual client
        this.Client = new ElasticClient(connectionSettings);
    }

    public ElasticClient Client { get; }
}

Now, in order for our provider to work properly, we have to register some services in the Startup.cs file.

Open Startup.cs and in the ConfigureServices method insert the following lines:

// Get the connection settings from appsettings.json and inject them into ElasticConnectionSettings
services.AddOptions();
services.Configure<ElasticConnectionSettings>(Configuration.GetSection("ElasticConnectionSettings"));

This will tell to our application that we have a section in appsettings.json called ElasticConnectionSettings that we want to be mapped to the ElasticConnectionSettings class we made earlier.

Step 4. Register the ElasticClientProvider class for injection

Just below the lines we added in Startup.cs in step 3, add the following:

// Register the client provider as a singleton
services.AddSingleton(typeof(ElasticClientProvider));

This will enable us to inject ElasticClientProvider wherever we need and get our ElasticClient from it.

Example usage:

public class SomeClassThatNeedsAClient
{
    public SomeClassThatNeedsAClient(ElasticClientProvider provider)
    {
        _client = provider.Client;
    }

    private readonly ElasticClient _client;
}

---

Congratulations!

In just a few simple steps, we managed to create, connect and provide an ElasticClient throughout our application! In part 2, we’ll be indexing 170000 recipes!

If you have any questions, don’t hesitate to ask in the comments! If you didn’t like something and have suggestions on how to improve, please contact me!