Elasticsearch优化

2020-12-15 01:30:38 编辑 0 阅读量:487

# 一、General recommendations
## 1. Don’t return large result setsedit
Elasticsearch is designed as a search engine, which makes it very good at getting back the top documents that match a query. However, it is not as good for workloads that fall into the database domain, such as retrieving all documents that match a particular query. If you need to do this, make sure to use the Scroll API.

## 2. Avoid large documentsedit
Given that the default http.max_context_length is set to 100MB, Elasticsearch will refuse to index any document that is larger than that. You might decide to increase that particular setting, but Lucene still has a limit of about 2GB.

Even without considering hard limits, large documents are usually not practical. Large documents put more stress on network, memory usage and disk, even for search requests that do not request the _source since Elasticsearch needs to fetch the _id of the document in all cases, and the cost of getting this field is bigger for large documents due to how the filesystem cache works. Indexing this document can use an amount of memory that is a multiplier of the original size of the document. Proximity search (phrase queries for instance) and highlighting also become more expensive since their cost directly depends on the size of the original document.

It is sometimes useful to reconsider what the unit of information should be. For instance, the fact you want to make books searchable doesn’t necesarily mean that a document should consist of a whole book. It might be a better idea to use chapters or even paragraphs as documents, and then have a property in these documents that identifies which book they belong to. This does not only avoid the issues with large documents, it also makes the search experience better. For instance if a user searches for two words foo and bar, a match across different chapters is probably very poor, while a match within the same paragraph is likely good.

## 3. Avoid sparsityedit（稀疏）
The data-structures behind Lucene, which Elasticsearch relies on in order to index and store data, work best with dense data, ie. when all documents have the same fields. This is especially true for fields that have norms enabled (which is the case for text fields by default) or doc values enabled (which is the case for numerics, date, ip and keyword by default).

The reason is that Lucene internally identifies documents with so-called doc ids, which are integers between 0 and the total number of documents in the index. These doc ids are used for communication between the internal APIs of Lucene: for instance searching on a term with a match query produces an iterator of doc ids, and these doc ids are then used to retrieve the value of the norm in order to compute a score for these documents. The way this norm lookup is implemented currently is by reserving one byte for each document. The norm value for a given doc id can then be retrieved by reading the byte at index doc_id. While this is very efficient and helps Lucene quickly have access to the norm values of every document, this has the drawback that documents that do not have a value will also require one byte of storage.

In practice, this means that if an index has M documents, norms will require M bytes of storage per field, even for fields that only appear in a small fraction of the documents of the index. Although slightly more complex with doc values due to the fact that doc values have multiple ways that they can be encoded depending on the type of field and on the actual data that the field stores, the problem is very similar. In case you wonder: fielddata, which was used in Elasticsearch pre-2.0 before being replaced with doc values, also suffered from this issue, except that the impact was only on the memory footprint since fielddata was not explicitly materialized on disk.

Note that even though the most notable impact of sparsity is on storage requirements, it also has an impact on indexing speed and search speed since these bytes for documents that do not have a field still need to be written at index time and skipped over at search time.

It is totally fine to have a minority of sparse fields in an index. But beware that if sparsity becomes the rule rather than the exception, then the index will not be as efficient as it could be.

This section mostly focused on norms and doc values because those are the two features that are most affected by sparsity. Sparsity also affect the efficiency of the inverted index (used to index text/keyword fields) and dimensional points (used to index geo_point and numerics) but to a lesser extent.

Here are some recommendations that can help avoid sparsity:

#### 3.1 Avoid putting unrelated data in the same indexedit
You should avoid putting documents that have totally different structures into the same index in order to avoid sparsity. It is often better to put these documents into different indices, you could also consider giving fewer shards to these smaller indices since they will contain fewer documents overall.

Note that this advice does not apply in the case that you need to use parent/child relations between your documents since this feature is only supported on documents that live in the same index.

#### 3.2 Normalize document structuresedit
Even if you really need to put different kinds of documents in the same index, maybe there are opportunities to reduce sparsity. For instance if all documents in the index have a timestamp field but some call it timestamp and others call it creation_date, it would help to rename it so that all documents have the same field name for the same data.

#### 3.3 Avoid typesedit
Types might sound like a good way to store multiple tenants in a single index. They are not: given that types store everything in a single index, having multiple types that have different fields in a single index will also cause problems due to sparsity as described above. If your types do not have very similar mappings, you might want to consider moving them to a dedicated index.

## 4. Disable norms and doc_values on sparse fieldsedit
If none of the above recommendations apply in your case, you might want to check whether you actually need norms and doc_values on your sparse fields. norms can be disabled if producing scores is not necessary on a field, this is typically true for fields that are only used for filtering. doc_values can be disabled on fields that are neither used for sorting nor for aggregations. Beware that this decision should not be made lightly since these parameters cannot be changed on a live index, so you would have to reindex if you realize that you need norms or doc_values.

# 二、Tune for search speededit
## 1. Give memory to the filesystem cacheedit
Elasticsearch heavily relies on the filesystem cache in order to make search fast. In general, you should make sure that at least half the available memory goes to the filesystem cache so that elasticsearch can keep hot regions of the index in physical memory.

## 2. Use faster hardwareedit
If your search is I/O bound, you should investigate giving more memory to the filesystem cache (see above) or buying faster drives. In particular SSD drives are known to perform better than spinning disks. Always use local storage, remote filesystems such as NFS or SMB should be avoided. Also beware of virtualized storage such as Amazon’s Elastic Block Storage. Virtualized storage works very well with Elasticsearch, and it is appealing since it is so fast and simple to set up, but it is also unfortunately inherently slower on an ongoing basis when compared to dedicated local storage. If you put an index on EBS, be sure to use provisioned IOPS otherwise operations could be quickly throttled.

If your search is CPU-bound, you should investigate buying faster CPUs.

## 3. Document modelingedit
Documents should be modeled so that search-time operations are as cheap as possible.

In particular, joins should be avoided. nested can make queries several times slower and parent-child relations can make queries hundreds of times slower. So if the same questions can be answered without joins by denormalizing documents, significant speedups can be expected.

## 4. Pre-index dataedit
You should leverage patterns in your queries to optimize the way data is indexed. For instance, if all your documents have a price field and most queries run range aggregations on a fixed list of ranges, you could make this aggregation faster by pre-indexing the ranges into the index and using a terms aggregations.

For instance, if documents look like:
```
PUT index/type/1
{
  "designation": "spoon",
  "price": 13
}
```
and search requests look like:
```
GET index/_search
{
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 10 },
          { "from": 10, "to": 100 },
          { "from": 100 }
        ]
      }
    }
  }
}
```
Then documents could be enriched by a price_range field at index time, which should be mapped as a keyword:
```
PUT index
{
  "mappings": {
    "type": {
      "properties": {
        "price_range": {
          "type": "keyword"
        }
      }
    }
  }
}

PUT index/type/1
{
  "designation": "spoon",
  "price": 13,
  "price_range": "10-100"
}
```
And then search requests could aggregate this new field rather than running a range aggregation on the price field.
```
GET index/_search
{
  "aggs": {
    "price_ranges": {
      "terms": {
        "field": "price_range"
      }
    }
  }
}
```
## 5. Mappingsedit
The fact that some data is numeric does not mean it should always be mapped as a numeric field. Typically, fields storing identifiers such as an ISBN or any number identifying a record from another database, might benefit from being mapped as keyword rather than integer or long.

## 6. Avoid scriptsedit
In general, scripts should be avoided. If they are absolutely needed, you should prefer the painless and expressions engines.

Search rounded datesedit
Queries on date fields that use now are typically not cacheable since the range that is being matched changes all the time. However switching to a rounded date is often acceptable in terms of user experience, and has the benefit of making better use of the query cache.

For instance the below query:
```
PUT index/type/1
{
  "my_date": "2016-05-11T16:30:55.328Z"
}

GET index/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "range": {
          "my_date": {
            "gte": "now-1h",
            "lte": "now"
          }
        }
      }
    }
  }
}
```
could be replaced with the following query:
```
GET index/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "range": {
          "my_date": {
            "gte": "now-1h/m",
            "lte": "now/m"
          }
        }
      }
    }
  }
}
```
In that case we rounded to the minute, so if the current time is 16:31:29, the range query will match everything whose value of the my_date field is between 15:31:00 and 16:31:59. And if several users run a query that contains this range in the same minute, the query cache could help speed things up a bit. The longer the interval that is used for rounding, the more the query cache can help, but beware that too aggressive rounding might also hurt user experience.

Note
It might be tempting to split ranges into a large cacheable part and smaller not cacheable parts in order to be able to leverage the query cache, as shown below:
```
GET index/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "should": [
            {
              "range": {
                "my_date": {
                  "gte": "now-1h",
                  "lte": "now-1h/m"
                }
              }
            },
            {
              "range": {
                "my_date": {
                  "gt": "now-1h/m",
                  "lt": "now/m"
                }
              }
            },
            {
              "range": {
                "my_date": {
                  "gte": "now/m",
                  "lte": "now"
                }
              }
            }
          ]
        }
      }
    }
  }
}
```
However such practice might make the query run slower in some cases since the overhead introduced by the bool query may defeat the savings from better leveraging the query cache.

## 7. Force-merge read-only indicesedit
Indices that are read-only would benefit from being merged down to a single segment. This is typically the case with time-based indices: only the index for the current time frame is getting new documents while older indices are read-only.

Important
Don’t force-merge indices that are still being written to — leave merging to the background merge process.

## 8. Warm up global ordinalsedit
Global ordinals are a data-structure that is used in order to run terms aggregations on keyword fields. They are loaded lazily in memory because elasticsearch does not know which fields will be used in terms aggregations and which fields won’t. You can tell elasticsearch to load global ordinals eagerly at refresh-time by configuring mappings as described below:
```
PUT index
{
  "mappings": {
    "type": {
      "properties": {
        "foo": {
          "type": "keyword",
          "eager_global_ordinals": true
        }
      }
    }
  }
}
```
## 9. Warm up the filesystem cacheedit
If the machine running elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. You can explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the index.store.preload setting.

Warning
Loading data into the filesystem cache eagerly on too many indices or too many files will make search slower if the filesystem cache is not large enough to hold all the data. Use with caution.

---
参考：
- [1][Tune for search speed](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/tune-for-search-speed.html)
- [2][General recommendations](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/general-recommendations.html)

Elasticsearch优化

附件: