Elasticsearch 5.4.3搜索性能调优

2020-12-18 09:02:09 编辑 0 阅读量:1178

关于Elasticsearch的性能调优网上有很多说明，但是最靠谱的就是官网[Tune for search speed](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/tune-for-search-speed.html)

个人理解实战如下：

## 一、硬件上
1. 要有足够的内存，用来给Elasticsearch（jvm）和lucene（系统文件缓存）使用，Elasticsearch主要处理filter搜索缓存，聚合缓存等，lucene处理match等检索，涉及到segment的打开和缓存：
    - Elasticsearch的内存不要超过32G，因为jvm超过32G，内存指针长度会增加，造成40G实际使用量还比不上31G的
	- 要预留一般的内容供lucene使用
2. 硬盘速度足够快，能用ssd最好

## 二、个人配置
1. 单个document不要过大
2. 索引的内容不能太稀疏，这个是lucene本身的机制决定的 -> 也就是说一个索引下多个doc_type最好是mappings类似，其实在`6.*`中，一个index只能有一个doc_type
3. 在keyword建立全局序号（这个可能会影响index的速度），就是`warm_up_global_ordinals`：
```
PUT your_index
{
  "mappings": {
    "your_doc_type": {
      "properties": {
        "your_field": {
          "type": "keyword",
          "eager_global_ordinals": true  # 建立全局序号，并提前预加载
        }
      }
    }
  }
}
```

4. 将索引段提前加载到文件系统内存中, `warm_up_the_filesystem_cache`:
```
PUT your_index/_settings
{
  "index.store.preload": ["nvd", "dvd", "tim", "doc", "dim"]  # 内存不够的话优先nvd和dvd
}
```

5. 设置refresh_interval
refresh_interval影响的是数据添加Elasticsearch后，多长时间能被检索，延长这个时间可以减小Elasticsearch的压力

6. 强制合并段
```
POST your_index/_forcemerge
{
  "max_num_segments": 1
}
```

## 三、使用中
1. keyword的合理的mappings
有些keyword不要设置成numeric，如果某个段用一个长数字来表面身份，然后很多文档也用该数字，那么这个数字设为keyword更好，因为Elasticsearch可以用全局序号来表示这个长数字，可以有效节约存储空间，而numeric本身就作为一个全局序号，没办法合并。
>The fact that some data is numeric does not mean it should always be mapped as a numeric field. Typically, fields storing identifiers such as an ISBN or any number identifying a record from another database, might benefit from being mapped as keyword rather than integer or long.
2. 范围域的聚合可以提前设置好范围
比如，年龄10-20， 20-30，与其用range来聚合，还不如另增一个字段，值为"10-20", "20-30"这样就可以用terms聚合代替range聚合
3. 尽量避免脚本，要也是用`painless`和`expressions`
4. 不要返回太大的结果集
Elasticsearch搜索过程是将查询条件分配到各个节点，然后根据设置的返回大小返回top n的查询结果，然后再排序，返回一定数量的文档。如果设置了很大的返回结果集，或者翻页（ofset）很大，那么从各个节点取回的文档数量也会剧增，到处网络io变大，cpu占用增加，查询变慢。所以，翻页过大要严格禁止。

## 四、一些说明
1. fielddata和inverted indices
Fielddata exists because inverted indices are efficient only for certain operations. The inverted index excels((("inverted index", "fielddata versus"))) at finding documents that contain a term. It does not perform well in the opposite direction: determining which terms exist in a single document. Aggregations need this secondary access pattern.
Fielddata is not just used for aggregations.((("fielddata", "uses other than aggregations"))) It is required for any operation that needs to look up the value contained in a specific document. Besides aggregations, this includes sorting, scripts that access field values, parent-child relationships (see <>), and certain types of queries or filters, such as the <> filter.
所以，也就是说Elasticsearch处理fielddata（正排索引），lucene处理inverted indices（倒排索引）
2. 添加慢查询的log记录，便于分析

---
参考：
- [1][Tune for search speed](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/tune-for-search-speed.html#_warm_up_global_ordinals)
- [2][Pre-loading data into the file system cache](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/_pre_loading_data_into_the_file_system_cache.html)
- [3][Hardware--很有用](https://es.xiaoleilu.com/510_Deployment/20_hardware.html)
- [4][预加载 fielddata--已过时](https://www.elastic.co/guide/cn/elasticsearch/guide/current/preload-fielddata.html)
- [5][ElasticSearch 预加载(preloading fileddata)--已过时](http://blog.csdn.net/sjzxlpp/article/details/66975040)
- [6][预加载数据到文件系统缓存](http://cwiki.apachecn.org/pages/viewpage.action?pageId=10028674)
- [7][Elasticsearch 5.x 性能](http://blog.csdn.net/it_lihongmin/article/details/78770540)

Elasticsearch 5.4.3搜索性能调优

附件: