| Option | Values | Description | Default |
| You can just choose one of those 3 options: | |||
include |
Encoder Split Options | Define which of the string contents should be included (inclusion properties defaults to false) | { letter: true, number: true } |
exclude |
Encoder Split Options | Define which of the string contents should be excluded (exclusion properties defaults to true) | false |
split |
false RegExp String Encoder Split Options |
The expression used to split the content into terms | → include { letter: true, number: true } |
| Other options: | |||
dedupe |
Boolean | Deduplicate consecutive letters, e.g. "missing" to "mising" | true |
numeric |
Boolean | By default, the extended numeric support (Triplets) inherits from chosen Encoder Split Options. You probably might want to disable Triplets to get a more exact result (fewer entries) in some cases. | true |
minlength |
Number | Set the minimum term length which should be added to the index. This limit does not apply to the forward tokenizer. You still get results when just typing "f" on a term "flexsearch" when e.g. minlength: 4 was used. |
1 |
maxlength |
Number | Set the maximum term length which should be added to the index. Larger content will drop. | 1 |
rtl |
Boolean | Force Right-To-Left encoding (you should just apply this when the string content was not already encoded as RTL) | false |
normalize |
true enable normalization (default)false disable normalizationfunction(str) => str custom function
|
The normalization stage will apply basic charset normalization e.g. by replacing "é" to "e" | true |
prepare |
function(str) => str custom function
|
The preparation stage is a custom function direct followed when normalization was done | false |
finalize |
function([str]) => [str] custom function
|
The finalization stage is a custom function executed at the last task in the encoding pipeline (here it gets an array of tokens and need to return an array of tokens) | false |
filter |
Set(["and", "to", "be"])function(str) => bool custom function
encoder.addFilter("and")
|
Stop-word filter is like a blacklist of words to be filtered out from indexing at all (e.g. "and", "to" or "be"). This is also very useful when using Context Search | false |
stemmer |
Map([["ing", ""], ["ies", "y"]])
encoder.addStemmer("ing", "")
|
Stemmer will normalize several linguistic mutations of the same word (e.g. "run" and "running", or "property" and "properties"). This is also very useful when using Context Search | false |
mapper |
Map([["é", "e"], ["ß", "ss"]])
encoder.addMapper("é", "e")
|
Mapper will replace a single char (e.g. "é" into "e") | false |
matcher |
Map([["and", "&"], ["usd", "$"]])
encoder.addMatcher("and", "&")
|
Matcher will do same as Mapper but instead of single chars it will replace char sequences | false |
replacer |
[/[^a-z0-9]/g, "", /([^aeo])h(.)/g, "$1$2"])
encoder.addReplacer(/[^a-z0-9]/g, "")
|
Replacer takes custom regular expressions and couldn't get optimized in the same way as Mapper or Matcher. You should take this as the last option when no other replacement can do the same. | false |
cache |
Boolean | In some very rare situations (large consecutive content with high cardinality) it might be useful to disable the internal event-loop-cache | true |
| Option | Values | Description | Default |
letter |
Boolean | Toggle inclusion of letters on/off | true |
number |
Boolean | Toggle inclusion of numerics on/off | true |
symbol |
Boolean | Toggle inclusion of symbols on/off | false |
punctuation |
Boolean | Toggle inclusion of punctuation on/off | false |
control |
Boolean | Toggle inclusion of control chars on/off | false |
char |
String Array[String] |
Toggle inclusion of specific chars on/off | false |
## Right-To-Left Support
> [!NOTE]
> When a string is already encoded/interpreted as Right-To-Left you didn't need to use that. This option is just useful, when the source content wasn't encoded as RTL.
Just set the property `rtl: true` when creating the `Encoder`:
```js
const encoder = new Encoder({ rtl: true });
```
## CJK Word Break (Chinese, Japanese, Korean)
```js
const index = new Index({ encoder: Charset.CJK });
index.add(0, "一个单词");
var results = index.search("单词");
```
## Built-In Language Packs
- English: `en`
- German: `de`
- French: `fr`
### Import Language Packs: ES6 Modules
The most simple way to assign charset/language specific encoding via modules is:
```js
import EnglishPreset from "flexsearch/lang/en";
const index = Index({
charset: EnglishPreset
});
```
You can stack up and combine multiple presets:
```js
import { Charset } from "flexsearch";
import EnglishPreset from "flexsearch/lang/en";
const index = Index({
charset: new Encoder(
Charset.LatinAdvanced,
EnglishPreset,
{ minlength: 3 }
)
});
```
You can also assign the encoder preset directly:
```js
const index = Index({
encoder: Charset.Default
});
```
#### Import Language Packs: ES5 Legacy Browser
When loading language packs, make sure that the library was loaded before:
```html
```
The language packs are registered on `FlexSearch.Language`:
```js
const index = FlexSearch.Index({
encoder: FlexSearch.Language["en"]
});
```
You can stack up and combine multiple presets:
```js
const index = FlexSearch.Index({
charset: new FlexSearch.Encoder(
FlexSearch.Charset.LatinAdvanced,
FlexSearch.Language["en"],
{ minlength: 3 }
)
});
```
#### Import Language Packs: Node.js
In Node.js all built-in language packs files are available by its scope:
```js
const EnglishPreset = require("flexsearch/lang/en");
const index = new Index({
encoder: EnglishPreset
});
```
### Share Encoders
Assigning the `Encoder` instance to the top level configuration will share the encoder to all fields. You should avoid this when contents of fields don't have the same type of content (e.g. one field contains terms, another contains numeric IDs).
Sharing the encoder can improve encoding efficiency and memory allocation, but when not properly used also has negative effect to the performance. You can share encoders to any type of index, also through multiple instances of indexes (also documents).
You should group similar types of contents to one encoder respectively. When you have different content types then define one for each of them.
In this example there are two Document-Indexes for two different documents "orders" and "billings". You can also share encoder to different fields of just one document.
```js
// usual term encoding
const encoder_terms = Encoder(
Charset.LatinAdvanced,
// just add letters (no numbers)
{ include: { letter: true } }
);
// numeric encoding
const encoder_numeric = new Encoder(Charset.Default);
const orders = Document({
document: {
id: "id",
index: [{
field: "product_title",
encoder: encoder_terms
},{
field: "product_details",
encoder: encoder_terms
},{
field: "order_date",
encoder: encoder_numeric
},{
field: "customer_id",
encoder: encoder_numeric
}]
}
});
const billings = Document({
document: {
id: "id",
index: [{
field: "product_title",
encoder: encoder_terms
},{
field: "product_content",
encoder: encoder_terms
},{
field: "billing_date",
encoder: encoder_numeric
},{
field: "customer_id",
encoder: encoder_numeric
}]
}
});
```
### Merge Documents
When you have multiple document types (indexed by multiple indexes) but some of the data has same fields (like in the example above) and you can refer them by any identifier or key, you should consider merging those documents into one. This will hugely improve index size.
E.g. when you merge "orders" and "billings" from example above by ID, then you can use just one index:
```js
const encoder_terms = Encoder(
Charset.LatinAdvanced,
// just add letters (no numbers)
{ include: { letter: true } }
);
const encoder_numeric = new Encoder(
Charset.Default
);
const merged = Document({
document: {
id: "id",
index: [{
field: "product_title",
encoder: encoder_terms
},{
field: "product_details",
encoder: encoder_terms
},{
field: "order_date",
encoder: encoder_numeric
},{
field: "billing_date",
encoder: encoder_numeric
},{
field: "customer_id",
encoder: encoder_numeric
}]
}
});
```