diff --git a/README.md b/README.md index 37d16c7..aafd0d0 100644 --- a/README.md +++ b/README.md @@ -1,1053 +1,3 @@ -<<<<<<< HEAD -# FlexSearch v0.8 (Preview) - -```bash -npm install git+https://github.com/nextapps-de/flexsearch/tree/v0.8-preview -``` - -## What's New - -- Persistent indexes support for: `IndexedDB` (Browser), `Redis`, `SQLite`, `Postgres`, `MongoDB`, `Clickhouse` -- Enhanced language customization via the new `Encoder` class -- Result Highlighting -- Query performance achieve results up to 4.5 times faster compared to the previous generation v0.7.x by also improving the quality of results -- Enhanced support for larger indexes or larger result sets -- Improved offset and limit processing achieve up to 100 times faster traversal performance through large datasets -- Support for larger In-Memory index with extended key size (the defaults maximum keystore limit is: 2^24) -- Greatly enhanced performance of the whole text encoding pipeline -- Improved indexing of numeric content (Triplets) -- Intermediate result sets and `Resolver` -- Basic Resolver: `and`, `or`, `xor`, `not`, `limit`, `offset`, `boost`, `resolve` -- Improved charset collection -- New charset preset `soundex` which further reduces memory consumption by also increasing "fuzziness" -- Performance gain when polling tasks to the index by using "Event-Loop-Caches" -- Up to 100 times faster deletion/replacement when not using the additional "fastupdate" register -- Regex Pre-Compilation (transforms hundreds of regex rules into just a few) -- Extended support for multiple tags (DocumentIndex) -- Custom Fields ("Virtual Fields") -- Custom Filter -- Custom Score Function -- Added French language preset (stop-word filter, stemmer) -- Enhanced Worker Support -- Export / Import index in chunks -- Improved Build System + Bundler (Supported: CommonJS, ESM, Global Namespace), also the import of language packs are now supported for Node.js -- Full covering index.d.ts type definitions -- Fast-Boot Serialization optimized for Server-Side-Rendering (PHP, Python, Ruby, Rust, Java, Go, Node.js, ...) - -Compare Benchmark: [0.7.0](https://nextapps-de.github.io/flexsearch/test/flexsearch-0.7.0/) vs. [0.8.0](https://nextapps-de.github.io/flexsearch/test/flexsearch-0.8.0/) - -## Persistent Indexes - -FlexSearch provides a new Storage Adapter where indexes are delegated through persistent storages. - -Supported: - -- [IndexedDB (Browser)](src/db/indexeddb/) -- [Redis](src/db/redis/) -- [SQLite](src/db/sqlite/) -- [Postgres](src/db/postgres/) -- [MongoDB](src/db/mongodb/) -- [Clickhouse](src/db/clickhouse/) - -The `.export()` and `.import()` methods are still available for non-persistent In-Memory indexes. - -<<<<<<< HEAD -When it comes to raw search speed FlexSearch outperforms every single searching library out there and also provides flexible search capabilities like multi-field search, phonetic transformations or partial matching. -======= -All search capabilities are available on persistent indexes like: -- Context-Search -- Suggestions -- Cursor-based Queries (Limit/Offset) -- Scoring (supports a resolution of up to 32767 slots) -- Document-Search -- Partial Search -- Multi-Tag-Search -- Boost Fields -- Custom Encoder -- Resolver -- Tokenizer (Strict, Forward, Reverse, Full) -- Document Store (incl. enrich results) -- Worker Threads to run in parallel -- Auto-Balanced Cache (top queries + last queries) ->>>>>>> 7755e7d (bundle pre-release) - -All persistent variants are optimized for larger sized indexes under heavy workload. Almost every task will be streamlined to run in batch/parallel, getting the most out of the selected database engine. Whereas the InMemory index can't share their data between different nodes when running in a cluster, every persistent storage can handle this by default. - -### Examples Node.js - -- [nodejs-commonjs](example/nodejs-commonjs): - - [basic](example/nodejs-commonjs/basic) - - [basic-suggestion](example/nodejs-commonjs/basic-suggestion) - - [basic-persistent](example/nodejs-commonjs/basic-persistent) - - [basic-resolver](example/nodejs-commonjs/basic-resolver) - - [basic-worker](example/nodejs-commonjs/basic-worker) - - [basic-worker-extern-config](example/nodejs-commonjs/basic-worker-extern-config) - - [basic-export-import](example/nodejs-commonjs/basic-export-import) - - [document](example/nodejs-commonjs/document) - - [document-persistent](example/nodejs-commonjs/document-persistent) - - [document-worker](example/nodejs-commonjs/document-worker) - - [document-worker-extern-config](example/nodejs-commonjs/document-worker-extern-config) - - [document-export-import](example/nodejs-commonjs/document-export-import) - - [language-pack](example/nodejs-commonjs/language-pack) -- [nodejs-esm](example/nodejs-esm): - - [basic](example/nodejs-esm/basic) - - [basic-suggestion](example/nodejs-esm/basic-suggestion) - - [basic-persistent](example/nodejs-esm/basic-persistent) - - [basic-resolver](example/nodejs-esm/basic-resolver) - - [basic-worker](example/nodejs-esm/basic-worker) - - [basic-worker-extern-config](example/nodejs-esm/basic-worker-extern-config) - - [basic-export-import](example/nodejs-esm/basic-export-import) - - [document](example/nodejs-esm/document) - - [document-persistent](example/nodejs-esm/document-persistent) - - [document-worker](example/nodejs-esm/document-worker) - - [document-worker-extern-config](example/nodejs-esm/document-worker-extern-config) - - [document-export-import](example/nodejs-esm/document-export-import) - - [language-pack](example/nodejs-esm/language-pack) - -### Examples Browser - -- [browser-legacy](example/browser-legacy): - - [basic](example/browser-legacy/basic) - - [basic-suggestion](example/browser-legacy/basic-suggestion) - - [basic-persistent](example/browser-legacy/basic-persistent) - - [basic-resolver](example/browser-legacy/basic-resolver) - - [basic-worker](example/browser-legacy/basic-worker) - - [document](example/browser-legacy/document) - - [document-highlighting](example/browser-legacy/document-highlighting) - - [document-persistent](example/browser-legacy/document-persistent) - - [document-worker](example/browser-legacy/document-worker) - - [language-pack](example/browser-legacy/language-pack) -- [browser-module](example/browser-module): - - [basic](example/browser-module/basic) - - [basic-suggestion](example/browser-module/basic-suggestion) - - [basic-persistent](example/browser-module/basic-persistent) - - [basic-resolver](example/browser-module/basic-resolver) - - [basic-worker](example/browser-module/basic-worker) - - [basic-worker-extern-config](example/browser-module/basic-worker-extern-config) - - [document](example/browser-module/document) - - [document-highlighting](example/browser-module/document-highlighting) - - [document-persistent](example/browser-module/document-persistent) - - [document-worker](example/browser-module/document-worker) - - [document-worker-extern-config](example/browser-module/document-worker-extern-config) - - [language-pack](example/browser-module/language-pack) - -```js -import FlexSearchIndex from "./index.js"; -import Database from "./db/indexeddb/index.js"; -// create an index -const index = new FlexSearchIndex(); -// create db instance with optional prefix -const db = new Database("my-store"); -// mount and await before transfering data -await flexsearch.mount(db); - -// update the index as usual -index.add(1, "content..."); -index.update(2, "content..."); -index.remove(3); - -// changes are automatically committed by default -// when you need to wait for the task completion, then you -// can use the commit method explicitely: -await index.commit(); -``` - -Alternatively mount a store by index creation: - -```js -const index = new FlexSearchIndex({ - db: new Storage("my-store") -}); - -// await for the db response before access the first time -await index.db; -// apply changes to the index -// ... -``` - -Query against a persistent storage just as usual: - -```js -const result = await index.search("gulliver"); -``` - -Auto-Commit is enabled by default and will process changes asynchronously in batch. -You can fully disable the auto-commit feature and perform them manually: - -```js -const index = new FlexSearchIndex({ - db: new Storage("my-store"), - commit: false -}); -// update the index -index.add(1, "content..."); -index.update(2, "content..."); -index.remove(3); - -// transfer all changes to the db -await index.commit(); -``` - -You can call the commit method manually also when `commit: true` option was set. - -### Benchmark - -The benchmark was measured in "terms per second". - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
StoreAddSearch 1Search NReplaceRemoveNot FoundScaling
terms per secterms per secterms per secterms per secterms per secterms per sec
IndexedDB123,29883,82362,37057,410171,053425,744No
Redis1,566,091201,534859,463117,013129,595875,526Yes
Sqlite269,81229,627129,735174,4451,406,553122,566No
Postgres354,89424,32976,189324,5463,702,64750,305Yes
MongoDB515,93819,68481,558243,353485,19267,751Yes
Clickhouse1,436,99211,50722,196931,0263,276,84716,644Yes
- -__Search 1:__ Single term query
-__Search N:__ Multi term query (Context-Search) - -The benchmark was executed against a single client. - -## Encoder - -Search capabilities highly depends on language processing. The old workflow wasn't really practicable. The new Encoder class is a huge improvement and fully replaces the encoding part. Some FlexSearch options was moved to the new `Encoder` instance. - -New Encoding Pipeline: -1. charset normalization -2. custom preparation -3. split into terms (apply includes/excludes) -4. filter (pre-filter) -5. matcher (substitute terms) -6. stemmer (substitute term endings) -7. filter (post-filter) -8. replace chars (mapper) -9. custom regex (replacer) -10. letter deduplication -11. apply finalize - -### Example - -```js -const encoder = new Encoder({ - normalize: true, - dedupe: true, - cache: true, - include: { - letter: true, - number: true, - symbol: false, - punctuation: false, - control: false, - char: "@" - } -}); -``` - -You can use an `include` __instead__ of an `exclude` definition: - -```js -const encoder = new Encoder({ - exclude: { - letter: false, - number: false, - symbol: true, - punctuation: true, - control: true - } -}); -``` - -Instead of using `include` or `exclude` you can pass a regular expression to the field `split`: - -```js -const encoder = new Encoder({ - split: /\s+/ -}); -``` - -> The definitions `include` and `exclude` is a replacement for `split`. You can just define one of those 3. - -Adding custom functions to the encoder pipeline: - -```js -const encoder = new Encoder({ - normalize: function(str){ - return str.toLowerCase(); - }, - prepare: function(str){ - return str.replace(/&/g, " and "); - }, - finalize: function(arr){ - return arr.filter(term => term.length > 2); - } -}); -``` - -Assign encoder to an index: - -```js -const index = new Index({ - encoder: encoder -}); -``` - -Define language specific transformations: - -```js -const encoder = new Encoder({ - replacer: [ - /[´`’ʼ]/g, "'" - ], - filter: new Set([ - "and", - ]), - matcher: new Map([ - ["xvi", "16"] - ]), - stemmer: new Map([ - ["ly", ""] - ]), - mapper: new Map([ - ["é", "e"] - ]) -}); -``` - -Or use predefined language and extend it with custom options: - -```js -import EnglishBookPreset from "./lang/en.js"; -const encoder = new Encoder(EnglishBookPreset, { - filter: false -}); -``` - -Equivalent: - -```js -import EnglishBookPreset from "./lang/en.js"; -const encoder = new Encoder(EnglishBookPreset); -encoder.assign({ filter: false }); -``` - -Assign extensions to the encoder instance: - -```js -import LatinEncoderPreset from "./charset/latin/simple.js"; -import EnglishBookPreset from "./lang/en.js"; -// stack definitions to the encoder instance -const encoder = new Encoder() - .assign(LatinEncoderPreset) - .assign(EnglishBookPreset) -// override preset options ... - .assign({ minlength: 3 }); -// assign further presets ... -``` - -> When adding extension to the encoder every previously assigned configuration is still intact, very much like Mixins, also when assigning custom functions. - -Add custom transformations to an existing index: - -```js -import LatinEncoderPreset from "./charset/latin/default.js"; -const encoder = new Encoder(LatinEncoderPreset); -encoder.addReplacer(/[´`’ʼ]/g, "'"); -encoder.addFilter("and"); -encoder.addMatcher("xvi", "16"); -encoder.addStemmer("ly", ""); -encoder.addMapper("é", "e"); -``` - -Shortcut for just assigning one encoder configuration to an index: - -```js -import LatinEncoderPreset from "./charset/latin/default.js"; -const index = new Index({ - encoder: LatinEncoderPreset -}); -``` - -## Resolver - -Retrieve an unresolved result: - -```js -const raw = index.search("a short query", { - resolve: false -}); -``` - -<<<<<<< HEAD - - -<<<<<<< HEAD -Library Comparison "Gulliver's Travels": -- Performance Benchmark -- Scoring Benchmark -- Memory Consumption - -Plugins (extern projects): -- React: https://github.com/angeloashmore/react-use-flexsearch -- Vue: https://github.com/Noction/vue-use-flexsearch -- Gatsby: https://www.gatsbyjs.org/packages/gatsby-plugin-flexsearch/ - -### Get Latest - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
BuildFileCDN
flexsearch.bundle.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.7.31/dist/flexsearch.bundle.js
flexsearch.light.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.7.31/dist/flexsearch.light.js
flexsearch.compact.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.7.31/dist/flexsearch.compact.js
flexsearch.es5.js *Downloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.7.31/dist/flexsearch.es5.js
ES6 ModulesDownloadThe /dist/module/ folder of this Github repository
- -* The bundle "flexsearch.es5.js" includes polyfills for EcmaScript 5 Support. - -#### Get Latest (NPM) - -```cmd -npm install flexsearch -``` - -### Compare Web-Bundles - -> The Node.js package includes all features from `flexsearch.bundle.js`. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Featureflexsearch.bundle.jsflexsearch.compact.jsflexsearch.light.js
- Presets - ✓✓-
- Async Search - ✓✓-
- Workers (Web + Node.js) - ✓--
- Contextual Indexes - ✓✓✓
- Index Documents (Field-Search) - ✓✓-
- Document Store - ✓✓-
- Partial Matching - ✓✓✓
- Relevance Scoring - ✓✓✓
- Auto-Balanced Cache by Popularity - ✓--
- Tags - ✓--
- Suggestions - ✓✓-
- Phonetic Matching - ✓✓-
Customizable Charset/Language (Matcher, Encoder, Tokenizer, Stemmer, Filter, Split, RTL)✓✓✓
Export / Import Indexes✓--
File Size (gzip)6.8 kb5.3 kb2.9 kb
- - -## Performance Benchmark (Ranking) - -Run Comparison: Performance Benchmark "Gulliver's Travels" - -Operation per seconds, higher is better, except the test "Memory" on which lower is better. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
RankLibraryMemoryQuery (Single Term)Query (Multi Term)Query (Long)Query (Dupes)Query (Not Found)
1FlexSearch177084129158685651158520171423202006
2JSii2765641581496129095098534109
3Wade424204717878016693225824213754
4JS Search1938221640341037795830167605
5Elasticlunr.js6465412757328652378613982
6BulkSearch1021306931413333326521825569
7MiniSearch24348440610945723998917624
8bm251571914297893668841823
9Lunr.js2219255271272266267
10FuzzySearch1573735338153243
11Fuse764190462123
- - -## Load Library - -There are 3 types of indexes: - -1. `Index` is a flat high performance index which stores id-content-pairs. -2. `Worker` / `WorkerIndex` is also a flat index which stores id-content-pairs but runs in background as a dedicated worker thread. -3. `Document` is multi-field index which can store complex JSON documents (could also exist of worker indexes). - -The most of you probably need just one of them according to your scenario. - -### Browser - -#### Legacy ES5 Script Tag (Bundled) - -```html - - -``` - -#### ESM/ES6 Modules: - -```html - -``` - -#### ESM/ES6 Bundled Module: - -```html - -``` - -Or via CDN: -```html - -``` - -AMD / CommonJS: - -```javascript -var FlexSearch = require("./node_modules/flexsearch/dist/flexsearch.bundle.min.js"); -``` - -### Node.js - -```npm -npm install flexsearch -``` - -In your code include as follows: -======= -======= ->>>>>>> 06878b8 (re-bundle) -You can apply and chain different resolver methods to the raw result, e.g.: ->>>>>>> 7755e7d (bundle pre-release) - -```js -raw.and( ... ) - .and( ... ) - .boost(2) - .or( ... , ... ) - .limit(100) - .xor( ... ) - .not( ... ) - // final resolve - .resolve({ - limit: 10, - offset: 0, - enrich: true - }); -``` - -The default resolver: - -```js -const raw = index.search("a short query", { - resolve: false -}); -const result = raw.resolve(); -``` - -Or use declaration style: - -```js -import Resolver from "./resolver.js"; -const raw = new Resolver({ - index: index, - query: "a short query" -}); -const result = raw.resolve(); -``` - -### Chainable Boolean Operations - -The basic concept explained: - -```js -// 1. get one or multiple unresolved results -const raw1 = index.search("a short query", { - resolve: false -}); -const raw2 = index.search("another query", { - resolve: false, - boost: 2 -}); - -// 2. apply and chain resolver operations -const raw3 = raw1.and(raw2, /* ... */); -// you can access the aggregated result by raw3.result -console.log("The aggregated result is:", raw3.result) -// apply further operations ... - -// 3. resolve final result -const result = raw3.resolve({ - limit: 100, - offset: 0 -}); -console.log("The final result is:", result) -``` - -Use inline queries: - -```js -const result = index.search("further query", { - // set resolve to false on the first query - resolve: false, - boost: 2 -}) -.or( // union - index.search("a query") - .and( // intersection - index.search("another query", { - boost: 2 - }) - ) -) -.not( // exclusion - index.search("some query") -) -// resolve the result -.resolve({ - limit: 100, - offset: 0 -}); -``` - -```js -import Resolver from "./resolver.js"; -const result = new Resolver({ - index: index, - query: "further query", - boost: 2 -}) -.or({ - and: [{ // inner expression - index: index, - query: "a query" - },{ - index: index, - query: "another query", - boost: 2 - }] -}) -.not({ // exclusion - index: index, - query: "some query" -}) -.resolve({ - limit: 100, - offset: 0 -}); -``` - -When all queries are made against the same index, you can skip the index in every declaration followed after initially calling `new Resolve()`: - -```js -import Resolver from "./resolver.js"; -const result = new Resolver({ - index: index, - query: "a query" -}) -.and({ query: "another query", boost: 2 }) -.or ({ query: "further query", boost: 2 }) -.not({ query: "some query" }) -.resolve(100); -``` -======= FlexSearch v0.8: [Overview and Migration Guide](doc/0.8.0.md)

@@ -1094,7 +44,6 @@ You can help me by making a personal donation to keep this project alive and als

->>>>>>> 2e0e190 (push v0.8 to master) - field: "custom_field", - custom: function(data){ - return "custom field content"; - } - // <------ - }] - } -}; -``` - -The configuration which needs to be available as a default export is: - -<<<<<<< HEAD -> __Note:__ This feature is disabled by default because of its extended memory usage. Read here get more information about and how to enable. - -FlexSearch introduce a new scoring mechanism called __Contextual Search__ which was invented by Thomas Wilkerling, the author of this library. A Contextual Search incredibly boost up queries to a complete new level but also requires some additional memory (depending on ___depth___). -The basic idea of this concept is to limit relevance by its context instead of calculating relevance through the whole distance of its corresponding document. -This way contextual search also improves the results of relevance-based queries on a large amount of text data. - -

- -

- - -## Enable Contextual Scoring - -Create an index and use the default context: -======= ->>>>>>> 7755e7d (bundle pre-release) -```js -{ - field: "custom_field", - custom: function(data){ - return "custom field content"; - } -}; -``` - -You're welcome to make some suggestions how to improve the handling of extern configuration. - -### Example Node.js: - -An extern configuration for one WorkerIndex, let's assume it is located in `./custom_field.js`: -```js -const { Charset } = require("flexsearch"); -const { LatinSimple } = Charset; -// it requires a default export: -module.exports = { - encoder: LatinSimple, - tokenize: "forward", - // custom function: - custom: function(data){ - return "custom field content"; - } -}; -``` - -Create Worker Index along the configuration above: -```js -const { Document } = require("flexsearch"); -const flexsearch = new Document({ - worker: true, - document: { - index: [{ - // the field name needs to be set here - field: "custom_field", - // path to your config from above: - config: "./custom_field.js", - }] - } -}); -``` - -### Browser (ESM) - -An extern configuration for one WorkerIndex, let's assume it is located in `./custom_field.js`: -```js -import { Charset } from "./dist/flexsearch.bundle.module.min.js"; -const { LatinSimple } = Charset; -// it requires a default export: -export default { - encoder: LatinSimple, - tokenize: "forward", - // custom function: - custom: function(data){ - return "custom field content"; - } -}; -``` - -Create Worker Index with the configuration above: -```js -import { Document } from "./dist/flexsearch.bundle.module.min.js"; -// you will need to await for the response! -const flexsearch = await new Document({ - worker: true, - document: { - index: [{ - // the field name needs to be set here - field: "custom_field", - // Absolute URL to your config from above: - config: "http://localhost/custom_field.js" - }] - } -}); -``` - -Here it needs the __absolute URL__, because the WorkerIndex context is from type `Blob` and you can't use relative URLs starting from this context. - -### Test Case - -As a test the whole IMDB data collection was indexed, containing of: - -JSON Documents: 9,273,132
-Fields: 83,458,188
-Tokens: 128,898,832
- -The used index configuration has 2 fields (using bidirectional context of `depth: 1`), 1 custom field, 2 tags and a full datastore of all input json documents. - -A non-Worker Document index requires 181 seconds to index all contents.
-The Worker index just takes 32 seconds to index them all, by processing every field and tag in parallel. For such large content it is a quite impressive result. - -### CSP-friendly Worker (Browser) - -When just using worker by passing the option `worker: true`, the worker will be created by code generation under the hood. This might have issues when using strict CSP settings. - -You can overcome this issue by passing the filepath to the worker file like `worker: "./worker.js"`. The original worker file is located at `src/worker/worker.js`. - -## Fuzzy-Search - -Fuzzysearch describes a basic concept of how making queries more tolerant. FlexSearch provides several methods to achieve fuzziness: - -1. Use a tokenizer: `forward`, `reverse` or `full` -2. Don't forget to use any of the builtin encoder `simple` > `balance` > `advanced` > `extra` > `soundex` (sorted by fuzziness) -3. Use one of the language specific presets e.g. `/lang/en.js` for en-US specific content -4. Enable suggestions by passing the search option `suggest: true` - -Additionally, you can apply custom `Mapper`, `Replacer`, `Stemmer`, `Filter` or by assigning a custom `normalize(str)`, `prepare(str)` or `finalize(arr)` function to the Encoder. - -### Compare Fuzzy-Search Encoding - -Original term which was indexed: "Struldbrugs" - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Encoder:LatinExactLatinDefaultLatinSimpleLatinBalanceLatinAdvancedLatinExtraLatinSoundex
Index Size3.1 Mb1.9 Mb1.8 Mb1.7 Mb1.6 Mb1.1 Mb0.7 Mb
Struldbrugs✓✓✓✓✓✓✓
struldbrugs✓✓✓✓✓✓
strũldbrųĝgs✓✓✓✓✓
strultbrooks✓✓✓✓
shtruhldbrohkz✓✓✓
zdroltbrykz✓✓
struhlbrogger✓
- -The index size was measured after indexing the book "Gulliver's Travels". - -### Custom Encoder - -Since it is very simple to create a custom Encoder, you are welcome to create your own. -e.g. -```js -function customEncoder(content){ - const tokens = []; - // split content into terms/tokens - // apply your changes to each term/token - // you will need to return an Array of terms/tokens - // so just iterate through the input string and - // push tokens to the array - // ... - return tokens; -} - -const index = new Index({ - // set to strict when your tokenization was already done - tokenize: "strict", - encode: customEncoder -}); -``` - -If you get some good results please feel free to share your encoder. - -## Fast-Boot Serialization for Server-Side-Rendering (PHP, Python, Ruby, Rust, Java, Go, Node.js, ...) - -> This is an experimental feature with limited support which probably might drop in future release. You're welcome to give some feedback. - -When using Server-Side-Rendering you can create a different export which instantly boot up. Especially when using Server-side rendered content, this could help to restore a __static__ index on page load. Document-Indexes aren't supported yet for this method. - -> When your index is too large you should use the default export/import mechanism. - -As the first step populate the FlexSearch index with your contents. - -You have two options: - -### 1. Create a function as string - -```js -const fn_string = index.serialize(); -``` - -The contents of `fn_string` is a valid Javascript-Function declared as `inject(index)`. Store it or place this somewhere in your code. - -This function basically looks like: - -```js -function inject(index){ - index.reg = new Set([/* ... */]); - index.map = new Map([/* ... */]); - index.ctx = new Map([/* ... */]); -} -``` - -You can save this function by e.g. `fs.writeFileSync("inject.js", fn_string);` or place it as string in your SSR-generated markup. - -After creating the index on client side just call the inject method like: - -```js -const index = new Index({/* use same configuration! */}); -inject(index); -``` - -That's it. - -> You'll need to use the same configuration as you used before the export. Any changes on the configuration needs to be re-indexed. - -### 2. Create just a function body as string - -Alternatively you can use lazy function declaration by passing `false` to the serialize function: - -```js -const fn_body = index.serialize(false); -``` - -You will get just the function body which looks like: - -```js -index.reg = new Set([/* ... */]); -index.map = new Map([/* ... */]); -index.ctx = new Map([/* ... */]); -``` - -Now you can place this in your code directly (name your index as `index`), or you can also create an inject function from it, e.g.: - -```js -const inject = new Function("index", fn_body); -``` - -This function is callable like the above example: - -```js -const index = new Index(); -inject(index); -``` -======= Extern Projects & Plugins: - React: https://github.com/angeloashmore/react-use-flexsearch - Vue: https://github.com/Noction/vue-use-flexsearch - Gatsby: https://www.gatsbyjs.org/packages/gatsby-plugin-flexsearch/ ->>>>>>> 2e0e190 (push v0.8 to master) ## Load Library (Node.js, ESM, Legacy Browser)