diff --git a/README.md b/README.md index c41f629..19f73c6 100644 --- a/README.md +++ b/README.md @@ -1,57 +1,388 @@ - -FlexSearch v0.8-Preview: https://github.com/nextapps-de/flexsearch/tree/v0.8-preview +# FlexSearch v0.8 (Preview) -Let's discuss the upcoming FlexSearch v0.8 here: https://github.com/nextapps-de/flexsearch/discussions/415 -

-

-

- FlexSearch.js: Next-Generation full text search library for Browser and Node.js -

-

-

Web's fastest and most memory-flexible full-text search library with zero dependencies.

+## What's New - - - - +- Persistent indexes support for: `IndexedDB` (Browser), `Redis`, `SQLite`, `Postgres`, `MongoDB`, `Clickhouse` +- Enhanced language customization via the new `Encoder` class +- Searching single terms is up to 7 times faster, the overall benchmark score was doubled +- Enhanced support for larger indexes or larger result sets +- Improved offset and limit processing achieve up to 100 times faster traversal performance through large datasets +- Support for larger In-Memory index with extended key size (the defaults maximum keystore limit is: 2^24) +- Greatly enhanced performance of the whole text encoding pipeline +- Improved indexing of numeric content (Triplets) +- Intermediate result sets and `Resolver` +- Basic Resolver: `and`, `or`, `xor`, `not`, `limit`, `offset`, `enrich`, `resolve`, Output formatter +- Improved charset collection +- New charset preset `soundex` which further reduces memory consumption by also increasing "fuzziness" +- Performance gain when polling tasks to the index by using "Event-Loop-Caches" +- Up to 100 times faster deletion/replacement when not using the additional "fastupdate" register +- Regex Pre-Compilation (transforms hundreds of regex rules into just a few) +- Extended support for multiple tags (DocumentIndex) +- Custom Fields ("Virtual Fields") +- Custom Filter +- Custom Score Function +- Added French language preset (stop-word filter, stemmer) +- Enhanced Worker Support +- Improved Build System + Bundler (Supported: CommonJS, ESM, Global Namespace) +- Full covering index.d.ts type definitions -Basic Start  •  API Reference  •  Document Indexes  •  Using Worker  •  Changelog +Compare Benchmark: [0.7.0](https://nextapps-de.github.io/flexsearch/test/flexsearch-0.7.0/) vs. [0.8.0](https://nextapps-de.github.io/flexsearch/test/flexsearch-0.8.0/) -## Support this Project +## Persistent Indexes -You can help me by making a personal donation to keep this project alive and also to provide all the contribution to solve your needs. +FlexSearch provides a new Storage Adapter where indexes are delegated through persistent storages. -Donate using Open Collective -Donate using Github Sponsors -Donate using Liberapay -Donate using Patreon -Donate using Bountysource -Donate using PayPal +Supported: -### FlexSearch Sponsors +- [IndexedDB (Browser)](db/indexeddb/) +- [Redis](db/redis/) +- [SQLite](db/sqlite/) +- [Postgres](db/postgres/) +- [MongoDB](db/mongo/) +- [Clickhouse](db/clickhouse/) - -
- Donate using Open Collective
- Antithesis Operations LLC -
- -

+The `.export()` and `.import()` methods are still available for non-persistent In-Memory indexes. +<<<<<<< HEAD When it comes to raw search speed FlexSearch outperforms every single searching library out there and also provides flexible search capabilities like multi-field search, phonetic transformations or partial matching. +======= +All search capabilities are available on persistent indexes like: +- Context-Search +- Suggestions +- Cursor-based Queries (Limit/Offset) +- Scoring (supports a resolution of up to 32767 slots) +- Document-Search +- Partial Search +- Multi-Tag-Search +- Boost Fields +- Custom Encoder +- Resolver +- Tokenizer (Strict, Forward, Reverse, Full) +- Document Store (incl. enrich results) +- Worker Threads to run in parallel +- Auto-Balanced Cache (top queries + last queries) +>>>>>>> 7755e7d (bundle pre-release) -Depending on the used options it also provides the most memory-efficient index. FlexSearch introduce a new scoring algorithm called "contextual index" based on a pre-scored lexical dictionary architecture which actually performs queries up to 1,000,000 times faster compared to other libraries. -FlexSearch also provides you a non-blocking asynchronous processing model as well as web workers to perform any updates or queries on the index in parallel through dedicated balanced threads. +All persistent variants are optimized for larger sized indexes under heavy workload. Almost every task will be streamlined to run in batch/parallel, getting the most out of the selected database engine. Whereas the InMemory index can't share their data between different nodes when running in a cluster, every persistent storage can handle this by default. -Supported Platforms: -- Browser -- Node.js +### Example + +```js +import FlexSearchIndex from "./index.js"; +import Database from "./db/indexeddb/index.js"; +// create an index +const index = new FlexSearchIndex(); +// create db instance with optional prefix +const db = new Database("my-store"); +// mount and await before transfering data +await flexsearch.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +Alternatively mount a store by index creation: + +```js +const index = new FlexSearchIndex({ + db: new Storage("my-store") +}); + +// await for the db response before access the first time +await index.db; +// apply changes to the index +// ... +``` + +Query against a persistent storage just as usual: + +```js +const result = await index.search("gulliver"); +``` + +Auto-Commit is enabled by default and will process changes asynchronously in batch. +You can fully disable the auto-commit feature and perform them manually: + +```js +const index = new FlexSearchIndex({ + db: new Storage("my-store"), + commit: false +}); +// update the index +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// transfer all changes to the db +await index.commit(); +``` + +You can call the commit method manually also when `commit: true` option was set. + +### Benchmark + +The benchmark was measured in "terms per second". + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
StoreAddSearch 1Search NReplaceRemoveNot FoundScaling
terms per secterms per secterms per secterms per secterms per secterms per sec
IndexedDB123,29883,82362,37057,410171,053425,744No
Redis1,566,091201,534859,463117,013129,595875,526Yes
Sqlite269,81229,627129,735174,4451,406,553122,566No
Postgres354,89424,32976,189324,5463,702,64750,305Yes
MongoDB515,93819,68481,558243,353485,19267,751Yes
Clickhouse1,436,99211,50722,196931,0263,276,84716,644Yes
+ +__Search 1:__ Single term query
+__Search N:__ Multi term query (Context-Search) + +The benchmark was executed against a single client. + +## Encoder + +Search capabilities highly depends on language processing. The old workflow wasn't really practicable. The new Encoder class is a huge improvement and fully replaces the encoding part. Some FlexSearch options was moved to the new `Encoder` instance. + +New Encoding Pipeline: +1. charset normalization +2. custom preparation +3. split into terms (apply includes/excludes) +4. filter (pre-filter) +5. matcher (substitute terms) +6. stemmer (substitute term endings) +7. filter (post-filter) +8. replace chars (mapper) +9. custom regex (replacer) +10. letter deduplication +11. apply finalize + +### Example + +```js +const encoder = new Encoder({ + normalize: true, + dedupe: true, + cache: true, + include: { + letter: true, + number: true, + symbol: false, + punctuation: false, + control: false, + char: "@" + } +}); +``` + +```js +const encoder = new Encoder({ + normalize: function(str){ + return str.toLowerCase(); + }, + prepare: function(str){ + return str.replace(/&/g, " and "); + }, + exclude: { + letter: false, + number: false, + symbol: true, + punctuation: true, + control: true + } +}); +``` + +Define language specific transformations: + +```js +const encoder = new Encoder({ + replacer: [ + /[´`’ʼ]/g, "'" + ], + filter: new Set([ + "and", + ]), + matcher: new Map([ + ["xvi", "16"] + ]), + stemmer: new Map([ + ["ly", ""] + ]), + mapper: new Map([ + ["é", "e"] + ]) +}); +``` + +Or use predefined language and add custom options: + +```js +import EnglishBookPreset from "./lang/en.js"; +const encoder = new Encoder({ + assign: EnglishBookPreset, + filter: false +}); +``` + +Equivalent: + +```js +import EnglishBookPreset from "./lang/en.js"; +const encoder = new Encoder(EnglishBookPreset); +encoder.assign({ filter: false }); +``` + +Assign extensions to the encoder instance: + +```js +import LatinEncoder from "./lang/latin/simple.js"; +import EnglishBookPreset from "./lang/en.js"; +// stack definitions to the encoder instance +const encoder = new Encoder() + .assign(LatinEncoder) + .assign(EnglishBookPreset) +// override preset options ... + .assign({ minlength: 3 }); +// assign further presets ... +``` + +Add custom transformations to an existing index: + +```js +import LatinEncoder from "./lang/latin/default.js"; +const encoder = new Encoder(LatinEncoder); +encoder.addReplacer(/[´`’ʼ]/g, "'"); +encoder.addFilter("and"); +encoder.addMatcher("xvi", "16"); +encoder.addStemmer("ly", ""); +encoder.addMapper("é", "e"); +``` + +## Resolver + +Retrieve an unresolved result: + +```js +const raw = index.search("a short query", { + resolve: false +}); +``` +<<<<<<< HEAD Library Comparison "Gulliver's Travels": - Performance Benchmark - Scoring Benchmark @@ -475,1715 +806,694 @@ npm install flexsearch ``` In your code include as follows: +======= +You can apply and chain different resolver methods to the raw result, e.g.: +>>>>>>> 7755e7d (bundle pre-release) ```js -const { Index, Document, Worker } = require("flexsearch"); - -const index = new Index(options); -const document = new Document(options); -const worker = new Worker(options); +raw.and( ... ) + .and( ... ) + .boost(2) + .or( ... , ... ) + .limit(100) + .xor( ... ) + .not( ... ) + // final resolve + .resolve({ + limit: 10, + offset: 0, + enrich: true + }); ``` -Or: +The default resolver: + +```js +const raw = index.search("a short query", { + resolve: false +}); +const result = raw.resolve(); ``` -## Basic Usage and Variants +Or use declaration style: ```js -index.add(id, text); -index.search(text); -index.search(text, limit); -index.search(text, options); -index.search(text, limit, options); -index.search(options); +import Resolver from "./resolver.js"; +const raw = new Resolver({ + index: index, + query: "a short query" +}); +const result = raw.resolve(); ``` +### Chainable Boolean Operations + +The basic concept explained: + + +```js +// 1. get one or multiple unresolved results +const raw1 = index.search("a short query", { + resolve: false +}); +const raw2 = index.search("another query", { + resolve: false, + boost: 2 +}); + +// 2. apply and chain resolver operations +const raw3 = raw1.and(raw2, /* ... */); +// you can access the aggregated result by raw3.result +console.log("The aggregated result is:", raw3.result) +// apply further operations ... + +// 3. resolve final result +const result = raw3.resolve({ + limit: 100, + offset: 0 +}); +console.log("The final result is:", result) ``` +Use inline queries: + + ```js -var index = new Index(); -``` - -Create a new index and choosing one of the presets: - -```js -var index = new Index("performance"); -``` - -Create a new index with custom options: - -```js -var index = new Index({ - charset: "latin:extra", - tokenize: "reverse", - resolution: 9 +const result = index.search("further query", { + // set resolve to false on the first query + resolve: false, + boost: 2 +}) +.or( // union + index.search("a query") + .and( // intersection + index.search("another query", { + boost: 2 + }) + ) +) +.not( // exclusion + index.search("some query") +) +// resolve the result +.resolve({ + limit: 100, + offset: 0 }); ``` -Create a new index and extend a preset with custom options: + +Or use a fully declarative style (also recommended when run in parallel): + +```js +import Resolver from "./resolver.js"; +const result = new Resolver({ + index: index, + query: "further query", + boost: 2 +}) +.or({ + and: [{ // inner expression + index: index, + query: "a query" + },{ + index: index, + query: "another query", + boost: 2 + }] +}) +.not({ // exclusion + index: index, + query: "some query" +}) +.resolve({ + limit: 100, + offset: 0 +}); +``` + +When all queries are made against the same index, you can skip the index in every declaration followed after initially calling `new Resolve()`: + +```js +import Resolver from "./resolver.js"; +const result = new Resolver({ + index: index, + query: "a query" +}) +.and({ query: "another query", boost: 2 }) +.or ({ query: "further query", boost: 2 }) +.not({ query: "some query" }) +.resolve(100); +``` + +### Custom Result Decoration + +```js +import highlight from "./resolve/highlight.js"; +import collapse from "./resolve/collapse.js"; +const raw = index.search("a short query", { + resolve: false +}); +// resolve result for display +const template = highlight(raw, { + wrapper: "", + item: "
  • $1
    • ", + text: "$1", + highlight: "$1" +}); +document.body.appendChild(template); +// resolve result for further processing +const result = collapse(raw); +``` + +Alternatively: + +```js +const template = highlight(raw, { + wrapper: function(){ + const wrapper = document.createElement("ul"); + return wrapper; + }, + item: function(wrapper){ + const item = document.createElement("li"); + wrapper.append(item); + }, + text: function(item, content){ + const node = document.createTextNode(content); + item.append(node); + }, + highlight: function(item, content){ + const node = document.createElement("b"); + node.textContent = content; + item.append(node); + } +}); +document.body.appendChild(template); +``` + +### Custom Resolver + +```js +function CustomResolver(raw){ + // console.log(raw) + let output; + // generate output ... + return output; +} + +const result = index.search("a short query", { + resolve: CustomResolver +}); +``` + +## Big In-Memory Keystores + +The default maximum keystore limit for the In-Memory index is 2^24 of distinct terms/partials being stored (so-called "cardinality"). An additional register could be enabled and is dividing the index into self-balanced partitions. + +```js +const index = new FlexSearchIndex({ + // e.g. set keystore range to 8-Bit: + // 2^8 * 2^24 = 2^32 keys total + keystore: 8 +}); +``` + +You can theoretically store up to 2^88 keys (64-Bit address range). + +The internal ID arrays scales automatically when limit of 2^31 has reached by using Proxy. + +> Persistent storages has no keystore limit by default. You should not enable keystore when using persistent indexes, as long as you do not stress the buffer too hard before calling `index.commit()`. + + + +## Multi-Tag-Search + +Assume this document schema (a dataset from IMDB): +```js +{ + "tconst": "tt0000001", + "titleType": "short", + "primaryTitle": "Carmencita", + "originalTitle": "Carmencita", + "isAdult": 0, + "startYear": "1894", + "endYear": "", + "runtimeMinutes": "1", + "genres": [ + "Documentary", + "Short" + ] +} +``` + +An appropriate document descriptor could look like: +```js +import LatinEncoder from "./lang/latin/simple.js"; + +const flexsearch = new Document({ + encoder: new Encoder(LatinEncoder), + resolution: 3, + document: { + id: "tconst", + //store: true, // document store + index: [{ + field: "primaryTitle", + tokenize: "forward" + },{ + field: "originalTitle", + tokenize: "forward" + }], + tag: [ + "startYear", + "genres" + ] + } +}); +``` +The field contents of `primaryTitle` and `originalTitle` are encoded by the forward tokenizer. The field contents of `startYear` and `genres` are added as tags. + + + +Get all entries of a specific tag: +```js +const result = flexsearch.search({ + //enrich: true, // enrich documents + tag: { "genres": "Documentary" }, + limit: 1000, + offset: 0 +}); +``` + +Get entries of multiple tags (intersection): +```js +const result = flexsearch.search({ + //enrich: true, // enrich documents + tag: { + "genres": ["Documentary", "Short"], + "startYear": "1894" + } +}); +``` + +Combine tags with queries (intersection): +```js +const result = flexsearch.search({ + query: "Carmen", // forward tokenizer + tag: { + "genres": ["Documentary", "Short"], + "startYear": "1894" + } +}); +``` + +Alternative declaration: +```js +const result = flexsearch.search("Carmen", { + tag: [{ + field: "genres", + tag: ["Documentary", "Short"] + },{ + field: "startYear", + tag: "1894" }] }); ``` -Field options gets inherited when also global options was passed, e.g.: +## Filter Fields (Index / Tags / Datastore) ```js -const index = new Document({ - tokenize: "strict", - optimize: true, - resolution: 9, +const flexsearch = new Document({ document: { id: "id", - index:[{ - field: "title", - tokenize: "forward" - },{ - field: "content", - minlength: 3, - context: { - depth: 1, - resolution: 3 + index: [{ + // custom field: + field: "somefield", + filter: function(data){ + // return false to filter out + // return anything else to keep + return true; + } + }], + tag: [{ + field: "city", + filter: function(data){ + // return false to filter out + // return anything else to keep + return true; + } + }], + store: [{ + field: "anotherfield", + filter: function(data){ + // return false to filter out + // return anything else to keep + return true; } }] } }); ``` -Note: The context options from the field "content" also gets inherited by the corresponding field options, whereas this field options was inherited by the global option. - -### Nested Data Fields (Complex Objects) - -Assume the document array looks more complex (has nested branches etc.), e.g.: - -```json -{ - "record": { - "id": 0, - "title": "some title", - "content": { - "header": "some text", - "footer": "some text" - } - } -} -``` - -Then use the colon separated notation `root:child:child` to define hierarchy within the document descriptor: - -```js -const index = new Document({ - document: { - id: "record:id", - index: [ - "record:title", - "record:content:header", - "record:content:footer" - ] - } -}); -``` -> Just add fields you want to query against. Do not add fields to the index, you just need in the result (but did not query against). For this purpose you can store documents independently of its index (read below). - -When you want to query through a field you have to pass the exact key of the field you have defined in the `doc` as a field name (with colon syntax): - -```js -index.search(query, { - index: [ - "record:title", - "record:content:header", - "record:content:footer" - ] -}); -``` - -Same as: - -```js -index.search(query, [ - "record:title", - "record:content:header", - "record:content:footer" -]); -``` - -Using field-specific options: - -```js -index.search([{ - field: "record:title", - query: "some query", - limit: 100, - suggest: true -},{ - field: "record:title", - query: "some other query", - limit: 100, - suggest: true -}]); -``` - -You can perform a search through the same field with different queries. - -> When passing field-specific options you need to provide the full configuration for each field. They get not inherited like the document descriptor. - -### Complex Documents - -You need to follow 2 rules for your documents: - -1. The document cannot start with an Array at the root index. This will introduce sequential data and isn't supported yet. See below for a workaround for such data. - -```js -[ // <-- not allowed as document start! - { - "id": 0, - "title": "title" - } -] -``` - -2. The id can't be nested inside an array (also none of the parent fields can't be an array). This will introduce sequential data and isn't supported yet. See below for a workaround for such data. +## Custom Fields (Index / Tags / Datastore) +Dataset example: ```js { - "records": [ // <-- not allowed when ID or tag lives inside! - { - "id": 0, - "title": "title" - } - ] + "id": 10001, + "firstname": "John", + "lastname": "Doe", + "city": "Berlin", + "street": "Alexanderplatz", + "number": "1a", + "postal": "10178" } ``` -Here an example for a supported complex document: - -```json -{ - "meta": { - "tag": "cat", - "id": 0 - }, - "contents": [ - { - "body": { - "title": "some title", - "footer": "some text" - }, - "keywords": ["some", "key", "words"] - }, - { - "body": { - "title": "some title", - "footer": "some text" - }, - "keywords": ["some", "key", "words"] - } - ] -} -``` - -The corresponding document descriptor (when all fields should be indexed) looks like: - +You can apply custom fields derived from data or by anything else: ```js -const index = new Document({ +const flexsearch = new Document({ document: { - id: "meta:id", - tag: "meta:tag", - index: [ - "contents[]:body:title", - "contents[]:body:footer", - "contents[]:keywords" - ] - } -}); -``` - -Again, when searching you have to use the same colon-separated-string from your field definition. - -```js -index.search(query, { - index: "contents[]:body:title" -}); -``` - -### Not Supported Documents (Sequential Data) - -This example breaks both rules from above: - -```js -[ // <-- not allowed as document start! - { - "tag": "cat", - "records": [ // <-- not allowed when ID or tag lives inside! - { - "id": 0, - "body": { - "title": "some title", - "footer": "some text" - }, - "keywords": ["some", "key", "words"] - }, - { - "id": 1, - "body": { - "title": "some title", - "footer": "some text" - }, - "keywords": ["some", "key", "words"] - } - ] - } -] -``` - -You need to apply some kind of structure normalization. - -A workaround to such a data structure looks like this: - -```js -const index = new Document({ - document: { - id: "record:id", - tag: "tag", - index: [ - "record:body:title", - "record:body:footer", - "record:body:keywords" - ] - } -}); - -function add(sequential_data){ - - for(let x = 0, data; x < sequential_data.length; x++){ - - data = sequential_data[x]; - - for(let y = 0, record; y < data.records.length; y++){ - - record = data.records[y]; - - index.add({ - id: record.id, - tag: data.tag, - record: record - }); - } - } -} - -// now just use add() helper method as usual: - -add([{ - // sequential structured data - // take the data example above -}]); -``` - -You can skip the first loop when your document data has just one index as the outer array. - -### Add/Update/Remove Documents to/from the Index - -Add a document to the index: - -```js -index.add({ - id: 0, - title: "Foo", - content: "Bar" - }); -``` - -Update index with a single object or an array of objects: - -```js -index.update({ - data:{ - id: 0, - title: "Foo", - body: { - content: "Bar" - } - } -}); -``` - -Remove a single object or an array of objects from the index: - -```js -index.remove(docs); -``` - -When the id is known, you can also simply remove by (faster): - -```js -index.remove(id); -``` - -### Join / Append Arrays - -On the complex example above, the field `keywords` is an array but here the markup did not have brackets like `keywords[]`. That will also detect the array but instead of appending each entry to a new context, the array will be joined into on large string and added to the index. - -The difference of both kinds of adding array contents is the relevance when searching. When adding each item of an array via `append()` to its own context by using the syntax `field[]`, then the relevance of the last entry concurrent with the first entry. When you left the brackets in the notation, it will join the array to one whitespace-separated string. Here the first entry has the highest relevance, whereas the last entry has the lowest relevance. - -So assuming the keyword from the example above are pre-sorted by relevance to its popularity, then you want to keep this order (information of relevance). For this purpose do not add brackets to the notation. Otherwise, it would take the entries in a new scoring context (the old order is getting lost). - -Also you can left bracket notation for better performance and smaller memory footprint. Use it when you did not need the granularity of relevance by the entries. - -### Field-Search - -Search through all fields: - -```js -index.search(query); -``` - -Search through a specific field: - -```js -index.search(query, { index: "title" }); -``` - -Search through a given set of fields: - -```js -index.search(query, { index: ["title", "content"] }); -``` - -Same as: - -```js -index.search(query, ["title", "content"]); -``` - -Pass custom modifiers and queries to each field: - -```js -index.search([{ - field: "content", - query: "some query", - limit: 100, - suggest: true -},{ - field: "content", - query: "some other query", - limit: 100, - suggest: true -}]); -``` - -You can perform a search through the same field with different queries. - -See all available field-search options. - -### The Result Set - -Schema of the result-set: - -> `fields[] => { field, result[] => { document }}` - -The first index is an array of fields the query was applied to. Each of this field has a record (object) with 2 properties "field" and "result". The "result" is also an array and includes the result for this specific field. The result could be an array of IDs or as enriched with stored document data. - -A non-enriched result set now looks like: - -```js -[{ - field: "title", - result: [0, 1, 2] -},{ - field: "content", - result: [3, 4, 5] -}] -``` - -An enriched result set now looks like: - -```js -[{ - field: "title", - result: [ - { id: 0, doc: { /* document */ }}, - { id: 1, doc: { /* document */ }}, - { id: 2, doc: { /* document */ }} - ] -},{ - field: "content", - result: [ - { id: 3, doc: { /* document */ }}, - { id: 4, doc: { /* document */ }}, - { id: 5, doc: { /* document */ }} - ] -}] -``` - -When using `pluck` instead of "field" you can explicitly select just one field and get back a flat representation: - -```js -index.search(query, { pluck: "title", enrich: true }); -``` - -```js -[ - { id: 0, doc: { /* document */ }}, - { id: 1, doc: { /* document */ }}, - { id: 2, doc: { /* document */ }} -] -``` - -This result set is a replacement of "boolean search". Instead of applying your bool logic to a nested object, you can apply your logic by yourself on top of the result-set dynamically. This opens hugely capabilities on how you process the results. Therefore, the results from the fields aren't squashed into one result anymore. That keeps some important information, like the name of the field as well as the relevance of each field results which didn't get mixed anymore. - -> A field search will apply a query with the boolean "or" logic by default. Each field has its own result to the given query. - -There is one situation where the `bool` property is being still supported. When you like to switch the default "or" logic from the field search into "and", e.g.: - -```js -index.search(query, { - index: ["title", "content"], - bool: "and" -}); -``` - -You will just get results which contains the query in both fields. That's it. - -### Tags - -Like the `key` for the ID just define the path to the tag: - -```js -const index = new Document({ - document: { id: "id", - tag: "tag", - index: "content" + index: [{ + // custom field: + field: "fullname", + custom: function(data){ + // return custom string + return data.firstname + " " + + data.lastname; + } + },{ + // custom field: + field: "location", + custom: function(data){ + return data.street + " " + + data.number + ", " + + data.postal + " " + + data.city; + } + }], + tag: [{ + // existing field + field: "city" + },{ + // custom field: + field: "category", + custom: function(data){ + let tags = []; + // push one or multiple tags + // .... + return tags; + } + }], + store: [{ + field: "anotherfield", + custom: function(data){ + // return a falsy value to filter out + // return anything else as to keep in store + return data; + } + }] } }); ``` +> Filter is also available in custom functions when returning `false`. + +Perform a query against the custom field as usual: ```js -index.add({ - id: 0, - tag: "cat", - content: "Some content ..." +const result = flexsearch.search({ + query: "10178 Berlin Alexanderplatz", + field: "location" }); ``` -Your data also can have multiple tags as an array: - ```js -index.add({ - id: 1, - tag: ["animal", "dog"], - content: "Some content ..." +const result = flexsearch.search({ + query: "john doe", + tag: { "city": "Berlin" } }); ``` -You can perform a tag-specific search by: +## Custom Score Function ```js -index.search(query, { - index: "content", - tag: "animal" +const index = new FlexSearchIndex({ + resolution: 10, + score: function(content, term, term_index, partial, partial_index){ + // you'll need to return a number between 0 and "resolution" + // score is starting from 0, which is the highest score + // for a resolution of 10 you can return 0 - 9 + // ... + return 3; + } }); ``` -This just gives you result which was tagged with the given tag. +A common situation is you have some predefined labels which are related to some kind of order, e.g. the importance or priority. A priority label could be `high`, `moderate`, `low` so you can derive the scoring from those properties. Another example is when you have something already ordered and you would like to keep this order as relevance. -Use multiple tags when searching: +The parameters from the score function explained: + +1. `content` is the whole content as an array of terms (encoded) +2. `term` is the current term which is actually processed (encoded) +3. `term_index` is the index of the term in the content array +4. `partial` is the current partial of a term which is actually processed +5. `partial_index` is the index position of the partial within the term + +Partials params are empty when using tokenizer `strict`. Let's take an example by using the tokenizer `full`. + +The content: "This is an ex[amp]()le of partial encoding"
    +The highlighting part marks the partial which is actually processed. Then your score function will called by passing these parameters: ```js -index.search(query, { - index: "content", - tag: ["cat", "dog"] -}); +function score(content, term, term_index, partial, partial_index){ + content = ["this", "is", "an", "example", "of", "partial", "encoding"] + term = "example" + term_index = 3 + partial = "amp" + partial_index = 2 +} ``` -This gives you result which are tagged with one of the given tag. - -> Multiple tags will apply as the boolean "or" by default. It just needs one of the tags to be existing. - -This is another situation where the `bool` property is still supported. When you like to switch the default "or" logic from the tag search into "and", e.g.: - -```js -index.search(query, { - index: "content", - tag: ["dog", "animal"], - bool: "and" -}); -``` - -You will just get results which contains both tags (in this example there is just one records which has the tag "dog" and "animal"). - -### Tag Search - -You can also fetch results from one or more tags when no query was passed: - -```js -index.search({ tag: ["cat", "dog"] }); -``` - -In this case the result-set looks like: +## Merge Document Results +By default, the result set of Field-Search has a structure grouped by field names: ```js [{ - tag: "cat", - result: [ /* all cats */ ] -},{ - tag: "dog", - result: [ /* all dogs */ ] -}] -``` - -### Limit & Offset - -> By default, every query is limited to 100 entries. Unbounded queries leads into issues. You need to set the limit as an option to adjust the size. - -You can set the limit and the offset for each query: - -```js -index.search(query, { limit: 20, offset: 100 }); -``` - -> You cannot pre-count the size of the result-set. That's a limit by the design of FlexSearch. When you really need a count of all results you are able to page through, then just assign a high enough limit and get back all results and apply your paging offset manually (this works also on server-side). FlexSearch is fast enough that this isn't an issue. - -## Document Store - -Only a document index can have a store. You can use a document index instead of a flat index to get this functionality also when only storing ID-content-pairs. - -You can define independently which fields should be indexed and which fields should be stored. This way you can index fields which should not be included in the search result. - -> Do not use a store when: 1. an array of IDs as the result is good enough, or 2. you already have the contents/documents stored elsewhere (outside the index). - -> When the `store` attribute was set, you have to include all fields which should be stored explicitly (acts like a whitelist). - -> When the `store` attribute was not set, the original document is stored as a fallback. - -This will add the whole original content to the store: - -```js -const index = new Document({ - document: { - index: "content", - store: true - } -}); - -index.add({ id: 0, content: "some text" }); -``` - -### Access documents from internal store - -You can get indexed documents from the store: - -```js -var data = index.get(1); -``` - -You can update/change store contents directly without changing the index by: - -```js -index.set(1, data); -``` - -To update the store and also update the index then just use `index.update`, `index.add` or `index.append`. - -When you perform a query, weather it is a document index or a flat index, then you will always get back an array of IDs. - -Optionally you can enrich the query results automatically with stored contents by: - -```js -index.search(query, { enrich: true }); -``` - -Your results look now like: - -```js -[{ - id: 0, - doc: { /* content from store */ } -},{ - id: 1, - doc: { /* content from store */ } -}] -``` - -### Configure Storage (Recommended) - -This will add just specific fields from a document to the store (the ID isn't necessary to keep in store): - -```js -const index = new Document({ - document: { - index: "content", - store: ["author", "email"] - } -}); - -index.add(id, content); -``` - -You can configure independently what should being indexed and what should being stored. It is highly recommended to make use of this whenever you can. - -Here a useful example of configuring doc and store: - -```js -const index = new Document({ - document: { - index: "content", - store: ["author", "email"] - } -}); - -index.add({ - id: 0, - author: "Jon Doe", - email: "john@mail.com", - content: "Some content for the index ..." -}); -``` - -You can query through the contents and will get back the stored values instead: - -```js -index.search("some content", { enrich: true }); -``` - -Your results are now looking like: - -```js -[{ - field: "content", + field: "fieldname-1", result: [{ - id: 0, - doc: { - author: "Jon Doe", - email: "john@mail.com", - } + id: 1001, + doc: {/* stored document */} + }] +},{ + field: "fieldname-2", + result: [{ + id: 1001, + doc: {/* stored document */} + }] +},{ + field: "fieldname-3", + result: [{ + id: 1002, + doc: {/* stored document */} }] }] ``` -Both field "author" and "email" are not indexed. - - -### Chaining - -Simply chain methods like: - +By passing the search option `merge: true` the result set will be merged into: ```js -var index = FlexSearch.create() - .addMatcher({'â': 'a'}) - .add(0, 'foo') - .add(1, 'bar'); +[{ + id: 1001, + doc: {/* stored document */} + field: ["fieldname-1", "fieldname-2"] +},{ + id: 1002, + doc: {/* stored document */} + field: ["fieldname-3"] +}] ``` +## Extern Worker Configuration + +When using Worker by __also__ assign custom functions to the options e.g.: + +- Custom Encoder +- Custom Encoder methods (normalize, prepare, finalize) +- Custom Score (function) +- Custom Filter (function) +- Custom Fields (function) + +... then you'll need to move your __field configuration__ into a file which exports the configuration as a `default` export. The field configuration is not the whole Document-Descriptor. + +When not using custom functions in combination with Worker you can skip this part. + +Since every field resolves into a dedicated Worker, also every field which includes custom functions should have their own configuration file accordingly. + +Let's take this document descriptor: + ```js -index.remove(0).update(1, 'foo').add(2, 'foobar'); +{ + document: { + index: [{ + // this is the field configuration + // ----> + field: "custom_field", + custom: function(data){ + return "custom field content"; + } + // <------ + }] + } +}; ``` - -## Contextual Search +The configuration which needs to be available as a default export is: +<<<<<<< HEAD > __Note:__ This feature is disabled by default because of its extended memory usage. Read here get more information about and how to enable. FlexSearch introduce a new scoring mechanism called __Contextual Search__ which was invented by Thomas Wilkerling, the author of this library. A Contextual Search incredibly boost up queries to a complete new level but also requires some additional memory (depending on ___depth___). @@ -2198,951 +1508,435 @@ This way contextual search also Only the tokenizer "strict" is actually supported by the contextual index. - -> The contextual index requires additional amount of memory depending on depth. - - -### Auto-Balanced Cache (By Popularity) - -You need to initialize the cache and its limit during the creation of the index: +### Browser (ESM) +An extern configuration for one WorkerIndex, let's assume it is located in `./custom_field.js`: ```js -const index = new Index({ cache: 100 }); +import { Charset } from "./dist/flexsearch.bundle.module.min.js"; +const EncoderPreset = Charset["latin:simple"]; +// it requires a default export: +export default { + encoder: EncoderPreset, + tokenize: "forward", + // custom function: + custom: function(data){ + return "custom field content"; + } +}; ``` +Create Worker Index with the configuration above: ```js -const results = index.searchCache(query); -``` - -A common scenario for using a cache is an autocomplete or instant search when typing. - -> When passing a number as a limit the cache automatically balance stored entries related to their popularity. - -> When just using "true" the cache is unbounded and perform actually 2-3 times faster (because the balancer do not have to run). - - -## Worker Parallelism (Browser + Node.js) - -The new worker model from v0.7.0 is divided into "fields" from the document (1 worker = 1 field index). This way the worker becomes able to solve tasks (subtasks) completely. The downside of this paradigm is they might not have been perfect balanced in storing contents (fields may have different length of contents). On the other hand there is no indication that balancing the storage gives any advantage (they all require the same amount in total). - -When using a document index, then just apply the option "worker": -```js -const index = new Document({ - index: ["tag", "name", "title", "text"], - worker: true -}); - -index.add({ - id: 1, tag: "cat", name: "Tom", title: "some", text: "some" -}).add({ - id: 2, tag: "dog", name: "Ben", title: "title", text: "content" -}).add({ - id: 3, tag: "cat", name: "Max", title: "to", text: "to" -}).add({ - id: 4, tag: "dog", name: "Tim", title: "index", text: "index" +import { Document } from "./dist/flexsearch.bundle.module.min.js"; +// you will need to await for the response! +const flexsearch = await new Document({ + worker: true, + document: { + index: [{ + // the field name needs to be set here + field: "custom_field", + // Absolute URL to your config from above: + config: "http://localhost/config.js" + }] + } }); ``` -``` -Worker 1: { 1: "cat", 2: "dog", 3: "cat", 4: "dog" } -Worker 2: { 1: "Tom", 2: "Ben", 3: "Max", 4: "Tim" } -Worker 3: { 1: "some", 2: "title", 3: "to", 4: "index" } -Worker 4: { 1: "some", 2: "content", 3: "to", 4: "index" } -``` +Here it needs the __absolute URL__, because the WorkerIndex context is from type `Blob` and you can't use relative URLs starting from this context. -When you perform a field search through all fields then this task is being balanced perfectly through all workers, which can solve their subtasks independently. +### Test Case -### Worker Index +As a test the whole IMDB data collection was indexed, containing of: -Above we have seen that documents will create worker automatically for each field. You can also create a WorkerIndex directly (same like using `Index` instead of `Document`). +JSON Documents: 9,273,132
    +Fields: 83,458,188
    +Tokens: 128,898,832
    -Use as ES6 module: +The used index configuration has 2 fields (using bidirectional context of `depth: 1`), 1 custom field, 2 tags and a full datastore of all input json documents. +A non-Worker Document index requires 181 seconds to index all contents.
    +The Worker index takes just 32 seconds to index them all, by processing every field and tag in parallel. For such large content it is a quite impressive result. + +## Fuzzy-Search + +Fuzzysearch describes a basic concept of how making queries more tolerant. Something like Levinstein distance can't be added because of the core architecture. Instead, FlexSearch provides several methods to achieve fuzziness: + +1. Use a tokenizer: `forward`, `reverse` or `full` +2. Don't forget to use any of the builtin encoder `simple` > `balanced` > `advanced` > `extra` > `soundex` (sorted by fuzziness) +3. Use one of the language specific presets e.g. `/lang/en.js` for en-US specific content +4. Enable suggestions by passing the search option `suggest: true` + +Additionally, you can apply custom `Mapper`, `Replacer`, `Stemmer`, `Filter` or by assigning a custom `normalize` or `prepare` function to the Encoder. + +### Compare Fuzzy-Search Encoding + +Original term which was indexed: "Struldbrugs" + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    Encoder:exactdefaultsimplebalanceadvancedextrasoundex
    Index Size3.1 Mb1.9 Mb1.8 Mb1.7 Mb1.6 Mb1.1 Mb0.7 Mb
    Struldbrugs✓✓✓✓✓✓✓
    struldbrugs✓✓✓✓✓✓
    strũldbrųĝgs✓✓✓✓✓
    strultbrooks✓✓✓✓
    shtruhldbrohkz✓✓✓
    zdroltbrykz✓✓
    struhlbrogger✓
    + +The index size was measured after indexing the book "Gulliver's Travels". + +### Custom Encoder + +Since it is very simple to create a custom Encoder, you are welcome to create your own. +e.g. ```js -import WorkerIndex from "./worker/index.js"; -const index = new WorkerIndex(options); -index.add(1, "some") - .add(2, "content") - .add(3, "to") - .add(4, "index"); -``` - -Or when bundled version was used instead: - -```js -var index = new FlexSearch.Worker(options); -index.add(1, "some") - .add(2, "content") - .add(3, "to") - .add(4, "index"); -``` - -Such a WorkerIndex works pretty much the same as a created instance of `Index`. - -> A WorkerIndex only support the `async` variant of all methods. That means when you call `index.search()` on a WorkerIndex this will perform also in async the same way as `index.searchAsync()` will do. - -### Worker Threads (Node.js) - -The worker model for Node.js is based on "worker threads" and works exactly the same way: - -```js -const { Document } = require("flexsearch"); - -const index = new Document({ - index: ["tag", "name", "title", "text"], - worker: true -}); -``` - -Or create a single worker instance for a non-document index: - -```js -const { Worker } = require("flexsearch"); -const index = new Worker({ options }); -``` - -### The Worker Async Model (Best Practices) - -A worker will always perform as async. On a query method call you always should handle the returned promise (e.g. use `await`) or pass a callback function as the last parameter. - -```js -const index = new Document({ - index: ["tag", "name", "title", "text"], - worker: true -}); -``` - -All requests and sub-tasks will run in parallel (prioritize "all tasks completed"): - -```js -index.searchAsync(query, callback); -index.searchAsync(query, callback); -index.searchAsync(query, callback); -``` - -Also (prioritize "all tasks completed"): - -```js -index.searchAsync(query).then(callback); -index.searchAsync(query).then(callback); -index.searchAsync(query).then(callback); -``` - -Or when you have just one callback when all requests are done, simply use `Promise.all()` which also prioritize "all tasks completed": - -```js -Promise.all([ - index.searchAsync(query), - index.searchAsync(query), - index.searchAsync(query) -]).then(callback); -``` - -Inside the callback of `Promise.all()` you will also get an array of results as the first parameter respectively for each query you put into. - -When using `await` you can prioritize the order (prioritize "first task completed") and solve requests one by one and just process the sub-tasks in parallel: - -```js -await index.searchAsync(query); -await index.searchAsync(query); -await index.searchAsync(query); -``` - -Same for `index.add()`, `index.append()`, `index.remove()` or `index.update()`. Here there is a special case which isn't disabled by the library, but you need to keep in mind when using Workers. - -When you call the "synced" version on a worker index: - -```js -index.add(doc); -index.add(doc); -index.add(doc); -// contents aren't indexed yet, -// they just queued on the message channel -``` - -Of course, you can do that but keep in mind that the main thread does not have an additional queue for distributed worker tasks. Running these in a long loop fires content massively to the message channel via `worker.postMessage()` internally. Luckily the browser and Node.js will handle such incoming tasks for you automatically (as long enough free RAM is available). When using the "synced" version on a worker index, the content isn't indexed one line below, because all calls are treated as async by default. - -> When adding/updating/removing large bulks of content to the index (or high frequency), it is recommended to use the async version along with `async/await` to keep a low memory footprint during long processes. - - -## Export / Import - -### Export - -The export has slightly changed. The export now consist of several smaller parts, instead of just one large bulk. You need to pass a callback function which has 2 arguments "key" and "data". This callback function is called by each part, e.g.: - -```js -index.export(function(key, data){ - - // you need to store both the key and the data! - // e.g. use the key for the filename and save your data - - localStorage.setItem(key, data); -}); -``` - -Exporting data to the localStorage isn't really a good practice, but if size is not a concern than use it if you like. The export primarily exists for the usage in Node.js or to store indexes you want to delegate from a server to the client. - -> The size of the export corresponds to the memory consumption of the library. To reduce export size you have to use a configuration which has less memory footprint (use the table at the bottom to get information about configs and its memory allocation). - -When your save routine runs asynchronously you have to return a promise: - -```js -index.export(function(key, data){ - - return new Promise(function(resolve){ - - // do the saving as async - - resolve(); - }); -}); -``` - -> You cannot export the additional table for the "fastupdate" feature. These table exists of references and when stored they fully get serialized and becomes too large. The lib will handle these automatically for you. When importing data, the index automatically disables "fastupdate". - -### Import - -Before you can import data, you need to create your index first. For document indexes provide the same document descriptor you used when export the data. This configuration isn't stored in the export. - -```js -var index = new Index({ ... }); -``` - -To import the data just pass a key and data: - -```js -index.import(key, localStorage.getItem(key)); -``` - -You need to import every key! Otherwise, your index does not work. You need to store the keys from the export and use this keys for the import (the order of the keys can differ). - -This is just for demonstration and is not recommended, because you might have other keys in your localStorage which aren't supported as an import: - -```js -var keys = Object.keys(localStorage); - -for(let i = 0, key; i < keys.length; i++){ - - key = keys[i]; - index.import(key, localStorage.getItem(key)); +function customEncoder(content){ + const tokens = []; + // split content into terms/tokens + // apply your changes to each term/token + // you will need to return an Array of terms/tokens + // so just iterate through the input string and + // push tokens to the array + // ... + return tokens; } -``` -## Languages - -Language-specific definitions are being divided into two groups: - -1. Charset - 1. ___encode___, type: `function(string):string[]` - 2. ___rtl___, type: `boolean` -2. Language - 1. ___matcher___, type: `{string: string}` - 2. ___stemmer___, type: `{string: string}` - 3. ___filter___, type: `string[]` - -The charset contains the encoding logic, the language contains stemmer, stopword filter and matchers. Multiple language definitions can use the same charset encoder. Also this separation let you manage different language definitions for special use cases (e.g. names, cities, dialects/slang, etc.). - -To fully describe a custom language __on the fly__ you need to pass: - -```js -const index = FlexSearch({ - // mandatory: - encode: (content) => [words], - // optionally: - rtl: false, - stemmer: {}, - matcher: {}, - filter: [] +const index = new Index({ + // set to strict when your tokenization was already done + tokenize: "strict", + encode: customEncoder }); ``` -When passing no parameter it uses the `latin:default` schema by default. +If you get some good results please feel free to share your encoder. + +## Load Library (Node.js, ESM, Legacy Browser) + +> Do not use the "src" folder of this repo. It isn't meant to be used directly, instead it needs compilation. You can easily perform a custom build, but don't use the source folder for production. You will need at least any kind of compiler which resolve the compiler flags within the code. The "dist" folder is containing every version which you probably need including unminified ESM modules. + +```bash +npm install flexsearch +``` +The **_dist_** folder are located in: `node_modules/flexsearch/dist/` - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    FieldCategoryDescriptionBuildFileCDN
    encodecharsetThe encoder function. Has to return an array of separated words (or an empty string).flexsearch.bundle.debug.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.bundle.debug.js
    rtlcharsetA boolean property which indicates right-to-left encoding.flexsearch.bundle.min.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.bundle.min.js
    filterlanguageFilter are also known as "stopwords", they completely filter out words from being indexed.flexsearch.bundle.module.debug.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.bundle.module.debug.js
    stemmerlanguageStemmer removes word endings and is a kind of "partial normalization". A word ending just matched when the word length is bigger than the matched partial.flexsearch.bundle.module.min.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.bundle.module.min.js
    matcherlanguageMatcher replaces all occurrences of a given string regardless of its position and is also a kind of "partial normalization".flexsearch.es5.debug.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.es5.debug.js
    flexsearch.es5.min.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.es5.min.js
    flexsearch.light.debug.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.light.debug.js
    flexsearch.light.min.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.light.min.js
    flexsearch.light.module.debug.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.light.module.debug.js
    flexsearch.light.module.min.jsDownloadhttps://rawcdn.githack.com/nextapps-de/flexsearch/0.8.0/dist/flexsearch.light.module.min.js
    Javascript ModulesDownloadhttps://github.com/nextapps-de/flexsearch/tree/0.8.0/dist/module
    Javascript Modules (Minified)Downloadhttps://github.com/nextapps-de/flexsearch/tree/0.8.0/dist/module-min
    Javascript Modules (Debug)Downloadhttps://github.com/nextapps-de/flexsearch/tree/0.8.0/dist/module-debug
    flexsearch.custom.jsRead more about "Custom Build"
    -### 1. Language Packs: ES6 Modules +> All debug versions are providing debug information through the console and gives you helpful advices on certain situations. Do not use them in production, since they are special builds containing extra debugging processes which noticeably reduce performance. -The most simple way to assign charset/language specific encoding via modules is: +The abbreviations used at the end of the filenames indicates: -```js -import charset from "./dist/module/lang/latin/advanced.js"; -import lang from "./dist/module/lang/en.js"; +- `bundle` All features included, FlexSearch is available on `window.FlexSearch` +- `light` Only basic features are included, FlexSearch is available on `window.FlexSearch` +- `es5` bundle has support for EcmaScript5, FlexSearch is available on `window.FlexSearch` +- `module` indicates that this bundle is a Javascript module (ESM), FlexSearch members are available by `import { Index, Document, Worker, Encoder, Charset } from "./flexsearch.bundle.module.min.js"` or alternatively using the default export `import FlexSearch from "./flexsearch.bundle.module.min.js"` +- `min` bundle is minified +- `debug` bundle has enabled debug mode and contains additional code just for debugging purposes (do not use for production) -const index = FlexSearch({ - charset: charset, - lang: lang -}); -``` +### Non-Module Bundles (ES5 Legacy) -Just import the __default export__ by each module and assign them accordingly. +> Non-Module Bundles export all their features to the public namespace "FlexSearch" e.g. `window.FlexSearch.Index` or `window.FlexSearch.Document`. -The full qualified example from above is: - -```js -import { encode, rtl } from "./dist/module/lang/latin/advanced.js"; -import { stemmer, filter, matcher } from "./dist/module/lang/en.js"; - -const index = FlexSearch({ - encode: encode, - rtl: rtl, - stemmer: stemmer, - matcher: matcher, - filter: filter -}); -``` - -The example above is the standard interface which is at least exported from each charset/language. - -You can also define the encoder directly and left all other options: - -```js -import simple from "./dist/module/lang/latin/simple.js"; - -const index = FlexSearch({ - encode: simple -}); -``` - -#### Available Latin Encoders - -1. default -2. simple -3. balance -4. advanced -5. extra - -You can assign a charset by passing the charset during initialization, e.g. `charset: "latin"` for the default charset encoder or `charset: "latin:soundex"` for a encoder variant. - -#### Dialect / Slang - -Language definitions (especially matchers) also could be used to normalize dialect and slang of a specific language. - -### 2. Language Packs: ES5 (Language Packs) - -You need to make the charset and/or language definitions available by: - -1. All charset definitions are included in the `flexsearch.bundle.js` build by default, but no language-specific definitions are included -2. You can load packages located in `/dist/lang/` (files refers to languages, folders are charsets) -3. You can make a custom build - -When loading language packs, make sure that the library was loaded before: +Load the bundle by a script tag: ```html - - - + + ``` -When using the full "bundle" version the built-in latin encoders are already included and you just have to load the language file: +### Module (ESM) + +When using modules you can choose from 2 variants: `flexsearch.xxx.module.min.js` has all features bundled ready for production, whereas the folder `/dist/module/` export all the features in the same structure as the source code but here compiler flags was resolved. + +Also, for each variant there exist: +1. A debug version for the development +2. A pre-compiled minified version for production + +Use the bundled version exported as a module (default export): ```html - - + ``` -Because you loading packs as external packages (non-ES6-modules) you have to initialize them by shortcuts: - -```js -const index = FlexSearch({ - charset: "latin:soundex", - lang: "en" -}); -``` - -> Use the `charset:variant` notation to assign charset and its variants. When just passing the charset without a variant will automatically resolve as `charset:default`. - -You can also override existing definitions, e.g.: - -```js -const index = FlexSearch({ - charset: "latin", - lang: "en", - matcher: {} -}); -``` - -> Passed definitions will __not__ extend default definitions, they will replace them. - -When you like to extend a definition just create a new language file and put in all the logic. - -#### Encoder Variants - -It is pretty straight forward when using an encoder variant: +Or import FlexSearch members separately by: ```html - - - - + ``` -When using the full "bundle" version the built-in latin encoders are already included and you just have to load the language file: +Use non-bundled modules: ```html - - + +``` + +Also, pre-compiled non-bundled production-ready modules are located in `dist/module-min/`, whereas the debug version is located at `dist/module-debug/`. + +You can also load modules via CDN: + +```html + +``` + +### Node.js + +Install FlexSearch via NPM: + +```npm +npm install flexsearch ``` ```js -const index_advanced = FlexSearch({ - charset: "latin:advanced" -}); - -const index_extra = FlexSearch({ - charset: "latin:extra" -}); +const { Index, Document, Encoder } = require("flexsearch"); +const index = new Index(/* ... */); ``` -### Partial Tokenizer +When you are using ESM in Node.js then just use the Modules explained one section above. -In FlexSearch you can't provide your own partial tokenizer, because it is a direct dependency to the core unit. The built-in tokenizer of FlexSearch splits each word into fragments by different patterns: +## Migration -1. strict (supports contextual index) -2. forward -3. reverse (including forward) -4. full - -### Language Processing Pipeline - -This is the default pipeline provided by FlexSearch: - -

    - -

    - -#### Custom Pipeline - -At first take a look into the default pipeline in `src/common.js`. It is very simple and straight forward. The pipeline will process as some sort of inversion of control, the final encoder implementation has to handle charset and also language specific transformations. This workaround has left over from many tests. - -Inject the default pipeline by e.g.: - -```js -this.pipeline( - - /* string: */ str.toLowerCase(), - /* normalize: */ false, - /* split: */ split, - /* collapse: */ false -); -``` - -Use the pipeline schema from above to understand the iteration and the difference of pre-encoding and post-encoding. Stemmer and matchers needs to be applied after charset normalization but before language transformations, filters also. - -Here is a good example of extending pipelines: `src/lang/latin/extra.js` → `src/lang/latin/advanced.js` → `src/lang/latin/simple.js`. - -### How to contribute? - -Search for your language in `src/lang/`, if it exists you can extend or provide variants (like dialect/slang). If the language doesn't exist create a new file and check if any of the existing charsets (e.g. latin) fits to your language. When no charset exist, you need to provide a charset as a base for the language. - -A new charset should provide at least: - -1. `encode` A function which normalize the charset of a passed text content (remove special chars, lingual transformations, etc.) and __returns an array of separated words__. Also stemmer, matcher or stopword filter needs to be applied here. When the language has no words make sure to provide something similar, e.g. each chinese sign could also be a "word". Don't return the whole text content without split. -3. `rtl` A boolean flag which indicates right-to-left encoding - -Basically the charset needs just to provide an encoder function along with an indicator for right-to-left encoding: - -```js -export function encode(str){ return [str] } -export const rtl = false; -``` - - -## Encoder Matching Comparison - -> Reference String: __"Björn-Phillipp Mayer"__ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Querydefaultsimpleadvancedextra
    björnyesyesyesyes
    björyesyesyesyes
    bjornnoyesyesyes
    bjoernnonoyesyes
    philippnonoyesyes
    filipnonoyesyes
    björnphillipnoyesyesyes
    meiernonoyesyes
    björn meiernonoyesyes
    meier fhilipnonoyesyes
    byorn mairnononoyes
    (false positives)nononoyes
    - - -## Memory Allocation - -The book "Gulliver's Travels Swift Jonathan 1726" was fully indexed for the examples below. - -The most memory-optimized meaningful setting will allocate just 1.2 Mb for the whole book indexed! This is probably the most tiny memory footprint you will get from a search library. - -```js -import { encode } from "./lang/latin/extra.js"; - -index = new Index({ - encode: encode, - tokenize: "strict", - optimize: true, - resolution: 1, - minlength: 3, - fastupdate: false, - context: false -}); -``` - - -### Memory Consumption - -The book "Gulliver's Travels" (Swift Jonathan 1726) was completely indexed for this test: - -
    - - -### Compare Impact of Memory Allocation - -by default a lexical index is very small:
    -`depth: 0, bidirectional: 0, resolution: 3, minlength: 0` => 2.1 Mb - -a higher resolution will increase the memory allocation:
    -`depth: 0, bidirectional: 0, resolution: 9, minlength: 0` => 2.9 Mb - -using the contextual index will increase the memory allocation:
    -`depth: 1, bidirectional: 0, resolution: 9, minlength: 0` => 12.5 Mb - -a higher contextual depth will increase the memory allocation:
    -`depth: 2, bidirectional: 0, resolution: 9, minlength: 0` => 21.5 Mb - -a higher minlength will decrease memory allocation:
    -`depth: 2, bidirectional: 0, resolution: 9, minlength: 3` => 19.0 Mb - -using bidirectional will decrease memory allocation:
    -`depth: 2, bidirectional: 1, resolution: 9, minlength: 3` => 17.9 Mb - -enable the option "fastupdate" will increase memory allocation:
    -`depth: 2, bidirectional: 1, resolution: 9, minlength: 3` => 6.3 Mb - -### Full Comparison Table - -Every search library is constantly in competition with these 4 properties: - -1. Memory Allocation -2. Performance -3. Matching Capabilities -4. Relevance Order (Scoring) - -FlexSearch provides you many parameters you can use to adjust the optimal balance for your specific use-case. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    ModifierMemory Impact *Performance Impact **Matching Impact **Scoring Impact **
    resolution+1 (per level)+1 (per level)0+2 (per level)
    depth+4 (per level)-1 (per level)-10 + depth+10
    minlength-2 (per level)+2 (per level)-3 (per level)+2 (per level)
    bidirectional-20+3-1
    fastupdate+1+10 (update, remove)00
    optimize: true-7-10-3
    encoder: "icase"0000
    encoder: "simple"-2-1+20
    encoder: "advanced"-3-2+40
    encoder: "extra"-5-5+60
    encoder: "soundex"-6-2+80
    tokenize: "strict"0000
    tokenize: "forward"+3-2+50
    tokenize: "reverse"+5-4+70
    tokenize: "full"+8-5+100
    document index+3 (per field)-1 (per field)00
    document tags+1 (per tag)-1 (per tag)00
    store: true+5 (per document)000
    store: [fields]+1 (per field)000
    cache: true+10+1000
    cache: 100+1+900
    type of ids: number0000
    type of ids: string+3-300
    -* range from -10 to 10, lower is better (-10 => big decrease, 0 => unchanged, +10 => big increase)
    -** range from -10 to 10, higher is better - - -## Presets - -1. `memory` (primary optimize for memory) -2. `performance` (primary optimize for performance) -3. `match` (primary optimize for matching) -4. `score` (primary optimize for scoring) -5. `default` (the default balanced profile) - -These profiles are covering standard use cases. It is recommended to apply custom configuration instead of using profiles to get the best out for your situation. Every profile could be optimized further to its specific task, e.g. extreme performance optimized configuration or extreme memory and so on. - -You can pass a preset during creation/initialization of the index. - - - -## Best Practices - -##### Use numeric IDs - -It is recommended to use numeric id values as reference when adding content to the index. The byte length of passed ids influences the memory consumption significantly. If this is not possible you should consider to use a index table and map the ids with indexes, this becomes important especially when using contextual indexes on a large amount of content. - -##### Split Complexity - -Whenever you can, try to divide content by categories and add them to its own index, e.g.: - -```js -var action = new FlexSearch(); -var adventure = new FlexSearch(); -var comedy = new FlexSearch(); -``` - -This way you can also provide different settings for each category. This is actually the fastest way to perform a fuzzy search. - -To make this workaround more extendable you can use a short helper: - -```js -var index = {}; - -function add(id, cat, content){ - (index[cat] || ( - index[cat] = new FlexSearch - )).add(id, content); -} - -function search(cat, query){ - return index[cat] ? - index[cat].search(query) : []; -} -``` - -Add content to the index: -```js -add(1, "action", "Movie Title"); -add(2, "adventure", "Movie Title"); -add(3, "comedy", "Movie Title"); -``` - -Perform queries: -```js -var results = search("action", "movie title"); // --> [1] -``` - -Split indexes by categories improves performance significantly. - ---- - -Copyright 2018-2023 Thomas Wilkerling, Hosted by Nextapps GmbH
    -Released under the Apache 2.0 License
    +- The index option property "minlength" has moved to the Encoder Class +- The index option flag "optimize" was removed +- The index option flag "lang" was replaced by the Encoder Class `.assign()` +- Boost cannot apply upfront anymore when indexing, instead you can use the boost property on a query dynamically +- All definitions of the old text encoding process was replaced by similar definitions (Array changed to Set, Object changed to Map). You can use of the helper methods like `.addMatcher(char_match, char_replace)` which adds everything properly. +- The default value for `fastupdate` is set to `false` by default when not passed via options +- The method `index.encode()` has moved to `index.encoder.encode()` +- The options `charset` and `lang` was removed from index (replaced by `Encoder.assign({...})`) +- Every charset collection (files in folder `/lang/**.js`) is now exported as a config object (instead of a function). This config needs to be created by passing to the constructor `new Encoder(config)` or can be added to an existing instance via `encoder.assign(config)`. The reason was to keep the default encoder configuration when having multiple document indexes. +- The property `bool` from DocumentOptions was removed (replaced by `Resolver`) +- The static methods `FlexSearch.registerCharset()` and `FlexSearch.registerLanguage()` was removed, those collections are now exported to `FlexSearch.Charset` and `FlexSearch.Language` which can be accessed as module `import { Charset, Language } from "flexsearch"` diff --git a/dist/flexsearch.bundle.debug.js b/dist/flexsearch.bundle.debug.js index 0fd840d..cf42ae5 100644 --- a/dist/flexsearch.bundle.debug.js +++ b/dist/flexsearch.bundle.debug.js @@ -1,655 +1,70 @@ /**! - * FlexSearch.js v0.7.41 (Bundle) + * FlexSearch.js v0.8.0 (Bundle/Debug) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ (function _f(self){'use strict';try{if(module)self=module}catch(e){}self._factory=_f; -var t; -function u(a) { - return "undefined" !== typeof a ? a : !0; -} -function v(a) { - const b = Array(a); - for (let c = 0; c < a; c++) { - b[c] = x(); +var u; +function y(a, b, c) { + const d = typeof c, e = typeof a; + if ("undefined" !== d) { + if ("undefined" !== e) { + if (c) { + if ("function" === e && d === e) { + return function(h) { + return a(c(h)); + }; + } + b = a.constructor; + if (b === c.constructor) { + if (b === Array) { + return c.concat(a); + } + if (b === Map) { + var f = new Map(c); + for (var g of a) { + f.set(g[0], g[1]); + } + return f; + } + if (b === Set) { + g = new Set(c); + for (f of a.values()) { + g.add(f); + } + return g; + } + } + } + return a; + } + return c; } - return b; + return "undefined" === e ? b : a; } -function x() { +function z() { return Object.create(null); } function aa(a, b) { return b.length - a.length; } -function C(a) { +function B(a) { return "string" === typeof a; } -function D(a) { +function C(a) { return "object" === typeof a; } -function E(a) { - return "function" === typeof a; -} -;function ba(a, b) { - var c = ca; - if (a && (b && (a = F(a, b)), this.H && (a = F(a, this.H)), this.J && 1 < a.length && (a = F(a, this.J)), c || "" === c)) { - b = a.split(c); - if (this.filter) { - a = this.filter; - c = b.length; - const d = []; - for (let e = 0, f = 0; e < c; e++) { - const h = b[e]; - h && !a[h] && (d[f++] = h); - } - a = d; - } else { - a = b; - } - return a; - } - return a; -} -const ca = /[\p{Z}\p{S}\p{P}\p{C}]+/u, da = /[\u0300-\u036f]/g; -function ea(a, b) { - const c = Object.keys(a), d = c.length, e = []; - let f = "", h = 0; - for (let g = 0, k, m; g < d; g++) { - k = c[g], (m = a[k]) ? (e[h++] = G(b ? "(?!\\b)" + k + "(\\b|_)" : k), e[h++] = m) : f += (f ? "|" : "") + k; - } - f && (e[h++] = G(b ? "(?!\\b)(" + f + ")(\\b|_)" : "(" + f + ")"), e[h] = ""); - return e; -} -function F(a, b) { - for (let c = 0, d = b.length; c < d && (a = a.replace(b[c], b[c + 1]), a); c += 2) { - } - return a; -} -function G(a) { - return new RegExp(a, "g"); -} -function fa(a) { - let b = "", c = ""; - for (let d = 0, e = a.length, f; d < e; d++) { - (f = a[d]) !== c && (b += c = f); +function ba(a) { + const b = []; + for (const c of a.keys()) { + b.push(c); } return b; } -;var ia = {encode:ha, F:!1, G:""}; -function ha(a) { - return ba.call(this, ("" + a).toLowerCase(), !1); -} -;const ja = {}, I = {}; -function ka(a) { - J(a, "add"); - J(a, "append"); - J(a, "search"); - J(a, "update"); - J(a, "remove"); -} -function J(a, b) { - a[b + "Async"] = function() { - const c = this, d = arguments; - var e = d[d.length - 1]; - let f; - E(e) && (f = e, delete d[d.length - 1]); - e = new Promise(function(h) { - setTimeout(function() { - c.async = !0; - const g = c[b].apply(c, d); - c.async = !1; - h(g); - }); - }); - return f ? (e.then(f), this) : e; - }; -} -;function la(a, b, c, d) { - const e = a.length; - let f = [], h, g, k = 0; - d && (d = []); - for (let m = e - 1; 0 <= m; m--) { - const n = a[m], w = n.length, q = x(); - let r = !h; - for (let l = 0; l < w; l++) { - const p = n[l], A = p.length; - if (A) { - for (let B = 0, z, y; B < A; B++) { - if (y = p[B], h) { - if (h[y]) { - if (!m) { - if (c) { - c--; - } else { - if (f[k++] = y, k === b) { - return f; - } - } - } - if (m || d) { - q[y] = 1; - } - r = !0; - } - if (d && (z = (g[y] || 0) + 1, g[y] = z, z < e)) { - const H = d[z - 2] || (d[z - 2] = []); - H[H.length] = y; - } - } else { - q[y] = 1; - } - } - } - } - if (d) { - h || (g = q); - } else if (!r) { - return []; - } - h = q; - } - if (d) { - for (let m = d.length - 1, n, w; 0 <= m; m--) { - n = d[m]; - w = n.length; - for (let q = 0, r; q < w; q++) { - if (r = n[q], !h[r]) { - if (c) { - c--; - } else { - if (f[k++] = r, k === b) { - return f; - } - } - h[r] = 1; - } - } - } - } - return f; -} -function ma(a, b) { - const c = x(), d = x(), e = []; - for (let f = 0; f < a.length; f++) { - c[a[f]] = 1; - } - for (let f = 0, h; f < b.length; f++) { - h = b[f]; - for (let g = 0, k; g < h.length; g++) { - k = h[g], c[k] && !d[k] && (d[k] = 1, e[e.length] = k); - } - } - return e; -} -;function K(a) { - this.l = !0 !== a && a; - this.cache = x(); - this.h = []; -} -function na(a, b, c) { - D(a) && (a = a.query); - let d = this.cache.get(a); - d || (d = this.search(a, b, c), this.cache.set(a, d)); - return d; -} -K.prototype.set = function(a, b) { - if (!this.cache[a]) { - var c = this.h.length; - c === this.l ? delete this.cache[this.h[c - 1]] : c++; - for (--c; 0 < c; c--) { - this.h[c] = this.h[c - 1]; - } - this.h[0] = a; - } - this.cache[a] = b; -}; -K.prototype.get = function(a) { - const b = this.cache[a]; - if (this.l && b && (a = this.h.indexOf(a))) { - const c = this.h[a - 1]; - this.h[a - 1] = this.h[a]; - this.h[a] = c; - } - return b; -}; -const oa = {memory:{charset:"latin:extra", D:3, B:4, m:!1}, performance:{D:3, B:3, s:!1, context:{depth:2, D:1}}, match:{charset:"latin:extra", G:"reverse"}, score:{charset:"latin:advanced", D:20, B:3, context:{depth:3, D:9}}, "default":{}}; -function qa(a, b, c, d, e, f, h, g) { - setTimeout(function() { - const k = a(c ? c + "." + d : d, JSON.stringify(h)); - k && k.then ? k.then(function() { - b.export(a, b, c, e, f + 1, g); - }) : b.export(a, b, c, e, f + 1, g); - }); -} -;function L(a, b) { - if (!(this instanceof L)) { - return new L(a); - } - var c; - if (a) { - if (C(a)) { - oa[a] || console.warn("Preset not found: " + a), a = oa[a]; - } else { - if (c = a.preset) { - c[c] || console.warn("Preset not found: " + c), a = Object.assign({}, c[c], a); - } - } - c = a.charset; - var d = a.lang; - C(c) && (-1 === c.indexOf(":") && (c += ":default"), c = I[c]); - C(d) && (d = ja[d]); - } else { - a = {}; - } - let e, f, h = a.context || {}; - this.encode = a.encode || c && c.encode || ha; - this.register = b || x(); - this.D = e = a.resolution || 9; - this.G = b = c && c.G || a.tokenize || "strict"; - this.depth = "strict" === b && h.depth; - this.l = u(h.bidirectional); - this.s = f = u(a.optimize); - this.m = u(a.fastupdate); - this.B = a.minlength || 1; - this.C = a.boost; - this.map = f ? v(e) : x(); - this.A = e = h.resolution || 1; - this.h = f ? v(e) : x(); - this.F = c && c.F || a.rtl; - this.H = (b = a.matcher || d && d.H) && ea(b, !1); - this.J = (b = a.stemmer || d && d.J) && ea(b, !0); - if (c = b = a.filter || d && d.filter) { - c = b; - d = x(); - for (let g = 0, k = c.length; g < k; g++) { - d[c[g]] = 1; - } - c = d; - } - this.filter = c; - this.cache = (b = a.cache) && new K(b); -} -t = L.prototype; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.add = function(a, b, c, d) { - if (b && (a || 0 === a)) { - if (!d && !c && this.register[a]) { - return this.update(a, b); - } - b = this.encode(b); - if (d = b.length) { - const m = x(), n = x(), w = this.depth, q = this.D; - for (let r = 0; r < d; r++) { - let l = b[this.F ? d - 1 - r : r]; - var e = l.length; - if (l && e >= this.B && (w || !n[l])) { - var f = M(q, d, r), h = ""; - switch(this.G) { - case "full": - if (2 < e) { - for (f = 0; f < e; f++) { - for (var g = e; g > f; g--) { - if (g - f >= this.B) { - var k = M(q, d, r, e, f); - h = l.substring(f, g); - N(this, n, h, k, a, c); - } - } - } - break; - } - case "reverse": - if (1 < e) { - for (g = e - 1; 0 < g; g--) { - h = l[g] + h, h.length >= this.B && N(this, n, h, M(q, d, r, e, g), a, c); - } - h = ""; - } - case "forward": - if (1 < e) { - for (g = 0; g < e; g++) { - h += l[g], h.length >= this.B && N(this, n, h, f, a, c); - } - break; - } - default: - if (this.C && (f = Math.min(f / this.C(b, l, r) | 0, q - 1)), N(this, n, l, f, a, c), w && 1 < d && r < d - 1) { - for (e = x(), h = this.A, f = l, g = Math.min(w + 1, d - r), e[f] = 1, k = 1; k < g; k++) { - if ((l = b[this.F ? d - 1 - r - k : r + k]) && l.length >= this.B && !e[l]) { - e[l] = 1; - const p = this.l && l > f; - N(this, m, p ? f : l, M(h + (d / 2 > h ? 0 : 1), d, r, g - 1, k - 1), a, c, p ? l : f); - } - } - } - } - } - } - this.m || (this.register[a] = 1); - } - } - return this; -}; -function M(a, b, c, d, e) { - return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; -} -function N(a, b, c, d, e, f, h) { - let g = h ? a.h : a.map; - if (!b[c] || h && !b[c][h]) { - a.s && (g = g[d]), h ? (b = b[c] || (b[c] = x()), b[h] = 1, g = g[h] || (g[h] = x())) : b[c] = 1, g = g[c] || (g[c] = []), a.s || (g = g[d] || (g[d] = [])), f && g.includes(e) || (g[g.length] = e, a.m && (a = a.register[e] || (a.register[e] = []), a[a.length] = g)); - } -} -t.search = function(a, b, c) { - c || (!b && D(a) ? (c = a, a = c.query) : D(b) && (c = b)); - let d = [], e; - let f, h = 0; - if (c) { - a = c.query || a; - b = c.limit; - h = c.offset || 0; - var g = c.context; - f = c.suggest; - } - if (a && (a = this.encode("" + a), e = a.length, 1 < e)) { - c = x(); - var k = []; - for (let n = 0, w = 0, q; n < e; n++) { - if ((q = a[n]) && q.length >= this.B && !c[q]) { - if (this.s || f || this.map[q]) { - k[w++] = q, c[q] = 1; - } else { - return d; - } - } - } - a = k; - e = a.length; - } - if (!e) { - return d; - } - b || (b = 100); - g = this.depth && 1 < e && !1 !== g; - c = 0; - let m; - g ? (m = a[0], c = 1) : 1 < e && a.sort(aa); - for (let n, w; c < e; c++) { - w = a[c]; - g ? (n = ra(this, d, f, b, h, 2 === e, w, m), f && !1 === n && d.length || (m = w)) : n = ra(this, d, f, b, h, 1 === e, w); - if (n) { - return n; - } - if (f && c === e - 1) { - k = d.length; - if (!k) { - if (g) { - g = 0; - c = -1; - continue; - } - return d; - } - if (1 === k) { - return sa(d[0], b, h); - } - } - } - return la(d, b, h, f); -}; -function ra(a, b, c, d, e, f, h, g) { - let k = [], m = g ? a.h : a.map; - a.s || (m = ta(m, h, g, a.l)); - if (m) { - let n = 0; - const w = Math.min(m.length, g ? a.A : a.D); - for (let q = 0, r = 0, l, p; q < w; q++) { - if (l = m[q]) { - if (a.s && (l = ta(l, h, g, a.l)), e && l && f && (p = l.length, p <= e ? (e -= p, l = null) : (l = l.slice(e), e = 0)), l && (k[n++] = l, f && (r += l.length, r >= d))) { - break; - } - } - } - if (n) { - if (f) { - return sa(k, d, 0); - } - b[b.length] = k; - return; - } - } - return !c && k; -} -function sa(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function ta(a, b, c, d) { - c ? (d = d && b > c, a = (a = a[d ? b : c]) && a[d ? c : b]) : a = a[b]; - return a; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a, b) { - const c = this.register[a]; - if (c) { - if (this.m) { - for (let d = 0, e; d < c.length; d++) { - e = c[d], e.splice(e.indexOf(a), 1); - } - } else { - O(this.map, a, this.D, this.s), this.depth && O(this.h, a, this.A, this.s); - } - b || delete this.register[a]; - if (this.cache) { - b = this.cache; - for (let d = 0, e, f; d < b.h.length; d++) { - f = b.h[d], e = b.cache[f], e.includes(a) && (b.h.splice(d--, 1), delete b.cache[f]); - } - } - } - return this; -}; -function O(a, b, c, d, e) { - let f = 0; - if (a.constructor === Array) { - if (e) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), f++) : f++; - } else { - e = Math.min(a.length, c); - for (let h = 0, g; h < e; h++) { - if (g = a[h]) { - f = O(g, b, c, d, e), d || f || delete a[h]; - } - } - } - } else { - for (let h in a) { - (f = O(a[h], b, c, d, e)) || delete a[h]; - } - } - return f; -} -t.searchCache = na; -t.export = function(a, b, c, d, e, f) { - let h = !0; - "undefined" === typeof f && (h = new Promise(m => { - f = m; - })); - let g, k; - switch(e || (e = 0)) { - case 0: - g = "reg"; - if (this.m) { - k = x(); - for (let m in this.register) { - k[m] = 1; - } - } else { - k = this.register; - } - break; - case 1: - g = "cfg"; - k = {doc:0, opt:this.s ? 1 : 0}; - break; - case 2: - g = "map"; - k = this.map; - break; - case 3: - g = "ctx"; - k = this.h; - break; - default: - "undefined" === typeof c && f && f(); - return; - } - qa(a, b || this, c, g, d, e, k, f); - return h; -}; -t.import = function(a, b) { - if (b) { - switch(C(b) && (b = JSON.parse(b)), a) { - case "cfg": - this.s = !!b.opt; - break; - case "reg": - this.m = !1; - this.register = b; - break; - case "map": - this.map = b; - break; - case "ctx": - this.h = b; - } - } -}; -ka(L.prototype); -function ua(a) { - a = a.data; - var b = self._index; - const c = a.args; - var d = a.task; - switch(d) { - case "init": - d = a.options || {}; - a = a.factory; - b = d.encode; - d.cache = !1; - b && 0 === b.indexOf("function") && (d.encode = Function("return " + b)()); - a ? (Function("return " + a)()(self), self._index = new self.FlexSearch.Index(d), delete self.FlexSearch) : self._index = new L(d); - break; - default: - a = a.id, b = b[d].apply(b, c), postMessage("search" === d ? {id:a, msg:b} : {id:a}); - } -} -;let va = 0; -function P(a) { - if (!(this instanceof P)) { - return new P(a); - } - var b; - a ? E(b = a.encode) && (a.encode = b.toString()) : a = {}; - (b = (self || window)._factory) && (b = b.toString()); - const c = "undefined" === typeof window && self.exports, d = this; - this.o = wa(b, c, a.worker); - this.h = x(); - if (this.o) { - if (c) { - this.o.on("message", function(e) { - d.h[e.id](e.msg); - delete d.h[e.id]; - }); - } else { - this.o.onmessage = function(e) { - e = e.data; - d.h[e.id](e.msg); - delete d.h[e.id]; - }; - } - this.o.postMessage({task:"init", factory:b, options:a}); - } -} -Q("add"); -Q("append"); -Q("search"); -Q("update"); -Q("remove"); -function Q(a) { - P.prototype[a] = P.prototype[a + "Async"] = function() { - const b = this, c = [].slice.call(arguments); - var d = c[c.length - 1]; - let e; - E(d) && (e = d, c.splice(c.length - 1, 1)); - d = new Promise(function(f) { - setTimeout(function() { - b.h[++va] = f; - b.o.postMessage({task:a, id:va, args:c}); - }); - }); - return e ? (d.then(e), this) : d; - }; -} -function wa(a, b, c) { - let d; - try { - d = b ? new (require("worker_threads")["Worker"])(__dirname + "/node/node.js") : a ? new Worker(URL.createObjectURL(new Blob(["onmessage=" + ua.toString()], {type:"text/javascript"}))) : new Worker(C(c) ? c : "worker/worker.js", {type:"module"}); - } catch (e) { - } - return d; -} -;function S(a) { - if (!(this instanceof S)) { - return new S(a); - } - var b = a.document || a.doc || a, c; - this.K = []; - this.h = []; - this.A = []; - this.register = x(); - this.key = (c = b.key || b.id) && T(c, this.A) || "id"; - this.m = u(a.fastupdate); - this.C = (c = b.store) && !0 !== c && []; - this.store = c && x(); - this.I = (c = b.tag) && T(c, this.A); - this.l = c && x(); - this.cache = (c = a.cache) && new K(c); - a.cache = !1; - this.o = a.worker; - this.async = !1; - c = x(); - let d = b.index || b.field || b; - C(d) && (d = [d]); - for (let e = 0, f, h; e < d.length; e++) { - f = d[e], C(f) || (h = f, f = f.field), h = D(h) ? Object.assign({}, a, h) : a, this.o && (c[f] = new P(h), c[f].o || (this.o = !1)), this.o || (c[f] = new L(h, this.register)), this.K[e] = T(f, this.A), this.h[e] = f; - } - if (this.C) { - for (a = b.store, C(a) && (a = [a]), b = 0; b < a.length; b++) { - this.C[b] = T(a[b], this.A); - } - } - this.index = c; -} -function T(a, b) { - const c = a.split(":"); - let d = 0; - for (let e = 0; e < c.length; e++) { - a = c[e], 0 <= a.indexOf("[]") && (a = a.substring(0, a.length - 2)) && (b[d] = !0), a && (c[d++] = a); - } - d < c.length && (c.length = d); - return 1 < d ? c : c[0]; -} -function U(a, b) { - if (C(b)) { +function ca(a, b) { + if (B(b)) { a = a[b]; } else { for (let c = 0; a && c < b.length; c++) { @@ -658,22 +73,1481 @@ function U(a, b) { } return a; } -function V(a, b, c, d, e) { - a = a[e]; - if (d === c.length - 1) { - b[e] = a; - } else if (a) { - if (a.constructor === Array) { - for (b = b[e] = Array(a.length), e = 0; e < a.length; e++) { - V(a, b, c, d, e); +function da(a) { + let b = 0; + for (let c = 0, d; c < a.length; c++) { + (d = a[c]) && b < d.length && (b = d.length); + } + return b; +} +;const ea = /[^\p{L}\p{N}]+/u, fa = /(\d{3})/g, ha = /(\D)(\d{3})/g, ia = /(\d{3})(\D)/g, ja = "".normalize && /[\u0300-\u036f]/g, ka = !ja && new Map([["\u00aa", "a"], ["\u00b2", "2"], ["\u00b3", "3"], ["\u00b9", "1"], ["\u00ba", "o"], ["\u00bc", "1\u20444"], ["\u00bd", "1\u20442"], ["\u00be", "3\u20444"], ["\u00e0", "a"], ["\u00e1", "a"], ["\u00e2", "a"], ["\u00e3", "a"], ["\u00e4", "a"], ["\u00e5", "a"], ["\u00e7", "c"], ["\u00e8", "e"], ["\u00e9", "e"], ["\u00ea", "e"], ["\u00eb", "e"], ["\u00ec", +"i"], ["\u00ed", "i"], ["\u00ee", "i"], ["\u00ef", "i"], ["\u00f1", "n"], ["\u00f2", "o"], ["\u00f3", "o"], ["\u00f4", "o"], ["\u00f5", "o"], ["\u00f6", "o"], ["\u00f9", "u"], ["\u00fa", "u"], ["\u00fb", "u"], ["\u00fc", "u"], ["\u00fd", "y"], ["\u00ff", "y"], ["\u0101", "a"], ["\u0103", "a"], ["\u0105", "a"], ["\u0107", "c"], ["\u0109", "c"], ["\u010b", "c"], ["\u010d", "c"], ["\u010f", "d"], ["\u0113", "e"], ["\u0115", "e"], ["\u0117", "e"], ["\u0119", "e"], ["\u011b", "e"], ["\u011d", "g"], ["\u011f", +"g"], ["\u0121", "g"], ["\u0123", "g"], ["\u0125", "h"], ["\u0129", "i"], ["\u012b", "i"], ["\u012d", "i"], ["\u012f", "i"], ["\u0133", "ij"], ["\u0135", "j"], ["\u0137", "k"], ["\u013a", "l"], ["\u013c", "l"], ["\u013e", "l"], ["\u0140", "l"], ["\u0144", "n"], ["\u0146", "n"], ["\u0148", "n"], ["\u0149", "n"], ["\u014d", "o"], ["\u014f", "o"], ["\u0151", "o"], ["\u0155", "r"], ["\u0157", "r"], ["\u0159", "r"], ["\u015b", "s"], ["\u015d", "s"], ["\u015f", "s"], ["\u0161", "s"], ["\u0163", "t"], ["\u0165", +"t"], ["\u0169", "u"], ["\u016b", "u"], ["\u016d", "u"], ["\u016f", "u"], ["\u0171", "u"], ["\u0173", "u"], ["\u0175", "w"], ["\u0177", "y"], ["\u017a", "z"], ["\u017c", "z"], ["\u017e", "z"], ["\u017f", "s"], ["\u01a1", "o"], ["\u01b0", "u"], ["\u01c6", "dz"], ["\u01c9", "lj"], ["\u01cc", "nj"], ["\u01ce", "a"], ["\u01d0", "i"], ["\u01d2", "o"], ["\u01d4", "u"], ["\u01d6", "u"], ["\u01d8", "u"], ["\u01da", "u"], ["\u01dc", "u"], ["\u01df", "a"], ["\u01e1", "a"], ["\u01e3", "ae"], ["\u00e6", "ae"], +["\u01fd", "ae"], ["\u01e7", "g"], ["\u01e9", "k"], ["\u01eb", "o"], ["\u01ed", "o"], ["\u01ef", "\u0292"], ["\u01f0", "j"], ["\u01f3", "dz"], ["\u01f5", "g"], ["\u01f9", "n"], ["\u01fb", "a"], ["\u01ff", "\u00f8"], ["\u0201", "a"], ["\u0203", "a"], ["\u0205", "e"], ["\u0207", "e"], ["\u0209", "i"], ["\u020b", "i"], ["\u020d", "o"], ["\u020f", "o"], ["\u0211", "r"], ["\u0213", "r"], ["\u0215", "u"], ["\u0217", "u"], ["\u0219", "s"], ["\u021b", "t"], ["\u021f", "h"], ["\u0227", "a"], ["\u0229", "e"], +["\u022b", "o"], ["\u022d", "o"], ["\u022f", "o"], ["\u0231", "o"], ["\u0233", "y"], ["\u02b0", "h"], ["\u02b1", "h"], ["\u0266", "h"], ["\u02b2", "j"], ["\u02b3", "r"], ["\u02b4", "\u0279"], ["\u02b5", "\u027b"], ["\u02b6", "\u0281"], ["\u02b7", "w"], ["\u02b8", "y"], ["\u02e0", "\u0263"], ["\u02e1", "l"], ["\u02e2", "s"], ["\u02e3", "x"], ["\u02e4", "\u0295"], ["\u0390", "\u03b9"], ["\u03ac", "\u03b1"], ["\u03ad", "\u03b5"], ["\u03ae", "\u03b7"], ["\u03af", "\u03b9"], ["\u03b0", "\u03c5"], ["\u03ca", +"\u03b9"], ["\u03cb", "\u03c5"], ["\u03cc", "\u03bf"], ["\u03cd", "\u03c5"], ["\u03ce", "\u03c9"], ["\u03d0", "\u03b2"], ["\u03d1", "\u03b8"], ["\u03d2", "\u03a5"], ["\u03d3", "\u03a5"], ["\u03d4", "\u03a5"], ["\u03d5", "\u03c6"], ["\u03d6", "\u03c0"], ["\u03f0", "\u03ba"], ["\u03f1", "\u03c1"], ["\u03f2", "\u03c2"], ["\u03f5", "\u03b5"], ["\u0439", "\u0438"], ["\u0450", "\u0435"], ["\u0451", "\u0435"], ["\u0453", "\u0433"], ["\u0457", "\u0456"], ["\u045c", "\u043a"], ["\u045d", "\u0438"], ["\u045e", +"\u0443"], ["\u0477", "\u0475"], ["\u04c2", "\u0436"], ["\u04d1", "\u0430"], ["\u04d3", "\u0430"], ["\u04d7", "\u0435"], ["\u04db", "\u04d9"], ["\u04dd", "\u0436"], ["\u04df", "\u0437"], ["\u04e3", "\u0438"], ["\u04e5", "\u0438"], ["\u04e7", "\u043e"], ["\u04eb", "\u04e9"], ["\u04ed", "\u044d"], ["\u04ef", "\u0443"], ["\u04f1", "\u0443"], ["\u04f3", "\u0443"], ["\u04f5", "\u0447"]]); +function E(a = {}) { + if (!(this instanceof E)) { + return new E(...arguments); + } + for (a = 0; a < arguments.length; a++) { + this.assign(arguments[a]); + } +} +E.prototype.assign = function(a) { + this.normalize = y(a.normalize, !0, this.normalize); + let b = a.ga, c = b || a.na || a.split; + if ("object" === typeof c) { + let d = !b, e = ""; + a.ga || (e += "\\p{Z}"); + c.ka && (e += "\\p{L}"); + c.oa && (e += "\\p{N}", d = !!b); + c.qa && (e += "\\p{S}"); + c.pa && (e += "\\p{P}"); + c.control && (e += "\\p{C}"); + if (c = c.char) { + e += "object" === typeof c ? c.join("") : c; + } + this.split = new RegExp("[" + (b ? "^" : "") + e + "]+", "u"); + this.numeric = d; + } else { + this.split = y(c, ea, this.split), this.numeric = y(this.numeric, !0); + } + this.$ = y(a.$, null, this.$); + this.V = y(a.V, null, this.V); + this.rtl = a.rtl || !1; + this.C = y(a.C, !0, this.C); + this.filter = y((c = a.filter) && new Set(c), null, this.filter); + this.J = y((c = a.J) && new Map(c), null, this.J); + this.D = y((c = a.D) && new Map(c), null, this.D); + this.N = y((c = a.N) && new Map(c), null, this.N); + this.K = y(a.K, null, this.K); + this.Z = y(a.Z, 1, this.Z); + this.ha = y(a.ha, 0, this.ha); + if (this.cache = c = y(a.cache, !0, this.cache)) { + this.X = null, this.ea = "number" === typeof c ? c : 2e5, this.P = new Map(), this.U = new Map(), this.B = this.h = 128; + } + this.F = ""; + this.ba = null; + this.Y = ""; + this.ca = null; + if (this.J) { + for (const d of this.J.keys()) { + this.F += (this.F ? "|" : "") + d; + } + } + if (this.N) { + for (const d of this.N.keys()) { + this.Y += (this.Y ? "|" : "") + d; + } + } + return this; +}; +E.prototype.encode = function(a) { + if (this.cache && a.length <= this.h) { + if (this.X) { + if (this.P.has(a)) { + return this.P.get(a); } } else { - b = b[e] || (b[e] = x()), e = c[++d], V(a, b, c, d, e); + this.X = setTimeout(la, 0, this); + } + } + this.normalize && ("function" === typeof this.normalize ? a = this.normalize(a) : ja ? a = a.normalize("NFKD").replace(ja, "").toLowerCase() : (a = a.toLowerCase(), this.D = this.D ? new Map([...ka, ...this.D]) : new Map(ka))); + this.$ && (a = this.$(a)); + this.numeric && 3 < a.length && (a = a.replace(ha, "$1 $2").replace(ia, "$1 $2").replace(fa, "$1 ")); + const b = !(this.C || this.D || this.filter || this.J || this.N || this.K); + let c = [], d = this.split || "" === this.split ? a.split(this.split) : a; + for (let f = 0, g, h; f < d.length; f++) { + if (!(g = h = d[f])) { + continue; + } + if (g.length < this.Z) { + continue; + } + if (b) { + c.push(g); + continue; + } + if (this.filter && this.filter.has(g)) { + continue; + } + if (this.cache && g.length <= this.B) { + if (this.X) { + var e = this.U.get(g); + if (e || "" === e) { + e && c.push(e); + continue; + } + } else { + this.X = setTimeout(la, 0, this); + } + } + let k; + this.N && 2 < g.length && (this.ca || (this.ca = new RegExp("(?!^)(" + this.Y + ")$")), g = g.replace(this.ca, l => this.N.get(l)), k = 1); + this.J && 1 < g.length && (this.ba || (this.ba = new RegExp("(" + this.F + ")", "g")), g = g.replace(this.ba, l => this.J.get(l)), k = 1); + g && k && (g.length < this.Z || this.filter && this.filter.has(g)) && (g = ""); + if (g && (this.D || this.C && 1 < g.length)) { + e = ""; + for (let l = 0, m = "", n, q; l < g.length; l++) { + n = g.charAt(l), n === m && this.C || ((q = this.D && this.D.get(n)) || "" === q ? q === m && this.C || !(m = q) || (e += q) : e += m = n); + } + g = e; + } + if (g && this.K) { + for (e = 0; g && e < this.K.length; e += 2) { + g = g.replace(this.K[e], this.K[e + 1]); + } + } + this.cache && h.length <= this.B && (this.U.set(h, g), this.U.size > this.ea && (this.U.clear(), this.B = this.B / 1.1 | 0)); + g && c.push(g); + } + this.V && (c = this.V(c) || c); + this.cache && a.length <= this.h && (this.P.set(a, c), this.P.size > this.ea && (this.P.clear(), this.h = this.h / 1.1 | 0)); + return c; +}; +function la(a) { + a.X = null; + a.P.clear(); + a.U.clear(); +} +;function ma(a, b, c) { + a = ("object" === typeof a ? "" + a.query : a).toLowerCase(); + let d = this.cache.get(a); + if (!d) { + d = this.search(a, b, c); + if (d instanceof Promise) { + const e = this; + d.then(function(f) { + e.cache.set(a, f); + }); + } + this.cache.set(a, d); + } + return d; +} +function F(a) { + this.limit = a && !0 !== a ? a : 1000; + this.cache = new Map(); + this.h = ""; +} +F.prototype.set = function(a, b) { + this.cache.has(a) || (this.cache.set(this.h = a, b), this.limit && this.cache.size > this.limit && this.cache.delete(this.cache.keys().next().value)); +}; +F.prototype.get = function(a) { + const b = this.cache.get(a); + b && this.limit && this.h !== a && (this.cache.delete(a), this.cache.set(this.h = a, b)); + return b; +}; +F.prototype.remove = function(a) { + for (const b of this.cache) { + const c = b[0]; + b[1].includes(a) && this.cache.delete(c); + } +}; +F.prototype.clear = function() { + this.cache.clear(); + this.h = ""; +}; +function na(a, b, c, d) { + let e = []; + for (let f = 0, g; f < a.index.length; f++) { + if (g = a.index[f], b >= g.length) { + b -= g.length; + } else { + b = g[d ? "splice" : "slice"](b, c); + const h = b.length; + if (h && (e = e.length ? e.concat(b) : b, c -= h, d && (a.length -= h), !c)) { + break; + } + b = 0; + } + } + return e; +} +function I(a) { + if (!(this instanceof I)) { + return new I(a); + } + this.index = a ? [a] : []; + this.length = a ? a.length : 0; + const b = this; + return new Proxy([], {get(c, d) { + if ("length" === d) { + return b.length; + } + if ("push" === d) { + return function(e) { + b.index[b.index.length - 1].push(e); + b.length++; + }; + } + if ("pop" === d) { + return function() { + if (b.length) { + return b.length--, b.index[b.index.length - 1].pop(); + } + }; + } + if ("indexOf" === d) { + return function(e) { + let f = 0; + for (let g = 0, h, k; g < b.index.length; g++) { + h = b.index[g]; + k = h.indexOf(e); + if (0 <= k) { + return f + k; + } + f += h.length; + } + return -1; + }; + } + if ("includes" === d) { + return function(e) { + for (let f = 0; f < b.index.length; f++) { + if (b.index[f].includes(e)) { + return !0; + } + } + return !1; + }; + } + if ("slice" === d) { + return function(e, f) { + return na(b, e || 0, f || b.length, !1); + }; + } + if ("splice" === d) { + return function(e, f) { + return na(b, e || 0, f || b.length, !0); + }; + } + if ("constructor" === d) { + return Array; + } + if ("symbol" !== typeof d) { + return (c = b.index[d / 2 ** 31 | 0]) && c[d]; + } + }, set(c, d, e) { + c = d / 2 ** 31 | 0; + (b.index[c] || (b.index[c] = []))[d] = e; + b.length++; + return !0; + }}); +} +I.prototype.clear = function() { + this.index.length = 0; +}; +I.prototype.push = function() { +}; +function L(a = 8) { + if (!(this instanceof L)) { + return new L(a); + } + this.index = z(); + this.F = []; + this.size = 0; + 32 < a ? (this.h = oa, this.B = BigInt(a)) : (this.h = pa, this.B = a); +} +L.prototype.get = function(a) { + const b = this.index[this.h(a)]; + return b && b.get(a); +}; +L.prototype.set = function(a, b) { + var c = this.h(a); + let d = this.index[c]; + d ? (c = d.size, d.set(a, b), (c -= d.size) && this.size++) : (this.index[c] = d = new Map([[a, b]]), this.F.push(d)); +}; +function M(a = 8) { + if (!(this instanceof M)) { + return new M(a); + } + this.index = z(); + this.h = []; + 32 < a ? (this.F = oa, this.B = BigInt(a)) : (this.F = pa, this.B = a); +} +M.prototype.add = function(a) { + var b = this.F(a); + let c = this.index[b]; + c ? (b = c.size, c.add(a), (b -= c.size) && this.size++) : (this.index[b] = c = new Set([a]), this.h.push(c)); +}; +u = L.prototype; +u.has = M.prototype.has = function(a) { + const b = this.index[this.F(a)]; + return b && b.has(a); +}; +u.delete = M.prototype.delete = function(a) { + const b = this.index[this.F(a)]; + b && b.delete(a) && this.size--; +}; +u.clear = M.prototype.clear = function() { + this.index = z(); + this.h = []; + this.size = 0; +}; +u.values = M.prototype.values = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let b of this.h[a].values()) { + yield b; + } + } +}; +u.keys = M.prototype.keys = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let b of this.h[a].keys()) { + yield b; + } + } +}; +u.entries = M.prototype.entries = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let b of this.h[a].entries()) { + yield b; + } + } +}; +function pa(a) { + let b = 2 ** this.B - 1; + if ("number" == typeof a) { + return a & b; + } + let c = 0, d = this.B + 1; + for (let e = 0; e < a.length; e++) { + c = (c * d ^ a.charCodeAt(e)) & b; + } + return 32 === this.B ? c + 2 ** 31 : c; +} +function oa(a) { + let b = BigInt(2) ** this.B - BigInt(1); + var c = typeof a; + if ("bigint" === c) { + return a & b; + } + if ("number" === c) { + return BigInt(a) & b; + } + c = BigInt(0); + let d = this.B + BigInt(1); + for (let e = 0; e < a.length; e++) { + c = (c * d ^ BigInt(a.charCodeAt(e))) & b; + } + return c; +} +;function qa(a, b, c, d, e, f, g, h) { + (d = a(c ? c + "." + d : d, JSON.stringify(g))) && d.then ? d.then(function() { + b.export(a, b, c, e, f + 1, h); + }) : b.export(a, b, c, e, f + 1, h); +} +;const ra = z(), N = z(); +var sa = {normalize:function(a) { + return a.toLowerCase(); +}, C:!1}; +const ta = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +function ua(a) { + O.call(a, "add"); + O.call(a, "append"); + O.call(a, "search"); + O.call(a, "update"); + O.call(a, "remove"); +} +function O(a) { + this[a + "Async"] = function() { + var b = arguments; + const c = b[b.length - 1]; + let d; + "function" === typeof c && (d = c, delete b[b.length - 1]); + this.async = !0; + b = this[a].apply(this, b); + this.async = !1; + b.then ? b.then(d) : d(b); + return b; + }; +} +;z(); +P.prototype.add = function(a, b, c, d) { + if (b && (a || 0 === a)) { + if (!d && !c && this.A.has(a)) { + return this.update(a, b); + } + b = this.encoder.encode(b); + if (d = b.length) { + const l = z(), m = z(), n = this.depth, q = this.resolution; + for (let t = 0; t < d; t++) { + let p = b[this.rtl ? d - 1 - t : t]; + var e = p.length; + if (e && (n || !m[p])) { + var f = this.score ? this.score(b, p, t, null, 0) : Q(q, d, t), g = ""; + switch(this.tokenize) { + case "full": + if (2 < e) { + for (f = 0; f < e; f++) { + for (var h = e; h > f; h--) { + g = p.substring(f, h); + var k = this.score ? this.score(b, p, t, g, f) : Q(q, d, t, e, f); + R(this, m, g, k, a, c); + } + } + break; + } + case "reverse": + if (1 < e) { + for (h = e - 1; 0 < h; h--) { + g = p[h] + g, k = this.score ? this.score(b, p, t, g, h) : Q(q, d, t, e, h), R(this, m, g, k, a, c); + } + g = ""; + } + case "forward": + if (1 < e) { + for (h = 0; h < e; h++) { + g += p[h], R(this, m, g, f, a, c); + } + break; + } + default: + if (R(this, m, p, f, a, c), n && 1 < d && t < d - 1) { + for (e = z(), g = this.da, f = p, h = Math.min(n + 1, d - t), e[f] = 1, k = 1; k < h; k++) { + if ((p = b[this.rtl ? d - 1 - t - k : t + k]) && !e[p]) { + e[p] = 1; + const r = this.score ? this.score(b, f, t, p, k) : Q(g + (d / 2 > g ? 0 : 1), d, t, h - 1, k - 1), x = this.bidirectional && p > f; + R(this, l, x ? f : p, r, a, c, x ? p : f); + } + } + } + } + } + } + this.fastupdate || this.A.add(a); + } else { + b = ""; + } + } + this.db && (b || this.R.push({del:a}), this.fa && va(this)); + return this; +}; +function R(a, b, c, d, e, f, g) { + let h = g ? a.I : a.map, k; + if (!b[c] || !g || !(k = b[c])[g]) { + if (g ? (b = k || (b[c] = z()), b[g] = 1, (k = h.get(g)) ? h = k : h.set(g, h = new Map())) : b[c] = 1, (k = h.get(c)) ? h = k : h.set(c, h = k = []), h = h[d] || (h[d] = []), !f || !h.includes(e)) { + if (h.length === 2 ** 31 - 1) { + b = new I(h); + if (a.fastupdate) { + for (let l of a.A.values()) { + l.includes(h) && (l[l.indexOf(h)] = b); + } + } + k[d] = h = b; + } + h.push(e); + a.fastupdate && ((d = a.A.get(e)) ? d.push(h) : a.A.set(e, [h])); } } } -function X(a, b, c, d, e, f, h, g) { - if (a = a[h]) { +function Q(a, b, c, d, e) { + return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; +} +;function S(a, b, c, d) { + if (1 === a.length) { + return a = a[0], a = c || a.length > b ? b ? a.slice(c, c + b) : a.slice(c) : a, d ? wa(a) : a; + } + let e = []; + for (let f = 0, g, h; f < a.length; f++) { + if ((g = a[f]) && (h = g.length)) { + if (c) { + if (c >= h) { + c -= h; + continue; + } + c < h && (g = b ? g.slice(c, c + b) : g.slice(c), h = g.length, c = 0); + } + if (e.length) { + h > b && (g = g.slice(0, b), h = g.length), e.push(g); + } else { + if (h >= b) { + return h > b && (g = g.slice(0, b)), d ? wa(g) : g; + } + e = [g]; + } + b -= h; + if (!b) { + break; + } + } + } + if (!e.length) { + return e; + } + e = 1 < e.length ? [].concat.apply([], e) : e[0]; + return d ? wa(e) : e; +} +function wa(a) { + for (let b = 0; b < a.length; b++) { + a[b] = {score:b, id:a[b]}; + } + return a; +} +;T.prototype.or = function() { + const a = this; + let b = arguments; + var c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.or.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.or.apply(this, c); + } + let d = []; + c = []; + let e = 0, f = 0, g, h; + for (let k = 0, l; k < b.length; k++) { + if (l = b[k]) { + let m; + if (l instanceof T) { + m = l.result; + } else if (l.constructor === Array) { + m = l; + } else if (l.index) { + l.resolve = !1, m = l.index.search(l).result; + } else if (l.and) { + m = this.and(l.and); + } else if (l.xor) { + m = this.xor(l.xor); + } else if (l.G) { + m = this.G(l.G); + } else { + e = l.limit || 0; + f = l.offset || 0; + g = l.enrich; + h = l.resolve; + continue; + } + d[k] = m; + m instanceof Promise && c.push(m); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result.length && (d = [a.result].concat(d)); + a.result = xa(d, e, f, g, h, a.O); + return h ? a.result : a; + }); + } + this.result.length && (d = [this.result].concat(d)); + this.result = xa(d, e, f, g, h, a.O); + return h ? this.result : this; +}; +function xa(a, b, c, d, e, f) { + if (!a.length) { + return a; + } + "object" === typeof b && (c = b.offset || 0, d = b.enrich || !1, b = b.limit || 0); + if (2 > a.length) { + return e ? S(a[0], b, c, d) : a[0]; + } + d = []; + let g = 0, h = z(), k = da(a); + for (let l = 0, m; l < k; l++) { + for (let n = 0, q; n < a.length; n++) { + if (q = a[n]) { + if (m = q[l]) { + for (let t = 0, p; t < m.length; t++) { + if (p = m[t], !h[p]) { + if (h[p] = 1, c) { + c--; + } else { + if (e) { + d.push(p); + } else { + const r = l + (n ? f : 0); + d[r] || (d[r] = []); + d[r].push(p); + } + if (b && ++g === b) { + return d; + } + } + } + } + } + } + } + } + return d; +} +;T.prototype.and = function() { + if (this.result.length) { + const b = this; + let c = arguments; + var a = c[0]; + if (a instanceof Promise) { + return a.then(function() { + return b.and.apply(b, c); + }); + } + if (a[0] && a[0].index) { + return this.and.apply(this, a); + } + let d = []; + a = []; + let e = 0, f = 0, g; + for (let h = 0, k; h < c.length; h++) { + if (k = c[h]) { + let l; + if (k instanceof T) { + l = k.result; + } else if (k.constructor === Array) { + l = k; + } else if (k.index) { + k.resolve = !1, l = k.index.search(k).result; + } else if (k.or) { + l = this.or(k.or); + } else if (k.xor) { + l = this.xor(k.xor); + } else if (k.G) { + l = this.G(k.G); + } else { + e = k.limit || 0; + f = k.offset || 0; + g = k.resolve; + continue; + } + d[h] = l; + l instanceof Promise && a.push(l); + } + } + if (a.length) { + return Promise.all(a).then(function() { + d = [b.result].concat(d); + b.result = ya(d, e, f, g, b.O); + return g ? b.result : b; + }); + } + d = [this.result].concat(d); + this.result = ya(d, e, f, g, b.O); + return g ? this.result : this; + } + return this; +}; +function ya(a, b, c, d, e) { + if (2 > a.length) { + return []; + } + let f = [], g = 0, h = z(), k = da(a); + if (!k) { + return f; + } + for (let l = 0, m; l < a.length; l++) { + m = a[l]; + if (!m || !m.length) { + return []; + } + let n = z(), q = 0, t = l === a.length - 1; + for (let p = 0, r; p < k; p++) { + if (r = m[p]) { + for (let x = 0, A, w; x < r.length; x++) { + if (A = r[x], !l) { + n[A] = p + 1 + (l ? e : 0), q = 1; + } else if (t) { + if (w = h[A]) { + if (q = 1, c) { + c--; + } else { + if (d ? f.push(A) : (w--, p < w && (w = p), f[w] || (f[w] = []), f[w].push(A)), b && ++g === b) { + return f; + } + } + } + } else if (w = h[A]) { + p + 1 < w && (w = p + 1), n[A] = w, q = 1; + } + } + } + } + if (!q) { + return []; + } + h = n; + } + return f; +} +;T.prototype.xor = function() { + const a = this; + let b = arguments; + var c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.xor.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.xor.apply(this, c); + } + let d = []; + c = []; + let e = 0, f = 0, g, h; + for (let k = 0, l; k < b.length; k++) { + if (l = b[k]) { + let m; + if (l instanceof T) { + m = l.result; + } else if (l.constructor === Array) { + m = l; + } else if (l.index) { + l.resolve = !1, m = l.index.search(l).result; + } else if (l.or) { + m = this.or(l.or); + } else if (l.and) { + m = this.and(l.and); + } else if (l.G) { + m = this.G(l.G); + } else { + e = l.limit || 0; + f = l.offset || 0; + g = l.enrich; + h = l.resolve; + continue; + } + d[k] = m; + m instanceof Promise && c.push(m); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result.length && (d = [a.result].concat(d)); + a.result = za(d, e, f, g, !h, a.O); + return h ? a.result : a; + }); + } + this.result.length && (d = [this.result].concat(d)); + this.result = za(d, e, f, g, !h, a.O); + return h ? this.result : this; +}; +function za(a, b, c, d, e, f) { + if (!a.length) { + return a; + } + if (2 > a.length) { + return e ? S(a[0], b, c, d) : a[0]; + } + b = []; + c = z(); + for (let g = 0, h; g < a.length; g++) { + if (h = a[g]) { + for (let k = 0, l; k < h.length; k++) { + if (l = h[k]) { + for (let m = 0, n; m < l.length; m++) { + n = l[m], c[n] ? c[n]++ : c[n] = 1; + } + } + } + } + } + for (let g = 0, h; g < a.length; g++) { + if (h = a[g]) { + for (let k = 0, l; k < h.length; k++) { + if (l = h[k]) { + for (let m = 0, n; m < l.length; m++) { + n = l[m], 1 === c[n] && (e ? b.push(n) : (d = k + (g ? f : 0), b[d] || (b[d] = []), b[d].push(n))); + } + } + } + } + } + return b; +} +;T.prototype.G = function() { + const a = this; + let b = arguments; + var c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.G.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.G.apply(this, c); + } + let d = []; + c = []; + let e; + for (let f = 0, g; f < b.length; f++) { + if (g = b[f]) { + let h; + if (g instanceof T) { + h = g.result; + } else if (g.constructor === Array) { + h = g; + } else if (g.index) { + g.resolve = !1, h = g.index.search(g).result; + } else if (g.or) { + h = this.or(g.or); + } else if (g.and) { + h = this.and(g.and); + } else if (g.xor) { + h = this.xor(g.xor); + } else { + e = g.resolve; + continue; + } + d[f] = h; + h instanceof Promise && c.push(h); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result = Aa.call(a, d, e); + return e ? a.result : a; + }); + } + this.result = Aa.call(this, d, e); + return e ? this.result : this; +}; +function Aa(a, b) { + if (!a.length) { + return this.result; + } + const c = []; + a = new Set(a.flat().flat()); + for (let d = 0, e; d < this.result.length; d++) { + if (e = this.result[d]) { + for (let f = 0, g; f < e.length; f++) { + g = e[f], a.has(g) || (b ? c.push(g) : (c[d] || (c[d] = []), c[d].push(g))); + } + } + } + return c; +} +;function T(a) { + if (a && a.index) { + return a.resolve = !1, this.index = a.index, a.index.search(a); + } + if (!(this instanceof T)) { + return new T(a); + } + if (a instanceof T) { + return a; + } + this.index = null; + this.result = a || []; + this.O = 0; +} +T.prototype.limit = function(a) { + if (this.result.length) { + const b = []; + let c = 0; + for (let d = 0, e; d < this.result.length; d++) { + if (e = this.result[d], e.length + c < a) { + b[d] = e, c += e.length; + } else { + b[d] = e.slice(0, a - c); + this.result = b; + break; + } + } + } + return this; +}; +T.prototype.offset = function(a) { + if (this.result.length) { + const b = []; + let c = 0; + for (let d = 0, e; d < this.result.length; d++) { + e = this.result[d], e.length + c < a ? c += e.length : (b[d] = e.slice(a - c), c = a); + } + this.result = b; + } + return this; +}; +T.prototype.resolve = function(a, b, c) { + U = 1; + const d = this.result; + this.result = this.index = null; + return d.length ? ("object" === typeof a && (c = a.enrich, b = a.offset, a = a.limit), S(d, a || 100, b, c)) : d; +}; +function Ba(a, b, c, d) { + var e = a.length; + let f = [], g = 0, h, k, l; + d && (d = []); + for (let m = e - 1, n; 0 <= m; m--) { + l = a[m]; + e = z(); + n = !h; + for (let q = 0, t; q < l.length; q++) { + if ((t = l[q]) && t.length) { + for (let p = 0, r; p < t.length; p++) { + if (r = t[p], h) { + if (h[r]) { + if (!m) { + if (c) { + c--; + } else { + if (f[g++] = r, g === b) { + return f; + } + } + } + if (m || d) { + e[r] = 1; + } + n = !0; + } + d && !k[r] && (k[r] = 1, (d[q] || (d[q] = [])).push(r)); + } else { + e[r] = 1; + } + } + } + } + if (d) { + h || (k = e); + } else if (!n) { + return []; + } + h = e; + } + if (d) { + for (let m = d.length - 1, n, q; 0 <= m; m--) { + n = d[m]; + q = n.length; + for (let t = 0, p; t < q; t++) { + if (p = n[t], !h[p]) { + if (c) { + c--; + } else { + if (f[g++] = p, g === b) { + return f; + } + } + h[p] = 1; + } + } + } + } + return f; +} +function Ca(a, b) { + const c = z(), d = z(), e = []; + for (let f = 0; f < a.length; f++) { + c[a[f]] = 1; + } + for (let f = 0, g; f < b.length; f++) { + g = b[f]; + for (let h = 0, k; h < g.length; h++) { + k = g[h], c[k] && !d[k] && (d[k] = 1, e.push(k)); + } + } + return e; +} +;let U = 1; +P.prototype.search = function(a, b, c) { + c || (!b && C(a) ? (c = a, a = "") : C(b) && (c = b, b = 0)); + let d = [], e; + let f, g = 0, h, k, l; + if (c) { + a = c.query || a; + b = c.limit || b; + g = c.offset || 0; + var m = c.context; + f = c.suggest; + (h = U && !1 !== c.resolve) || (U = 0); + k = h && c.enrich; + l = this.db && c.tag; + } else { + h = this.resolve || U; + } + a = this.encoder.encode(a); + e = a.length; + b || !h || (b = 100); + if (1 === e) { + return Da.call(this, a[0], "", b, g, h, k, l); + } + m = this.depth && !1 !== m; + if (2 === e && m && !f) { + return Da.call(this, a[0], a[1], b, g, h, k, l); + } + let n = c = 0; + if (1 < e) { + const p = z(), r = []; + for (let x = 0, A; x < e; x++) { + if ((A = a[x]) && !p[A]) { + if (f || this.db || V(this, A)) { + r.push(A), p[A] = 1; + } else { + return h ? d : new T(d); + } + const w = A.length; + c = Math.max(c, w); + n = n ? Math.min(n, w) : w; + } + } + a = r; + e = a.length; + } + if (!e) { + return h ? d : new T(d); + } + let q = 0, t; + if (1 === e) { + return Da.call(this, a[0], "", b, g, h, k, l); + } + if (2 === e && m && !f) { + return Da.call(this, a[0], a[1], b, g, h, k, l); + } + 1 < e && (m ? (t = a[0], q = 1) : 9 < c && 3 < c / n && a.sort(aa)); + if (this.db) { + if (this.db.search && (m = this.db.search(this, a, b, g, f, h, k, l), !1 !== m)) { + return m; + } + const p = this; + return async function() { + for (let r, x; q < e; q++) { + x = a[q]; + t ? (r = await V(p, x, t), r = Ea(r, d, f, p.da, b, g, 2 === e), f && !1 === r && d.length || (t = x)) : (r = await V(p, x), r = Ea(r, d, f, p.resolution, b, g, 1 === e)); + if (r) { + return r; + } + if (f && q === e - 1) { + let A = d.length; + if (!A) { + if (t) { + t = ""; + q = -1; + continue; + } + return d; + } + if (1 === A) { + return h ? S(d[0], b, g) : new T(d[0]); + } + } + } + return h ? Ba(d, b, g, f) : new T(d[0]); + }(); + } + for (let p, r; q < e; q++) { + r = a[q]; + t ? (p = V(this, r, t), p = Ea(p, d, f, this.da, b, g, 2 === e), f && !1 === p && d.length || (t = r)) : (p = V(this, r), p = Ea(p, d, f, this.resolution, b, g, 1 === e)); + if (p) { + return p; + } + if (f && q === e - 1) { + m = d.length; + if (!m) { + if (t) { + t = ""; + q = -1; + continue; + } + return d; + } + if (1 === m) { + return h ? S(d[0], b, g) : new T(d[0]); + } + } + } + return h ? Ba(d, b, g, f) : new T(d[0]); +}; +function Da(a, b, c, d, e, f, g) { + a = V(this, a, b, c, d, e, f, g); + return this.db ? a.then(function(h) { + return e ? h : h && h.length ? e ? S(h, c, d) : new T(h) : e ? [] : new T([]); + }) : a && a.length ? e ? S(a, c, d) : new T(a) : e ? [] : new T([]); +} +function Ea(a, b, c, d, e, f, g) { + let h = []; + if (a) { + d = Math.min(a.length, d); + for (let k = 0, l = 0, m; k < d; k++) { + if (m = a[k]) { + if (f && m && g && (m.length <= f ? (f -= m.length, m = null) : (m = m.slice(f), f = 0)), m && (h[k] = m, g && (l += m.length, l >= e))) { + break; + } + } + } + if (h.length) { + if (g) { + return S(h, e, 0); + } + b.push(h); + return; + } + } + return !c && h; +} +function V(a, b, c, d, e, f, g, h) { + let k; + c && (k = a.bidirectional && b > c); + if (a.db) { + return c ? a.db.get(k ? c : b, k ? b : c, d, e, f, g, h) : a.db.get(b, "", d, e, f, g, h); + } + a = c ? (a = a.I.get(k ? b : c)) && a.get(k ? c : b) : a.map.get(b); + return a; +} +;P.prototype.remove = function(a, b) { + const c = this.A.size && (this.fastupdate ? this.A.get(a) : this.A.has(a)); + if (c) { + if (this.fastupdate) { + for (let d = 0, e; d < c.length; d++) { + if (e = c[d]) { + if (2 > e.length) { + e.pop(); + } else { + const f = e.indexOf(a); + f === c.length - 1 ? e.pop() : e.splice(f, 1); + } + } + } + } else { + Fa(this.map, a), this.depth && Fa(this.I, a); + } + b || this.A.delete(a); + } + this.db && (this.R.push({del:a}), this.fa && va(this)); + this.cache && this.cache.remove(a); + return this; +}; +function Fa(a, b) { + let c = 0; + if (a.constructor === Array) { + for (let d = 0, e, f; d < a.length; d++) { + if ((e = a[d]) && e.length) { + if (f = e.indexOf(b), 0 <= f) { + 1 < e.length ? (e.splice(f, 1), c++) : delete a[d]; + break; + } else { + c++; + } + } + } + } else { + for (let d of a) { + const e = d[0], f = Fa(d[1], b); + f ? c += f : a.delete(e); + } + } + return c; +} +;function P(a, b) { + if (!(this instanceof P)) { + return new P(a); + } + if (a) { + var c = B(a) ? a : a.preset; + c && (ta[c] || console.warn("Preset not found: " + c), a = Object.assign({}, ta[c], a)); + } else { + a = {}; + } + c = a.context || {}; + const d = a.encode || a.encoder || sa; + this.encoder = d.encode ? d : "object" === typeof d ? new E(d) : {encode:d}; + let e; + this.resolution = a.resolution || 9; + this.tokenize = e = a.tokenize || "strict"; + this.depth = "strict" === e && c.depth || 0; + this.bidirectional = !1 !== c.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + (e = a.keystore || 0) && (this.keystore = e); + this.map = e ? new L(e) : new Map(); + this.I = e ? new L(e) : new Map(); + this.A = b || (this.fastupdate ? e ? new L(e) : new Map() : e ? new M(e) : new Set()); + this.da = c.resolution || 1; + this.rtl = d.rtl || a.rtl || !1; + this.cache = (e = a.cache || null) && new F(e); + this.resolve = !1 !== a.resolve; + if (e = a.db) { + this.db = e.mount(this); + } + this.fa = !1 !== a.commit; + this.R = []; + this.h = null; +} +u = P.prototype; +u.mount = function(a) { + this.h && (clearTimeout(this.h), this.h = null); + return a.mount(this); +}; +u.commit = function(a, b) { + this.h && (clearTimeout(this.h), this.h = null); + return this.db.commit(this, a, b); +}; +function va(a) { + a.h || (a.h = setTimeout(function() { + a.h = null; + a.db.commit(a, void 0, void 0); + }, 0)); +} +u.clear = function() { + this.map.clear(); + this.I.clear(); + this.A.clear(); + this.cache && this.cache.clear(); + this.db && (this.h && clearTimeout(this.h), this.h = null, this.R = [{clear:!0}]); + return this; +}; +u.append = function(a, b) { + return this.add(a, b, !0); +}; +u.contain = function(a) { + return this.db ? this.db.has(a) : this.A.has(a); +}; +u.update = function(a, b) { + if (this.async) { + const c = this, d = this.remove(a); + return d.then ? d.then(() => c.add(a, b)) : this.add(a, b); + } + return this.remove(a).add(a, b); +}; +function Ga(a) { + let b = 0; + if (a.constructor === Array) { + for (let c = 0, d; c < a.length; c++) { + (d = a[c]) && (b += d.length); + } + } else { + for (const c of a) { + const d = c[0], e = Ga(c[1]); + e ? b += e : a.delete(d); + } + } + return b; +} +u.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + Ga(this.map); + this.depth && Ga(this.I); + return this; +}; +u.searchCache = ma; +u.export = function(a, b, c, d, e, f) { + let g = !0; + "undefined" === typeof f && (g = new Promise(l => { + f = l; + })); + let h, k; + switch(e || (e = 0)) { + case 0: + h = "reg"; + if (this.fastupdate) { + k = z(); + for (let l of this.A.keys()) { + k[l] = 1; + } + } else { + k = this.A; + } + break; + case 1: + h = "cfg"; + k = {doc:0, opt:this.B ? 1 : 0}; + break; + case 2: + h = "map"; + k = this.map; + break; + case 3: + h = "ctx"; + k = this.I; + break; + default: + "undefined" === typeof c && f && f(); + return; + } + qa(a, b || this, c, h, d, e, k, f); + return g; +}; +u.import = function(a, b) { + if (b) { + switch(B(b) && (b = JSON.parse(b)), a) { + case "cfg": + this.B = !!b.opt; + break; + case "reg": + this.fastupdate = !1; + this.A = b; + break; + case "map": + this.map = b; + break; + case "ctx": + this.I = b; + } + } +}; +ua(P.prototype); +async function Ha(a) { + a = a.data; + var b = self._index; + const c = a.args; + var d = a.task; + switch(d) { + case "init": + d = a.options || {}; + (b = d.config) && (d = b); + (b = a.factory) ? (Function("return " + b)()(self), self._index = new self.FlexSearch.Index(d), delete self.FlexSearch) : self._index = new P(d); + postMessage({id:a.id}); + break; + default: + a = a.id, b = b[d].apply(b, c), postMessage("search" === d ? {id:a, msg:b} : {id:a}); + } +} +;let Ia = 0; +function W(a) { + function b(f) { + f = f.data || f; + const g = f.id, h = g && e.h[g]; + h && (h(f.msg), delete e.h[g]); + } + if (!(this instanceof W)) { + return new W(a); + } + a || (a = {}); + let c = (self || window)._factory; + c && (c = c.toString()); + const d = "undefined" === typeof window && self.exports, e = this; + this.worker = Ja(c, d, a.worker); + this.h = z(); + if (this.worker) { + d ? this.worker.on("message", b) : this.worker.onmessage = b; + if (a.config) { + return new Promise(function(f) { + e.h[++Ia] = function() { + f(e); + }; + e.worker.postMessage({id:Ia, task:"init", factory:c, options:a}); + }); + } + this.worker.postMessage({task:"init", factory:c, options:a}); + } +} +X("add"); +X("append"); +X("search"); +X("update"); +X("remove"); +function X(a) { + W.prototype[a] = W.prototype[a + "Async"] = function() { + const b = this, c = [].slice.call(arguments); + var d = c[c.length - 1]; + let e; + "function" === typeof d && (e = d, c.splice(c.length - 1, 1)); + d = new Promise(function(f) { + b.h[++Ia] = f; + b.worker.postMessage({task:a, id:Ia, args:c}); + }); + return e ? (d.then(e), this) : d; + }; +} +function Ja(a, b, c) { + return b ? new (require("worker_threads")["Worker"])(__dirname + "/node/node.js") : a ? new window.Worker(URL.createObjectURL(new Blob(["onmessage=" + Ha.toString()], {type:"text/javascript"}))) : new window.Worker(B(c) ? c : "worker/worker.js", {type:"module"}); +} +;Y.prototype.add = function(a, b, c) { + C(a) && (b = a, a = ca(b, this.key)); + if (b && (a || 0 === a)) { + if (!c && this.A.has(a)) { + return this.update(a, b); + } + for (let h = 0, k; h < this.field.length; h++) { + k = this.M[h]; + var d = this.index.get(this.field[h]); + if ("function" === typeof k) { + var e = k(b); + e && d.add(a, e, !1, !0); + } else { + if (e = k.T, !e || e(b)) { + k instanceof String ? k = ["" + k] : B(k) && (k = [k]), Ka(b, k, this.W, 0, d, a, k[0], c); + } + } + } + if (this.tag) { + for (d = 0; d < this.L.length; d++) { + var f = this.L[d], g = this.aa[d]; + e = this.tag.get(g); + let h = z(); + if ("function" === typeof f) { + if (f = f(b), !f) { + continue; + } + } else { + const k = f.T; + if (k && !k(b)) { + continue; + } + f instanceof String && (f = "" + f); + f = ca(b, f); + } + if (e && f) { + B(f) && (f = [f]); + for (let k = 0, l, m; k < f.length; k++) { + if (l = f[k], !h[l] && (h[l] = 1, (g = e.get(l)) ? m = g : e.set(l, m = []), !c || !m.includes(a))) { + if (m.length === 2 ** 31 - 1) { + g = new I(m); + if (this.fastupdate) { + for (let n of this.A.values()) { + n.includes(m) && (n[n.indexOf(m)] = g); + } + } + e.set(l, m = g); + } + m.push(a); + this.fastupdate && ((g = this.A.get(a)) ? g.push(m) : this.A.set(a, [m])); + } + } + } else { + e || console.warn("Tag '" + g + "' was not found"); + } + } + } + if (this.store && (!c || !this.store.has(a))) { + let h; + if (this.H) { + h = z(); + for (let k = 0, l; k < this.H.length; k++) { + l = this.H[k]; + if ((c = l.T) && !c(b)) { + continue; + } + let m; + if ("function" === typeof l) { + m = l(b); + if (!m) { + continue; + } + l = [l.ia]; + } else if (B(l) || l instanceof String) { + h[l] = b[l]; + continue; + } + La(b, h, l, 0, l[0], m); + } + } + this.store.set(a, h || b); + } + } + return this; +}; +function La(a, b, c, d, e, f) { + a = a[e]; + if (d === c.length - 1) { + b[e] = f || a; + } else if (a) { + if (a.constructor === Array) { + for (b = b[e] = Array(a.length), e = 0; e < a.length; e++) { + La(a, b, c, d, e); + } + } else { + b = b[e] || (b[e] = z()), e = c[++d], La(a, b, c, d, e); + } + } +} +function Ka(a, b, c, d, e, f, g, h) { + if (a = a[g]) { if (d === b.length - 1) { if (a.constructor === Array) { if (c[d]) { @@ -684,228 +1558,455 @@ function X(a, b, c, d, e, f, h, g) { } a = a.join(" "); } - e.add(f, a, g, !0); + e.add(f, a, h, !0); } else { if (a.constructor === Array) { - for (h = 0; h < a.length; h++) { - X(a, b, c, d, e, f, h, g); + for (g = 0; g < a.length; g++) { + Ka(a, b, c, d, e, f, g, h); } } else { - h = b[++d], X(a, b, c, d, e, f, h, g); + g = b[++d], Ka(a, b, c, d, e, f, g, h); } } + } else { + e.db && e.remove(f); } } -t = S.prototype; -t.add = function(a, b, c) { - D(a) && (b = a, a = U(b, this.key)); - if (b && (a || 0 === a)) { - if (!c && this.register[a]) { - return this.update(a, b); - } - for (let d = 0, e, f; d < this.h.length; d++) { - f = this.h[d], e = this.K[d], C(e) && (e = [e]), X(b, e, this.A, 0, this.index[f], a, e[0], c); - } - if (this.I) { - let d = U(b, this.I), e = x(); - C(d) && (d = [d]); - for (let f = 0, h, g; f < d.length; f++) { - if (h = d[f], !e[h] && (e[h] = 1, g = this.l[h] || (this.l[h] = []), !c || !g.includes(a))) { - if (g[g.length] = a, this.m) { - const k = this.register[a] || (this.register[a] = []); - k[k.length] = g; - } - } - } - } - if (this.store && (!c || !this.store[a])) { - let d; - if (this.C) { - d = x(); - for (let e = 0, f; e < this.C.length; e++) { - f = this.C[e], C(f) ? d[f] = b[f] : V(b, d, f, 0, f[0]); - } - } - this.store[a] = d || b; - } - } - return this; -}; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a) { - D(a) && (a = U(a, this.key)); - if (this.register[a]) { - for (var b = 0; b < this.h.length && (this.index[this.h[b]].remove(a, !this.o), !this.m); b++) { - } - if (this.I && !this.m) { - for (let c in this.l) { - b = this.l[c]; - const d = b.indexOf(a); - -1 !== d && (1 < b.length ? b.splice(d, 1) : delete this.l[c]); - } - } - this.store && delete this.store[a]; - delete this.register[a]; - } - return this; -}; -t.search = function(a, b, c, d) { - c || (!b && D(a) ? (c = a, a = "") : D(b) && (c = b, b = 0)); - let e = [], f = [], h, g, k, m, n, w, q = 0; +;Y.prototype.search = function(a, b, c, d) { + c || (!b && C(a) ? (c = a, a = "") : C(b) && (c = b, b = 0)); + let e = [], f = [], g; + let h; + let k; + let l, m = 0; if (c) { if (c.constructor === Array) { k = c, c = null; } else { a = c.query || a; - k = (h = c.pluck) || c.index || c.field; - m = c.tag; - g = this.store && c.enrich; - n = "and" === c.bool; - b = c.limit || b || 100; - w = c.offset || 0; - if (m && (C(m) && (m = [m]), !a)) { - for (let l = 0, p; l < m.length; l++) { - if (p = xa.call(this, m[l], b, w, g)) { - e[e.length] = p, q++; + g = c.pluck; + h = c.merge; + k = g || c.field || c.index; + var n = this.tag && c.tag; + var q = this.store && c.enrich; + var t = c.suggest; + b = c.limit || b; + l = c.offset || 0; + b || (b = 100); + if (n && (!this.db || !d)) { + n.constructor !== Array && (n = [n]); + var p = []; + for (let w = 0, v; w < n.length; w++) { + v = n[w]; + if (B(v)) { + throw Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + if (v.field && v.tag) { + var r = v.tag; + if (r.constructor === Array) { + for (var x = 0; x < r.length; x++) { + p.push(v.field, r[x]); + } + } else { + p.push(v.field, r); + } + } else { + r = Object.keys(v); + for (let J = 0, K, D; J < r.length; J++) { + if (K = r[J], D = v[K], D.constructor === Array) { + for (x = 0; x < D.length; x++) { + p.push(K, D[x]); + } + } else { + p.push(K, D); + } + } } } - return q ? e : []; + if (!p.length) { + throw Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + n = p; + if (!a) { + t = []; + if (p.length) { + for (n = 0; n < p.length; n += 2) { + if (this.db) { + d = this.index.get(p[n]); + if (!d) { + console.warn("Tag '" + p[n] + ":" + p[n + 1] + "' will be skipped because there is no field '" + p[n] + "'."); + continue; + } + t.push(d = d.db.tag(p[n + 1], b, l, q)); + } else { + d = Ma.call(this, p[n], p[n + 1], b, l, q); + } + e.push({field:p[n], tag:p[n + 1], result:d}); + } + } + return t.length ? Promise.all(t).then(function(w) { + for (let v = 0; v < w.length; v++) { + e[v].result = w[v]; + } + return e; + }) : e; + } } - C(k) && (k = [k]); + B(k) && (k = [k]); } } - k || (k = this.h); - n = n && (1 < k.length || m && 1 < m.length); - const r = !d && (this.o || this.async) && []; - for (let l = 0, p, A, B; l < k.length; l++) { - let z; - A = k[l]; - C(A) || (z = A, A = z.field, a = z.query || a, b = z.limit || b, g = z.enrich || g); - if (r) { - r[l] = this.index[A].searchAsync(a, b, z || c); + k || (k = this.field); + p = !d && (this.worker || this.async) && []; + let A; + for (let w = 0, v, J, K; w < k.length; w++) { + J = k[w]; + if (this.db && this.tag && !this.M[w]) { + continue; + } + let D; + B(J) || (D = J, J = D.field, a = D.query || a, b = D.limit || b, t = D.suggest || t); + if (d) { + v = d[w]; } else { - d ? p = d[l] : p = this.index[A].search(a, b, z || c); - B = p && p.length; - if (m && B) { - const y = []; - let H = 0; - n && (y[0] = [p]); - for (let W = 0, pa, R; W < m.length; W++) { - if (pa = m[W], B = (R = this.l[pa]) && R.length) { - H++, y[y.length] = n ? [R] : R; - } - } - H && (p = n ? la(y, b || 100, w || 0) : ma(p, y), B = p.length); - } - if (B) { - f[q] = A, e[q++] = p; - } else if (n) { - return []; + if (r = D || c, x = this.index.get(J), n && (this.db && (r.tag = n, A = x.db.la, r.field = k), A || (r.enrich = !1)), p) { + p[w] = x.searchAsync(a, b, r); + r && q && (r.enrich = q); + continue; + } else { + v = x.search(a, b, r), r && q && (r.enrich = q); } } + K = v && v.length; + if (n && K) { + r = []; + x = 0; + if (this.db && d) { + if (!A) { + for (let G = k.length; G < d.length; G++) { + let H = d[G]; + if (H && H.length) { + x++, r.push(H); + } else if (!t) { + return e; + } + } + } + } else { + for (let G = 0, H, $a; G < n.length; G += 2) { + H = this.tag.get(n[G]); + if (!H) { + if (console.warn("Tag '" + n[G] + ":" + n[G + 1] + "' will be skipped because there is no field '" + n[G] + "'."), t) { + continue; + } else { + return e; + } + } + if ($a = (H = H && H.get(n[G + 1])) && H.length) { + x++, r.push(H); + } else if (!t) { + return e; + } + } + } + if (x) { + v = Ca(v, r); + K = v.length; + if (!K && !t) { + return e; + } + x--; + } + } + if (K) { + f[m] = J, e.push(v), m++; + } else if (1 === k.length) { + return e; + } } - if (r) { - const l = this; - return new Promise(function(p) { - Promise.all(r).then(function(A) { - p(l.search(a, b, c, A)); - }); + if (p) { + if (this.db && n && n.length && !A) { + for (q = 0; q < n.length; q += 2) { + d = this.index.get(n[q]); + if (!d) { + if (console.warn("Tag '" + n[q] + ":" + n[q + 1] + "' was not found because there is no field '" + n[q] + "'."), t) { + continue; + } else { + return e; + } + } + p.push(d.db.tag(n[q + 1], b, l, !1)); + } + } + const w = this; + return Promise.all(p).then(function(v) { + return v.length ? w.search(a, b, c, v) : v; }); } - if (!q) { - return []; + if (!m) { + return e; } - if (h && (!g || !this.store)) { + if (g && (!q || !this.store)) { return e[0]; } - for (let l = 0, p; l < f.length; l++) { - p = e[l]; - p.length && g && (p = ya.call(this, p)); - if (h) { - return p; + p = []; + for (let w = 0, v; w < f.length; w++) { + v = e[w]; + q && v.length && !v[0].doc && (this.db ? p.push(v = this.index.get(this.field[0]).db.enrich(v)) : v.length && (v = Na.call(this, v))); + if (g) { + return v; } - e[l] = {field:f[l], result:p}; + e[w] = {field:f[w], result:v}; } - return e; -}; -function xa(a, b, c, d) { - let e = this.l[a], f = e && e.length - c; - if (f && 0 < f) { - if (f > b || c) { - e = e.slice(c, c + b); + return q && this.db && p.length ? Promise.all(p).then(function(w) { + for (let v = 0; v < w.length; v++) { + e[v].result = w[v]; } - d && (e = ya.call(this, e)); - return {tag:a, result:e}; + return h ? Oa(e, b) : e; + }) : h ? Oa(e, b) : e; +}; +function Oa(a, b) { + const c = [], d = z(); + for (let e = 0, f, g; e < a.length; e++) { + f = a[e]; + g = f.result; + for (let h = 0, k, l, m; h < g.length; h++) { + if (l = g[h], k = l.id, m = d[k]) { + m.push(f.field); + } else { + if (c.length === b) { + return c; + } + l.field = d[k] = [f.field]; + c.push(l); + } + } + } + return c; +} +function Ma(a, b, c, d, e) { + let f = this.tag.get(a); + if (!f) { + return console.warn("Tag '" + a + "' was not found"), []; + } + if ((a = (f = f && f.get(b)) && f.length - d) && 0 < a) { + if (a > c || d) { + f = f.slice(d, d + c); + } + e && (f = Na.call(this, f)); + return f; } } -function ya(a) { +function Na(a) { const b = Array(a.length); for (let c = 0, d; c < a.length; c++) { - d = a[c], b[c] = {id:d, doc:this.store[d]}; + d = a[c], b[c] = {id:d, doc:this.store.get(d)}; } return b; } -t.contain = function(a) { - return !!this.register[a]; +;function Y(a) { + if (!(this instanceof Y)) { + return new Y(a); + } + const b = a.document || a.doc || a; + var c, d; + this.M = []; + this.field = []; + this.W = []; + this.key = (c = b.key || b.id) && Pa(c, this.W) || "id"; + (d = a.keystore || 0) && (this.keystore = d); + this.A = (this.fastupdate = !!a.fastupdate) ? d ? new L(d) : new Map() : d ? new M(d) : new Set(); + this.H = (c = b.store || null) && !0 !== c && []; + this.store = c && (d ? new L(d) : new Map()); + this.cache = (c = a.cache || null) && new F(c); + a.cache = !1; + this.worker = a.worker; + this.async = !1; + c = new Map(); + d = b.index || b.field || b; + B(d) && (d = [d]); + for (let e = 0, f, g; e < d.length; e++) { + f = d[e]; + B(f) || (g = f, f = f.field); + g = C(g) ? Object.assign({}, a, g) : a; + if (this.worker) { + const h = new W(g); + c.set(f, h); + h.worker || (this.worker = !1); + } + this.worker || c.set(f, new P(g, this.A)); + g.S ? this.M[e] = g.S : (this.M[e] = Pa(f, this.W), g.filter && ("string" === typeof this.M[e] && (this.M[e] = new String(this.M[e])), this.M[e].T = g.filter)); + this.field[e] = f; + } + if (this.H) { + d = b.store; + B(d) && (d = [d]); + for (let e = 0, f, g; e < d.length; e++) { + f = d[e], g = f.field || f, f.S ? (this.H[e] = f.S, f.S.ia = g) : (this.H[e] = Pa(g, this.W), f.filter && ("string" === typeof this.H[e] && (this.H[e] = new String(this.H[e])), this.H[e].T = f.filter)); + } + } + this.index = c; + this.tag = null; + if (c = b.tag) { + if ("string" === typeof c && (c = [c]), c.length) { + this.tag = new Map(); + this.L = []; + this.aa = []; + for (let e = 0, f, g; e < c.length; e++) { + f = c[e]; + g = f.field || f; + if (!g) { + throw Error("The tag field from the document descriptor is undefined."); + } + f.S ? this.L[e] = f.S : (this.L[e] = Pa(g, this.W), f.filter && ("string" === typeof this.L[e] && (this.L[e] = new String(this.L[e])), this.L[e].T = f.filter)); + this.aa[e] = g; + this.tag.set(g, new Map()); + } + } + } + a.db && this.mount(a.db); +} +u = Y.prototype; +u.mount = function(a) { + let b = this.field; + if (this.tag) { + for (let e = 0, f; e < this.aa.length; e++) { + f = this.aa[e]; + var c = this.index.get(f); + c || (this.index.set(f, c = new P({}, this.A)), b === this.field && (b = b.slice(0)), b.push(f)); + c.tag = this.tag.get(f); + } + } + c = []; + const d = {db:a.db, type:a.type, fastupdate:a.fastupdate}; + for (let e = 0, f, g; e < b.length; e++) { + d.field = g = b[e]; + f = this.index.get(g); + const h = new a.constructor(a.id, d); + h.id = a.id; + c[e] = h.mount(f); + f.document = !0; + e ? f.ja = !0 : f.store = this.store; + } + this.db = this.async = !0; + return Promise.all(c); }; -t.get = function(a) { - return this.store[a]; +u.commit = async function(a, b) { + const c = []; + for (const d of this.index.values()) { + c.push(d.db.commit(d, a, b)); + } + await Promise.all(c); + this.A.clear(); }; -t.set = function(a, b) { - this.store[a] = b; +function Pa(a, b) { + const c = a.split(":"); + let d = 0; + for (let e = 0; e < c.length; e++) { + a = c[e], "]" === a[a.length - 1] && (a = a.substring(0, a.length - 2)) && (b[d] = !0), a && (c[d++] = a); + } + d < c.length && (c.length = d); + return 1 < d ? c : c[0]; +} +u.append = function(a, b) { + return this.add(a, b, !0); +}; +u.update = function(a, b) { + return this.remove(a).add(a, b); +}; +u.remove = function(a) { + C(a) && (a = ca(a, this.key)); + for (var b of this.index.values()) { + b.remove(a, !0); + } + if (this.A.has(a)) { + if (this.tag && !this.fastupdate) { + for (let c of this.tag.values()) { + for (let d of c) { + b = d[0]; + const e = d[1], f = e.indexOf(a); + -1 < f && (1 < e.length ? e.splice(f, 1) : c.delete(b)); + } + } + } + this.store && this.store.delete(a); + this.A.delete(a); + } + this.cache && this.cache.remove(a); return this; }; -t.searchCache = na; -t.export = function(a, b, c, d, e, f) { - let h; - "undefined" === typeof f && (h = new Promise(g => { - f = g; +u.clear = function() { + for (const a of this.index.values()) { + a.clear(); + } + if (this.tag) { + for (const a of this.tag.values()) { + a.clear(); + } + } + this.store && this.store.clear(); + return this; +}; +u.contain = function(a) { + return this.db ? this.index.get(this.field[0]).db.has(a) : this.A.has(a); +}; +u.cleanup = function() { + for (const a of this.index.values()) { + a.cleanup(); + } + return this; +}; +u.get = function(a) { + return this.db ? this.index.get(this.field[0]).db.enrich(a).then(function(b) { + return b[0] && b[0].doc; + }) : this.store.get(a); +}; +u.set = function(a, b) { + this.store.set(a, b); + return this; +}; +u.searchCache = ma; +u.export = function(a, b, c, d, e, f) { + let g; + "undefined" === typeof f && (g = new Promise(k => { + f = k; })); e || (e = 0); d || (d = 0); - if (d < this.h.length) { - const g = this.h[d], k = this.index[g]; + if (d < this.field.length) { + c = this.field[d]; + var h = this.index[c]; b = this; - setTimeout(function() { - k.export(a, b, e ? g : "", d, e++, f) || (d++, e = 1, b.export(a, b, g, d, e, f)); - }); + h.export(a, b, e ? c : "", d, e++, f) || (d++, b.export(a, b, c, d, 1, f)); } else { - let g, k; switch(e) { case 1: - g = "tag"; - k = this.l; + b = "tag"; + h = this.h; c = null; break; case 2: - g = "store"; - k = this.store; + b = "store"; + h = this.store; c = null; break; default: f(); return; } - qa(a, this, c, g, d, e, k, f); + qa(a, this, c, b, d, e, h, f); } - return h; + return g; }; -t.import = function(a, b) { +u.import = function(a, b) { if (b) { - switch(C(b) && (b = JSON.parse(b)), a) { + switch(B(b) && (b = JSON.parse(b)), a) { case "tag": - this.l = b; + this.h = b; break; case "reg": - this.m = !1; - this.register = b; - for (let d = 0, e; d < this.h.length; d++) { - e = this.index[this.h[d]], e.register = b, e.m = !1; + this.fastupdate = !1; + this.A = b; + for (let d = 0, e; d < this.field.length; d++) { + e = this.index[this.field[d]], e.A = b, e.fastupdate = !1; } break; case "store": @@ -919,60 +2020,329 @@ t.import = function(a, b) { } } }; -ka(S.prototype); -var Aa = {encode:za, F:!1, G:""}; -const Ba = [G("[\u00e0\u00e1\u00e2\u00e3\u00e4\u00e5]"), "a", G("[\u00e8\u00e9\u00ea\u00eb]"), "e", G("[\u00ec\u00ed\u00ee\u00ef]"), "i", G("[\u00f2\u00f3\u00f4\u00f5\u00f6\u0151]"), "o", G("[\u00f9\u00fa\u00fb\u00fc\u0171]"), "u", G("[\u00fd\u0177\u00ff]"), "y", G("\u00f1"), "n", G("[\u00e7c]"), "k", G("\u00df"), "s", G(" & "), " and "]; -function za(a) { - var b = a = "" + a; - b.normalize && (b = b.normalize("NFD").replace(da, "")); - return ba.call(this, b.toLowerCase(), !a.normalize && Ba); +ua(Y.prototype); +const Qa = "undefined" !== typeof window && (window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB), Ra = ["map", "ctx", "tag", "reg", "cfg"]; +function Sa(a, b = {}) { + if (!(this instanceof Sa)) { + return new Sa(a, b); + } + "object" === typeof a && (b = a = a.name); + a || console.info("Default storage space was used, because a name was not passed."); + this.id = "flexsearch" + (a ? ":" + a.toLowerCase().replace(/[^a-z0-9_\-]/g, "") : ""); + this.field = b.field ? b.field.toLowerCase().replace(/[^a-z0-9_\-]/g, "") : ""; + this.la = !1; + this.db = null; + this.h = {}; } -;var Da = {encode:Ca, F:!1, G:"strict"}; -const Ea = /[^a-z0-9]+/, Fa = {b:"p", v:"f", w:"f", z:"s", x:"s", "\u00df":"s", d:"t", n:"m", c:"k", g:"k", j:"k", q:"k", i:"e", y:"e", u:"o"}; -function Ca(a) { - a = za.call(this, a).join(" "); - const b = []; - if (a) { - const c = a.split(Ea), d = c.length; - for (let e = 0, f, h = 0; e < d; e++) { - if ((a = c[e]) && (!this.filter || !this.filter[a])) { - f = a[0]; - let g = Fa[f] || f, k = g; - for (let m = 1; m < a.length; m++) { - f = a[m]; - const n = Fa[f] || f; - n && n !== k && (g += n, k = n); +u = Sa.prototype; +u.mount = function(a) { + if (a instanceof Y) { + return a.mount(this); + } + a.db = this; + return Ta(this); +}; +function Ta(a) { + navigator.storage && navigator.storage.persist(); + return a.db || new Promise(function(b, c) { + const d = Qa.open(a.id + (a.field ? ":" + a.field : ""), 1); + d.onupgradeneeded = function() { + const e = a.db = this.result; + Ra.forEach(f => { + e.objectStoreNames.contains(f) || e.createObjectStore(f); + }); + }; + d.onblocked = function(e) { + console.error("blocked", e); + c(); + }; + d.onerror = function(e) { + console.error(this.error, e); + c(); + }; + d.onsuccess = function() { + a.db = this.result; + a.db.onversionchange = function() { + a.close(); + }; + b(a); + }; + }); +} +u.close = function() { + this.db.close(); + this.db = null; +}; +u.clear = function() { + const a = this.db.transaction(Ra, "readwrite"); + for (let b = 0; b < Ra.length; b++) { + a.objectStore(Ra[b]).clear(); + } + return Z(a); +}; +u.get = function(a, b, c = 0, d = 0, e = !0, f = !1) { + a = this.db.transaction(b ? "ctx" : "map", "readonly").objectStore(b ? "ctx" : "map").get(b ? b + ":" + a : a); + const g = this; + return Z(a).then(function(h) { + let k = []; + if (!h || !h.length) { + return k; + } + if (e) { + if (!c && !d && 1 === h.length) { + return h[0]; + } + for (let l = 0, m; l < h.length; l++) { + if ((m = h[l]) && m.length) { + if (d >= m.length) { + d -= m.length; + continue; + } + const n = c ? d + Math.min(m.length - d, c) : m.length; + for (let q = d; q < n; q++) { + k.push(m[q]); + } + d = 0; + if (k.length === c) { + break; + } } - b[h++] = g; + } + return f ? g.enrich(k) : k; + } + return h; + }); +}; +u.tag = function(a, b = 0, c = 0, d = !1) { + a = this.db.transaction("tag", "readonly").objectStore("tag").get(a); + const e = this; + return Z(a).then(function(f) { + if (!f || !f.length || c >= f.length) { + return []; + } + if (!b && !c) { + return f; + } + f = f.slice(c, c + b); + return d ? e.enrich(f) : f; + }); +}; +u.enrich = function(a) { + "object" !== typeof a && (a = [a]); + const b = this.db.transaction("reg", "readonly").objectStore("reg"), c = []; + for (let d = 0; d < a.length; d++) { + c[d] = Z(b.get(a[d])); + } + return Promise.all(c).then(function(d) { + for (let e = 0; e < d.length; e++) { + d[e] = {id:a[e], doc:d[e] ? JSON.parse(d[e]) : null}; + } + return d; + }); +}; +u.has = function(a) { + a = this.db.transaction("reg", "readonly").objectStore("reg").getKey(a); + return Z(a); +}; +u.search = null; +u.info = function() { +}; +u.transaction = function(a, b, c) { + let d = this.h[a + ":" + b]; + if (d) { + return c.call(this, d); + } + let e = this.db.transaction(a, b); + this.h[a + ":" + b] = d = e.objectStore(a); + return new Promise((f, g) => { + e.onerror = h => { + this.h[a + ":" + b] = null; + e.abort(); + e = d = null; + g(h); + }; + e.oncomplete = h => { + e = d = this.h[a + ":" + b] = null; + f(h || !0); + }; + return c.call(this, d); + }); +}; +u.commit = async function(a, b, c) { + if (b) { + await this.clear(), a.R = []; + } else { + let d = a.R; + a.R = []; + for (let e = 0, f; e < d.length; e++) { + if (f = d[e], f.clear) { + await this.clear(); + b = !0; + break; + } else { + d[e] = f.ma; } } + b || (c || (d = d.concat(ba(a.A))), d.length && await this.remove(d)); } - return b; + a.A.size && (await this.transaction("map", "readwrite", function(d) { + for (const e of a.map) { + const f = e[0], g = e[1]; + g.length && (b ? d.put(g, f) : d.get(f).onsuccess = function() { + let h = this.result; + var k; + if (h && h.length) { + const l = Math.max(h.length, g.length); + for (let m = 0, n, q; m < l; m++) { + if ((q = g[m]) && q.length) { + if ((n = h[m]) && n.length) { + for (k = 0; k < q.length; k++) { + n.push(q[k]); + } + } else { + h[m] = q; + } + k = 1; + } + } + } else { + h = g, k = 1; + } + k && d.put(h, f); + }); + } + }), await this.transaction("ctx", "readwrite", function(d) { + for (const e of a.I) { + const f = e[0], g = e[1]; + for (const h of g) { + const k = h[0], l = h[1]; + l.length && (b ? d.put(l, f + ":" + k) : d.get(f + ":" + k).onsuccess = function() { + let m = this.result; + var n; + if (m && m.length) { + const q = Math.max(m.length, l.length); + for (let t = 0, p, r; t < q; t++) { + if ((r = l[t]) && r.length) { + if ((p = m[t]) && p.length) { + for (n = 0; n < r.length; n++) { + p.push(r[n]); + } + } else { + m[t] = r; + } + n = 1; + } + } + } else { + m = l, n = 1; + } + n && d.put(m, f + ":" + k); + }); + } + } + }), a.store ? await this.transaction("reg", "readwrite", function(d) { + for (const e of a.store) { + const f = e[0], g = e[1]; + d.put("object" === typeof g ? JSON.stringify(g) : 1, f); + } + }) : a.ja || await this.transaction("reg", "readwrite", function(d) { + for (const e of a.A.keys()) { + d.put(1, e); + } + }), a.tag && await this.transaction("tag", "readwrite", function(d) { + for (const e of a.tag) { + const f = e[0], g = e[1]; + g.length && (d.get(f).onsuccess = function() { + let h = this.result; + h = h && h.length ? h.concat(g) : g; + d.put(h, f); + }); + } + }), a.map.clear(), a.I.clear(), a.tag && a.tag.clear(), a.store && a.store.clear(), a.document || a.A.clear()); +}; +function Ua(a, b, c) { + const d = a.value; + let e, f, g = 0; + for (let h = 0, k; h < d.length; h++) { + if (k = c ? d : d[h]) { + for (let l = 0, m, n; l < b.length; l++) { + if (n = b[l], m = k.indexOf(f ? parseInt(n, 10) : n), 0 > m && !f && "string" === typeof n && !isNaN(n) && (m = k.indexOf(parseInt(n, 10))) && (f = 1), 0 <= m) { + if (e = 1, 1 < k.length) { + k.splice(m, 1); + } else { + d[h] = []; + break; + } + } + } + g += k.length; + } + if (c) { + break; + } + } + g ? e && a.update(d) : a.delete(); + a.continue(); } -;var Ha = {encode:Ga, F:!1, G:""}; -const Ia = [G("ae"), "a", G("oe"), "o", G("sh"), "s", G("th"), "t", G("ph"), "f", G("pf"), "f", G("(?![aeo])h(?![aeo])"), "", G("(?!^[aeo])h(?!^[aeo])"), ""]; -function Ga(a, b) { - a && (a = Ca.call(this, a).join(" "), 2 < a.length && (a = F(a, Ia)), b || (1 < a.length && (a = fa(a)), a && (a = a.split(" ")))); - return a || []; +u.remove = function(a) { + "object" !== typeof a && (a = [a]); + return Promise.all([this.transaction("map", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + const c = this.result; + c && Ua(c, a); + }; + }), this.transaction("ctx", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + const c = this.result; + c && Ua(c, a); + }; + }), this.transaction("tag", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + const c = this.result; + c && Ua(c, a, !0); + }; + }), this.transaction("reg", "readwrite", function(b) { + for (let c = 0; c < a.length; c++) { + b.delete(a[c]); + } + })]); +}; +function Z(a) { + return new Promise((b, c) => { + a.onsuccess = function() { + b(this.result); + }; + a.oncomplete = function() { + b(this.result); + }; + a.onerror = c; + a = null; + }); } -;var Ka = {encode:Ja, F:!1, G:""}; -const La = G("(?!\\b)[aeo]"); -function Ja(a) { - a && (a = Ga.call(this, a, !0), 1 < a.length && (a = a.replace(La, "")), 1 < a.length && (a = fa(a)), a && (a = a.split(" "))); - return a || []; -} -;I["latin:default"] = ia; -I["latin:simple"] = Aa; -I["latin:balance"] = Da; -I["latin:advanced"] = Ha; -I["latin:extra"] = Ka; -const Y = {Index:L, Document:S, Worker:P, registerCharset:function(a, b) { - I[a] = b; -}, registerLanguage:function(a, b) { - ja[a] = b; +;const Va = new Map([["b", "p"], ["v", "f"], ["w", "f"], ["z", "s"], ["x", "s"], ["d", "t"], ["n", "m"], ["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], ["i", "e"], ["y", "e"], ["u", "o"]]); +var Wa = {normalize:!0, C:!0, D:Va}; +const Xa = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]), Ya = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"]; +var Za = {normalize:!0, C:!0, D:Va, K:Ya, J:Xa}; +var ab = {normalize:!0, C:!0, D:Va, K:Ya.concat([/(?!^)[aeoy]/g, ""]), J:Xa}; +const bb = {a:"", e:"", i:"", o:"", u:"", y:"", b:1, f:1, p:1, v:1, c:2, g:2, j:2, k:2, q:2, s:2, x:2, z:2, "\u00df":2, d:3, t:3, l:4, m:5, n:5, r:6}; +N["latin:exact"] = {normalize:!1, C:!1}; +N["latin:default"] = sa; +N["latin:simple"] = {normalize:!0, C:!0}; +N["latin:balance"] = Wa; +N["latin:advanced"] = Za; +N["latin:extra"] = ab; +N["latin:soundex"] = {normalize:!0, C:!1, ga:{ka:!0}, V:function(a) { + for (let c = 0; c < a.length; c++) { + var b = a[c]; + let d = b.charAt(0), e = bb[d]; + for (let f = 1, g; f < b.length && (g = b.charAt(f), "h" === g || "w" === g || !(g = bb[g]) || g === e || (d += g, e = g, 4 !== d.length)); f++) { + } + a[c] = d; + } }}; -let Z; -(Z = self.define) && Z.amd ? Z([], function() { - return Y; -}) : self.exports ? self.exports = Y : self.FlexSearch = Y; +const cb = {Index:P, Encoder:E, Charset:N, Language:ra, Document:Y, Worker:W, Resolver:T, IndexedDB:Sa}, db = self; +let eb; +(eb = db.define) && eb.amd ? eb([], function() { + return cb; +}) : "object" === typeof db.exports ? db.exports = cb : db.FlexSearch = cb; }(this)); diff --git a/dist/flexsearch.bundle.min.js b/dist/flexsearch.bundle.min.js index aba4476..5812275 100644 --- a/dist/flexsearch.bundle.min.js +++ b/dist/flexsearch.bundle.min.js @@ -1,34 +1,89 @@ /**! - * FlexSearch.js v0.7.41 (Bundle) + * FlexSearch.js v0.8.0 (Bundle) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -(function _f(self){'use strict';try{if(module)self=module}catch(e){}self._factory=_f;var t;function u(a){return"undefined"!==typeof a?a:!0}function v(a){const b=Array(a);for(let c=0;c=this.B&&(w||!n[l])){var f=M(q,d,r),h="";switch(this.G){case "full":if(2f;g--)if(g-f>=this.B){var k=M(q,d,r,e,f);h=l.substring(f,g);N(this,n,h,k,a,c)}break}case "reverse":if(1=this.B&&N(this,n, -h,M(q,d,r,e,g),a,c);h=""}case "forward":if(1=this.B&&N(this,n,h,f,a,c);break}default:if(this.C&&(f=Math.min(f/this.C(b,l,r)|0,q-1)),N(this,n,l,f,a,c),w&&1=this.B&&!e[l]){e[l]=1;const p=this.l&&l>f;N(this,m,p?f:l,M(h+(d/2>h?0:1),d,r,g-1,k-1),a,c,p?l:f)}}}}this.m||(this.register[a]=1)}}return this}; -function M(a,b,c,d,e){return c&&1=this.B&&!c[q])if(this.s||f||this.map[q])k[w++]=q,c[q]=1;else return d;a=k;e=a.length}if(!e)return d;b||(b=100);g=this.depth&&1=d)))break;if(n){if(f)return sa(k,d,0);b[b.length]=k;return}}return!c&&k}function sa(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function ta(a,b,c,d){c?(d=d&&b>c,a=(a=a[d?b:c])&&a[d?c:b]):a=a[b];return a}t.contain=function(a){return!!this.register[a]};t.update=function(a,b){return this.remove(a).add(a,b)}; -t.remove=function(a,b){const c=this.register[a];if(c){if(this.m)for(let d=0,e;d{f=m}));let g,k;switch(e||(e=0)){case 0:g="reg";if(this.m){k=x();for(let m in this.register)k[m]=1}else k=this.register;break;case 1:g="cfg";k={doc:0,opt:this.s?1:0};break;case 2:g="map";k=this.map;break;case 3:g="ctx";k=this.h;break;default:"undefined"===typeof c&&f&&f();return}qa(a,b||this,c,g,d,e,k,f);return h}; -t.import=function(a,b){if(b)switch(C(b)&&(b=JSON.parse(b)),a){case "cfg":this.s=!!b.opt;break;case "reg":this.m=!1;this.register=b;break;case "map":this.map=b;break;case "ctx":this.h=b}};ka(L.prototype);function ua(a){a=a.data;var b=self._index;const c=a.args;var d=a.task;switch(d){case "init":d=a.options||{};a=a.factory;b=d.encode;d.cache=!1;b&&0===b.indexOf("function")&&(d.encode=Function("return "+b)());a?(Function("return "+a)()(self),self._index=new self.FlexSearch.Index(d),delete self.FlexSearch):self._index=new L(d);break;default:a=a.id,b=b[d].apply(b,c),postMessage("search"===d?{id:a,msg:b}:{id:a})}};let va=0;function P(a){if(!(this instanceof P))return new P(a);var b;a?E(b=a.encode)&&(a.encode=b.toString()):a={};(b=(self||window)._factory)&&(b=b.toString());const c="undefined"===typeof window&&self.exports,d=this;this.o=wa(b,c,a.worker);this.h=x();if(this.o){if(c)this.o.on("message",function(e){d.h[e.id](e.msg);delete d.h[e.id]});else this.o.onmessage=function(e){e=e.data;d.h[e.id](e.msg);delete d.h[e.id]};this.o.postMessage({task:"init",factory:b,options:a})}}Q("add");Q("append");Q("search"); -Q("update");Q("remove");function Q(a){P.prototype[a]=P.prototype[a+"Async"]=function(){const b=this,c=[].slice.call(arguments);var d=c[c.length-1];let e;E(d)&&(e=d,c.splice(c.length-1,1));d=new Promise(function(f){setTimeout(function(){b.h[++va]=f;b.o.postMessage({task:a,id:va,args:c})})});return e?(d.then(e),this):d}} -function wa(a,b,c){let d;try{d=b?new (require("worker_threads")["Worker"])(__dirname + "/node/node.js"):a?new Worker(URL.createObjectURL(new Blob(["onmessage="+ua.toString()],{type:"text/javascript"}))):new Worker(C(c)?c:"worker/worker.js",{type:"module"})}catch(e){}return d};function S(a){if(!(this instanceof S))return new S(a);var b=a.document||a.doc||a,c;this.K=[];this.h=[];this.A=[];this.register=x();this.key=(c=b.key||b.id)&&T(c,this.A)||"id";this.m=u(a.fastupdate);this.C=(c=b.store)&&!0!==c&&[];this.store=c&&x();this.I=(c=b.tag)&&T(c,this.A);this.l=c&&x();this.cache=(c=a.cache)&&new K(c);a.cache=!1;this.o=a.worker;this.async=!1;c=x();let d=b.index||b.field||b;C(d)&&(d=[d]);for(let e=0,f,h;eb||c)e=e.slice(c,c+b);d&&(e=ya.call(this,e));return{tag:a,result:e}}}function ya(a){const b=Array(a.length);for(let c=0,d;c{f=g}));e||(e=0);d||(d=0);if(dthis.N.get(l)),k=1);this.J&&1this.J.get(l)),k=1);g&&k&&(g.lengththis.ea&&(this.U.clear(), +this.B=this.B/1.1|0));g&&c.push(g)}this.V&&(c=this.V(c)||c);this.cache&&a.length<=this.h&&(this.P.set(a,c),this.P.size>this.ea&&(this.P.clear(),this.h=this.h/1.1|0));return c};function la(a){a.X=null;a.P.clear();a.U.clear()};function ma(a,b,c){a=("object"===typeof a?""+a.query:a).toLowerCase();let d=this.cache.get(a);if(!d){d=this.search(a,b,c);if(d instanceof Promise){const e=this;d.then(function(f){e.cache.set(a,f)})}this.cache.set(a,d)}return d}function F(a){this.limit=a&&!0!==a?a:1E3;this.cache=new Map;this.h=""}F.prototype.set=function(a,b){this.cache.has(a)||(this.cache.set(this.h=a,b),this.limit&&this.cache.size>this.limit&&this.cache.delete(this.cache.keys().next().value))}; +F.prototype.get=function(a){const b=this.cache.get(a);b&&this.limit&&this.h!==a&&(this.cache.delete(a),this.cache.set(this.h=a,b));return b};F.prototype.remove=function(a){for(const b of this.cache){const c=b[0];b[1].includes(a)&&this.cache.delete(c)}};F.prototype.clear=function(){this.cache.clear();this.h=""};function na(a,b,c,d){let e=[];for(let f=0,g;f=g.length)b-=g.length;else{b=g[d?"splice":"slice"](b,c);const h=b.length;if(h&&(e=e.length?e.concat(b):b,c-=h,d&&(a.length-=h),!c))break;b=0}return e} +function H(a){if(!(this instanceof H))return new H(a);this.index=a?[a]:[];this.length=a?a.length:0;const b=this;return new Proxy([],{get(c,d){if("length"===d)return b.length;if("push"===d)return function(e){b.index[b.index.length-1].push(e);b.length++};if("pop"===d)return function(){if(b.length)return b.length--,b.index[b.index.length-1].pop()};if("indexOf"===d)return function(e){let f=0;for(let g=0,h,k;gf;h--){g=p.substring(f,h);var k=this.score?this.score(b,p,t,g,f):Q(q,d,t,e,f);R(this,m,g,k,a,c)}break}case "reverse":if(1< +e){for(h=e-1;0g?0:1),d,t,h-1,k-1),x=this.bidirectional&&p>f;R(this,l,x?f:p,r,a,c,x?p:f)}}}}this.fastupdate||this.A.add(a)}else b=""}this.db&& +(b||this.R.push({del:a}),this.fa&&va(this));return this};function R(a,b,c,d,e,f,g){let h=g?a.I:a.map,k;if(!b[c]||!g||!(k=b[c])[g])if(g?(b=k||(b[c]=z()),b[g]=1,(k=h.get(g))?h=k:h.set(g,h=new Map)):b[c]=1,(k=h.get(c))?h=k:h.set(c,h=k=[]),h=h[d]||(h[d]=[]),!f||!h.includes(e)){if(h.length===2**31-1){b=new H(h);if(a.fastupdate)for(let l of a.A.values())l.includes(h)&&(l[l.indexOf(h)]=b);k[d]=h=b}h.push(e);a.fastupdate&&((d=a.A.get(e))?d.push(h):a.A.set(e,[h]))}} +function Q(a,b,c,d,e){return c&&1b?b?a.slice(c,c+b):a.slice(c):a,d?wa(a):a;let e=[];for(let f=0,g,h;f=h){c-=h;continue}cb&&(g=g.slice(0,b),h=g.length),e.push(g);else{if(h>=b)return h>b&&(g=g.slice(0,b)),d?wa(g):g;e=[g]}b-=h;if(!b)break}if(!e.length)return e;e=1a.length)return e?S(a[0],b,c,d):a[0];d=[];let g=0,h=z(),k=da(a);for(let l=0,m;la.length)return[];let f=[],g=0,h=z(),k=da(a);if(!k)return f;for(let l=0,m;la.length)return e?S(a[0],b,c,d):a[0];b=[];c=z();for(let g=0,h;g=e)))break;if(h.length){if(g)return S(h,e,0);b.push(h);return}}return!c&&h}function V(a,b,c,d,e,f,g,h){let k;c&&(k=a.bidirectional&&b>c);if(a.db)return c?a.db.get(k?c:b,k?b:c,d,e,f,g,h):a.db.get(b,"",d,e,f,g,h);a=c?(a=a.I.get(k?b:c))&&a.get(k?c:b):a.map.get(b);return a};P.prototype.remove=function(a,b){const c=this.A.size&&(this.fastupdate?this.A.get(a):this.A.has(a));if(c){if(this.fastupdate)for(let d=0,e;de.length)e.pop();else{const f=e.indexOf(a);f===c.length-1?e.pop():e.splice(f,1)}}else Fa(this.map,a),this.depth&&Fa(this.I,a);b||this.A.delete(a)}this.db&&(this.R.push({del:a}),this.fa&&va(this));this.cache&&this.cache.remove(a);return this}; +function Fa(a,b){let c=0;if(a.constructor===Array)for(let d=0,e,f;dc.add(a,b)):this.add(a,b)}return this.remove(a).add(a,b)};function Ga(a){let b=0;if(a.constructor===Array)for(let c=0,d;c{f=l}));let h,k;switch(e||(e=0)){case 0:h="reg";if(this.fastupdate){k=z();for(let l of this.A.keys())k[l]=1}else k=this.A;break;case 1:h="cfg";k={doc:0,opt:this.B?1:0};break;case 2:h="map";k=this.map;break;case 3:h="ctx";k=this.I;break;default:"undefined"===typeof c&&f&&f();return}qa(a,b||this,c,h,d,e,k,f);return g}; +u.import=function(a,b){if(b)switch(B(b)&&(b=JSON.parse(b)),a){case "cfg":this.B=!!b.opt;break;case "reg":this.fastupdate=!1;this.A=b;break;case "map":this.map=b;break;case "ctx":this.I=b}};ua(P.prototype);async function Ha(a){a=a.data;var b=self._index;const c=a.args;var d=a.task;switch(d){case "init":d=a.options||{};(b=d.config)&&(d=await import(b));(b=a.factory)?(Function("return "+b)()(self),self._index=new self.FlexSearch.Index(d),delete self.FlexSearch):self._index=new P(d);postMessage({id:a.id});break;default:a=a.id,b=b[d].apply(b,c),postMessage("search"===d?{id:a,msg:b}:{id:a})}};let Ia=0; +function W(a){function b(f){f=f.data||f;const g=f.id,h=g&&e.h[g];h&&(h(f.msg),delete e.h[g])}if(!(this instanceof W))return new W(a);a||(a={});let c=(self||window)._factory;c&&(c=c.toString());const d="undefined"===typeof window&&self.exports,e=this;this.worker=Ja(c,d,a.worker);this.h=z();if(this.worker){d?this.worker.on("message",b):this.worker.onmessage=b;if(a.config)return new Promise(function(f){e.h[++Ia]=function(){f(e)};e.worker.postMessage({id:Ia,task:"init",factory:c,options:a})});this.worker.postMessage({task:"init", +factory:c,options:a})}}X("add");X("append");X("search");X("update");X("remove");function X(a){W.prototype[a]=W.prototype[a+"Async"]=function(){const b=this,c=[].slice.call(arguments);var d=c[c.length-1];let e;"function"===typeof d&&(e=d,c.splice(c.length-1,1));d=new Promise(function(f){b.h[++Ia]=f;b.worker.postMessage({task:a,id:Ia,args:c})});return e?(d.then(e),this):d}} +function Ja(a,b,c){return b?new (require("worker_threads")["Worker"])(__dirname + "/node/node.js"):a?new window.Worker(URL.createObjectURL(new Blob(["onmessage="+Ha.toString()],{type:"text/javascript"}))):new window.Worker(B(c)?c:"worker/worker.js",{type:"module"})};Y.prototype.add=function(a,b,c){C(a)&&(b=a,a=ca(b,this.key));if(b&&(a||0===a)){if(!c&&this.A.has(a))return this.update(a,b);for(let h=0,k;hc||d)a=a.slice(d,d+c);e&&(a=Na.call(this,a));return a}} +function Na(a){const b=Array(a.length);for(let c=0,d;c{f=k}));e||(e=0);d||(d=0);if(d{e.objectStoreNames.contains(f)||e.createObjectStore(f)})};d.onblocked=function(e){console.error("blocked",e);c()};d.onerror=function(e){console.error(this.error,e);c()};d.onsuccess=function(){a.db=this.result;a.db.onversionchange=function(){a.close()};b(a)}})} +u.close=function(){this.db.close();this.db=null};u.clear=function(){const a=this.db.transaction(Ra,"readwrite");for(let b=0;b=m.length){d-=m.length;continue}const n=c?d+Math.min(m.length-d,c):m.length;for(let q=d;q=f.length)return[];if(!b&&!c)return f;f=f.slice(c,c+b);return d?e.enrich(f):f})}; +u.enrich=function(a){"object"!==typeof a&&(a=[a]);const b=this.db.transaction("reg","readonly").objectStore("reg"),c=[];for(let d=0;d{e.onerror=h=>{this.h[a+":"+b]=null;e.abort();e=d=null;g(h)};e.oncomplete=h=>{e=d=this.h[a+":"+b]=null;f(h||!0)};return c.call(this,d)})}; +u.commit=async function(a,b,c){if(b)await this.clear(),a.R=[];else{let d=a.R;a.R=[];for(let e=0,f;em&&!f&&"string"===typeof n&&!isNaN(n)&&(m=k.indexOf(parseInt(n,10)))&&(f=1),0<=m)if(e=1,1{a.onsuccess=function(){b(this.result)};a.oncomplete=function(){b(this.result)};a.onerror=c;a=null})};const Va=new Map([["b","p"],["v","f"],["w","f"],["z","s"],["x","s"],["d","t"],["n","m"],["c","k"],["g","k"],["j","k"],["q","k"],["i","e"],["y","e"],["u","o"]]);var Wa={normalize:!0,C:!0,D:Va};const Xa=new Map([["ai","ei"],["ae","a"],["oe","o"],["ue","u"],["sh","s"],["ch","c"],["th","t"],["ph","f"],["pf","f"]]),Ya=[/([^aeo])h([aeo$])/g,"$1$2",/([aeo])h([^aeo]|$)/g,"$1$2"];var Za={normalize:!0,C:!0,D:Va,K:Ya,J:Xa};var ab={normalize:!0,C:!0,D:Va,K:Ya.concat([/(?!^)[aeoy]/g,""]),J:Xa};const bb={a:"",e:"",i:"",o:"",u:"",y:"",b:1,f:1,p:1,v:1,c:2,g:2,j:2,k:2,q:2,s:2,x:2,z:2,"\u00df":2,d:3,t:3,l:4,m:5,n:5,r:6};M["latin:exact"]={normalize:!1,C:!1};M["latin:default"]=sa;M["latin:simple"]={normalize:!0,C:!0};M["latin:balance"]=Wa;M["latin:advanced"]=Za;M["latin:extra"]=ab;M["latin:soundex"]={normalize:!0,C:!1,ga:{ka:!0},V:function(a){for(let c=0;c= this.B && (w || !n[l])) { - var f = O(q, d, r), h = ""; - switch(this.G) { - case "full": - if (2 < e) { - for (f = 0; f < e; f++) { - for (var g = e; g > f; g--) { - if (g - f >= this.B) { - var k = O(q, d, r, e, f); - h = l.substring(f, g); - P(this, n, h, k, a, c); - } - } - } - break; - } - case "reverse": - if (1 < e) { - for (g = e - 1; 0 < g; g--) { - h = l[g] + h, h.length >= this.B && P(this, n, h, O(q, d, r, e, g), a, c); - } - h = ""; - } - case "forward": - if (1 < e) { - for (g = 0; g < e; g++) { - h += l[g], h.length >= this.B && P(this, n, h, f, a, c); - } - break; - } - default: - if (this.C && (f = Math.min(f / this.C(b, l, r) | 0, q - 1)), P(this, n, l, f, a, c), w && 1 < d && r < d - 1) { - for (e = x(), h = this.A, f = l, g = Math.min(w + 1, d - r), e[f] = 1, k = 1; k < g; k++) { - if ((l = b[this.F ? d - 1 - r - k : r + k]) && l.length >= this.B && !e[l]) { - e[l] = 1; - const p = this.l && l > f; - P(this, m, p ? f : l, O(h + (d / 2 > h ? 0 : 1), d, r, g - 1, k - 1), a, c, p ? l : f); - } - } - } - } - } - } - this.m || (this.register[a] = 1); - } - } - return this; -}; -function O(a, b, c, d, e) { - return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; -} -function P(a, b, c, d, e, f, h) { - let g = h ? a.h : a.map; - if (!b[c] || h && !b[c][h]) { - a.s && (g = g[d]), h ? (b = b[c] || (b[c] = x()), b[h] = 1, g = g[h] || (g[h] = x())) : b[c] = 1, g = g[c] || (g[c] = []), a.s || (g = g[d] || (g[d] = [])), f && g.includes(e) || (g[g.length] = e, a.m && (a = a.register[e] || (a.register[e] = []), a[a.length] = g)); - } -} -t.search = function(a, b, c) { - c || (!b && D(a) ? (c = a, a = c.query) : D(b) && (c = b)); - let d = [], e; - let f, h = 0; - if (c) { - a = c.query || a; - b = c.limit; - h = c.offset || 0; - var g = c.context; - f = c.suggest; - } - if (a && (a = this.encode("" + a), e = a.length, 1 < e)) { - c = x(); - var k = []; - for (let n = 0, w = 0, q; n < e; n++) { - if ((q = a[n]) && q.length >= this.B && !c[q]) { - if (this.s || f || this.map[q]) { - k[w++] = q, c[q] = 1; - } else { - return d; - } - } - } - a = k; - e = a.length; - } - if (!e) { - return d; - } - b || (b = 100); - g = this.depth && 1 < e && !1 !== g; - c = 0; - let m; - g ? (m = a[0], c = 1) : 1 < e && a.sort(aa); - for (let n, w; c < e; c++) { - w = a[c]; - g ? (n = pa(this, d, f, b, h, 2 === e, w, m), f && !1 === n && d.length || (m = w)) : n = pa(this, d, f, b, h, 1 === e, w); - if (n) { - return n; - } - if (f && c === e - 1) { - k = d.length; - if (!k) { - if (g) { - g = 0; - c = -1; - continue; - } - return d; - } - if (1 === k) { - return qa(d[0], b, h); - } - } - } - return ja(d, b, h, f); -}; -function pa(a, b, c, d, e, f, h, g) { - let k = [], m = g ? a.h : a.map; - a.s || (m = ra(m, h, g, a.l)); - if (m) { - let n = 0; - const w = Math.min(m.length, g ? a.A : a.D); - for (let q = 0, r = 0, l, p; q < w; q++) { - if (l = m[q]) { - if (a.s && (l = ra(l, h, g, a.l)), e && l && f && (p = l.length, p <= e ? (e -= p, l = null) : (l = l.slice(e), e = 0)), l && (k[n++] = l, f && (r += l.length, r >= d))) { - break; - } - } - } - if (n) { - if (f) { - return qa(k, d, 0); - } - b[b.length] = k; - return; - } - } - return !c && k; -} -function qa(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function ra(a, b, c, d) { - c ? (d = d && b > c, a = (a = a[d ? b : c]) && a[d ? c : b]) : a = a[b]; - return a; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a, b) { - const c = this.register[a]; - if (c) { - if (this.m) { - for (let d = 0, e; d < c.length; d++) { - e = c[d], e.splice(e.indexOf(a), 1); - } - } else { - Q(this.map, a, this.D, this.s), this.depth && Q(this.h, a, this.A, this.s); - } - b || delete this.register[a]; - if (this.cache) { - b = this.cache; - for (let d = 0, e, f; d < b.h.length; d++) { - f = b.h[d], e = b.cache[f], e.includes(a) && (b.h.splice(d--, 1), delete b.cache[f]); - } - } - } - return this; -}; -function Q(a, b, c, d, e) { - let f = 0; - if (a.constructor === Array) { - if (e) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), f++) : f++; - } else { - e = Math.min(a.length, c); - for (let h = 0, g; h < e; h++) { - if (g = a[h]) { - f = Q(g, b, c, d, e), d || f || delete a[h]; - } - } - } - } else { - for (let h in a) { - (f = Q(a[h], b, c, d, e)) || delete a[h]; - } - } - return f; -} -t.searchCache = la; -t.export = function(a, b, c, d, e, f) { - let h = !0; - "undefined" === typeof f && (h = new Promise(m => { - f = m; - })); - let g, k; - switch(e || (e = 0)) { - case 0: - g = "reg"; - if (this.m) { - k = x(); - for (let m in this.register) { - k[m] = 1; - } - } else { - k = this.register; - } - break; - case 1: - g = "cfg"; - k = {doc:0, opt:this.s ? 1 : 0}; - break; - case 2: - g = "map"; - k = this.map; - break; - case 3: - g = "ctx"; - k = this.h; - break; - default: - "undefined" === typeof c && f && f(); - return; - } - oa(a, b || this, c, g, d, e, k, f); - return h; -}; -t.import = function(a, b) { - if (b) { - switch(C(b) && (b = JSON.parse(b)), a) { - case "cfg": - this.s = !!b.opt; - break; - case "reg": - this.m = !1; - this.register = b; - break; - case "map": - this.map = b; - break; - case "ctx": - this.h = b; - } - } -}; -ia(N.prototype); -function sa(a) { - a = a.data; - var b = self._index; - const c = a.args; - var d = a.task; - switch(d) { - case "init": - d = a.options || {}; - a = a.factory; - b = d.encode; - d.cache = !1; - b && 0 === b.indexOf("function") && (d.encode = Function("return " + b)()); - a ? (Function("return " + a)()(self), self._index = new self.FlexSearch.Index(d), delete self.FlexSearch) : self._index = new N(d); - break; - default: - a = a.id, b = b[d].apply(b, c), postMessage("search" === d ? {id:a, msg:b} : {id:a}); - } -} -;let ta = 0; -function S(a) { - if (!(this instanceof S)) { - return new S(a); - } - var b; - a ? E(b = a.encode) && (a.encode = b.toString()) : a = {}; - (b = (self || window)._factory) && (b = b.toString()); - const c = "undefined" === typeof window && self.exports, d = this; - this.o = ua(b, c, a.worker); - this.h = x(); - if (this.o) { - if (c) { - this.o.on("message", function(e) { - d.h[e.id](e.msg); - delete d.h[e.id]; - }); - } else { - this.o.onmessage = function(e) { - e = e.data; - d.h[e.id](e.msg); - delete d.h[e.id]; - }; - } - this.o.postMessage({task:"init", factory:b, options:a}); - } -} -T("add"); -T("append"); -T("search"); -T("update"); -T("remove"); -function T(a) { - S.prototype[a] = S.prototype[a + "Async"] = function() { - const b = this, c = [].slice.call(arguments); - var d = c[c.length - 1]; - let e; - E(d) && (e = d, c.splice(c.length - 1, 1)); - d = new Promise(function(f) { - setTimeout(function() { - b.h[++ta] = f; - b.o.postMessage({task:a, id:ta, args:c}); - }); - }); - return e ? (d.then(e), this) : d; - }; -} -function ua(a, b, c) { - let d; - try { - d = b ? new (require("worker_threads")["Worker"])(__dirname + "/node/node.js") : a ? new Worker(URL.createObjectURL(new Blob(["onmessage=" + sa.toString()], {type:"text/javascript"}))) : new Worker(C(c) ? c : "worker/worker.js", {type:"module"}); - } catch (e) { - } - return d; -} -;function U(a) { - if (!(this instanceof U)) { - return new U(a); - } - var b = a.document || a.doc || a, c; - this.K = []; - this.h = []; - this.A = []; - this.register = x(); - this.key = (c = b.key || b.id) && V(c, this.A) || "id"; - this.m = u(a.fastupdate); - this.C = (c = b.store) && !0 !== c && []; - this.store = c && x(); - this.I = (c = b.tag) && V(c, this.A); - this.l = c && x(); - this.cache = (c = a.cache) && new M(c); - a.cache = !1; - this.o = a.worker; - this.async = !1; - c = x(); - let d = b.index || b.field || b; - C(d) && (d = [d]); - for (let e = 0, f, h; e < d.length; e++) { - f = d[e], C(f) || (h = f, f = f.field), h = D(h) ? Object.assign({}, a, h) : a, this.o && (c[f] = new S(h), c[f].o || (this.o = !1)), this.o || (c[f] = new N(h, this.register)), this.K[e] = V(f, this.A), this.h[e] = f; - } - if (this.C) { - for (a = b.store, C(a) && (a = [a]), b = 0; b < a.length; b++) { - this.C[b] = V(a[b], this.A); - } - } - this.index = c; -} -function V(a, b) { - const c = a.split(":"); - let d = 0; - for (let e = 0; e < c.length; e++) { - a = c[e], 0 <= a.indexOf("[]") && (a = a.substring(0, a.length - 2)) && (b[d] = !0), a && (c[d++] = a); - } - d < c.length && (c.length = d); - return 1 < d ? c : c[0]; -} -function X(a, b) { - if (C(b)) { +function ca(a, b) { + if (B(b)) { a = a[b]; } else { for (let c = 0; a && c < b.length; c++) { @@ -657,22 +72,1481 @@ function X(a, b) { } return a; } -function Y(a, b, c, d, e) { - a = a[e]; - if (d === c.length - 1) { - b[e] = a; - } else if (a) { - if (a.constructor === Array) { - for (b = b[e] = Array(a.length), e = 0; e < a.length; e++) { - Y(a, b, c, d, e); +function da(a) { + let b = 0; + for (let c = 0, d; c < a.length; c++) { + (d = a[c]) && b < d.length && (b = d.length); + } + return b; +} +;const ea = /[^\p{L}\p{N}]+/u, fa = /(\d{3})/g, ha = /(\D)(\d{3})/g, ia = /(\d{3})(\D)/g, ja = "".normalize && /[\u0300-\u036f]/g, ka = !ja && new Map([["\u00aa", "a"], ["\u00b2", "2"], ["\u00b3", "3"], ["\u00b9", "1"], ["\u00ba", "o"], ["\u00bc", "1\u20444"], ["\u00bd", "1\u20442"], ["\u00be", "3\u20444"], ["\u00e0", "a"], ["\u00e1", "a"], ["\u00e2", "a"], ["\u00e3", "a"], ["\u00e4", "a"], ["\u00e5", "a"], ["\u00e7", "c"], ["\u00e8", "e"], ["\u00e9", "e"], ["\u00ea", "e"], ["\u00eb", "e"], ["\u00ec", +"i"], ["\u00ed", "i"], ["\u00ee", "i"], ["\u00ef", "i"], ["\u00f1", "n"], ["\u00f2", "o"], ["\u00f3", "o"], ["\u00f4", "o"], ["\u00f5", "o"], ["\u00f6", "o"], ["\u00f9", "u"], ["\u00fa", "u"], ["\u00fb", "u"], ["\u00fc", "u"], ["\u00fd", "y"], ["\u00ff", "y"], ["\u0101", "a"], ["\u0103", "a"], ["\u0105", "a"], ["\u0107", "c"], ["\u0109", "c"], ["\u010b", "c"], ["\u010d", "c"], ["\u010f", "d"], ["\u0113", "e"], ["\u0115", "e"], ["\u0117", "e"], ["\u0119", "e"], ["\u011b", "e"], ["\u011d", "g"], ["\u011f", +"g"], ["\u0121", "g"], ["\u0123", "g"], ["\u0125", "h"], ["\u0129", "i"], ["\u012b", "i"], ["\u012d", "i"], ["\u012f", "i"], ["\u0133", "ij"], ["\u0135", "j"], ["\u0137", "k"], ["\u013a", "l"], ["\u013c", "l"], ["\u013e", "l"], ["\u0140", "l"], ["\u0144", "n"], ["\u0146", "n"], ["\u0148", "n"], ["\u0149", "n"], ["\u014d", "o"], ["\u014f", "o"], ["\u0151", "o"], ["\u0155", "r"], ["\u0157", "r"], ["\u0159", "r"], ["\u015b", "s"], ["\u015d", "s"], ["\u015f", "s"], ["\u0161", "s"], ["\u0163", "t"], ["\u0165", +"t"], ["\u0169", "u"], ["\u016b", "u"], ["\u016d", "u"], ["\u016f", "u"], ["\u0171", "u"], ["\u0173", "u"], ["\u0175", "w"], ["\u0177", "y"], ["\u017a", "z"], ["\u017c", "z"], ["\u017e", "z"], ["\u017f", "s"], ["\u01a1", "o"], ["\u01b0", "u"], ["\u01c6", "dz"], ["\u01c9", "lj"], ["\u01cc", "nj"], ["\u01ce", "a"], ["\u01d0", "i"], ["\u01d2", "o"], ["\u01d4", "u"], ["\u01d6", "u"], ["\u01d8", "u"], ["\u01da", "u"], ["\u01dc", "u"], ["\u01df", "a"], ["\u01e1", "a"], ["\u01e3", "ae"], ["\u00e6", "ae"], +["\u01fd", "ae"], ["\u01e7", "g"], ["\u01e9", "k"], ["\u01eb", "o"], ["\u01ed", "o"], ["\u01ef", "\u0292"], ["\u01f0", "j"], ["\u01f3", "dz"], ["\u01f5", "g"], ["\u01f9", "n"], ["\u01fb", "a"], ["\u01ff", "\u00f8"], ["\u0201", "a"], ["\u0203", "a"], ["\u0205", "e"], ["\u0207", "e"], ["\u0209", "i"], ["\u020b", "i"], ["\u020d", "o"], ["\u020f", "o"], ["\u0211", "r"], ["\u0213", "r"], ["\u0215", "u"], ["\u0217", "u"], ["\u0219", "s"], ["\u021b", "t"], ["\u021f", "h"], ["\u0227", "a"], ["\u0229", "e"], +["\u022b", "o"], ["\u022d", "o"], ["\u022f", "o"], ["\u0231", "o"], ["\u0233", "y"], ["\u02b0", "h"], ["\u02b1", "h"], ["\u0266", "h"], ["\u02b2", "j"], ["\u02b3", "r"], ["\u02b4", "\u0279"], ["\u02b5", "\u027b"], ["\u02b6", "\u0281"], ["\u02b7", "w"], ["\u02b8", "y"], ["\u02e0", "\u0263"], ["\u02e1", "l"], ["\u02e2", "s"], ["\u02e3", "x"], ["\u02e4", "\u0295"], ["\u0390", "\u03b9"], ["\u03ac", "\u03b1"], ["\u03ad", "\u03b5"], ["\u03ae", "\u03b7"], ["\u03af", "\u03b9"], ["\u03b0", "\u03c5"], ["\u03ca", +"\u03b9"], ["\u03cb", "\u03c5"], ["\u03cc", "\u03bf"], ["\u03cd", "\u03c5"], ["\u03ce", "\u03c9"], ["\u03d0", "\u03b2"], ["\u03d1", "\u03b8"], ["\u03d2", "\u03a5"], ["\u03d3", "\u03a5"], ["\u03d4", "\u03a5"], ["\u03d5", "\u03c6"], ["\u03d6", "\u03c0"], ["\u03f0", "\u03ba"], ["\u03f1", "\u03c1"], ["\u03f2", "\u03c2"], ["\u03f5", "\u03b5"], ["\u0439", "\u0438"], ["\u0450", "\u0435"], ["\u0451", "\u0435"], ["\u0453", "\u0433"], ["\u0457", "\u0456"], ["\u045c", "\u043a"], ["\u045d", "\u0438"], ["\u045e", +"\u0443"], ["\u0477", "\u0475"], ["\u04c2", "\u0436"], ["\u04d1", "\u0430"], ["\u04d3", "\u0430"], ["\u04d7", "\u0435"], ["\u04db", "\u04d9"], ["\u04dd", "\u0436"], ["\u04df", "\u0437"], ["\u04e3", "\u0438"], ["\u04e5", "\u0438"], ["\u04e7", "\u043e"], ["\u04eb", "\u04e9"], ["\u04ed", "\u044d"], ["\u04ef", "\u0443"], ["\u04f1", "\u0443"], ["\u04f3", "\u0443"], ["\u04f5", "\u0447"]]); +function E(a = {}) { + if (!(this instanceof E)) { + return new E(...arguments); + } + for (a = 0; a < arguments.length; a++) { + this.assign(arguments[a]); + } +} +E.prototype.assign = function(a) { + this.normalize = y(a.normalize, !0, this.normalize); + let b = a.ga, c = b || a.na || a.split; + if ("object" === typeof c) { + let d = !b, e = ""; + a.ga || (e += "\\p{Z}"); + c.ka && (e += "\\p{L}"); + c.oa && (e += "\\p{N}", d = !!b); + c.qa && (e += "\\p{S}"); + c.pa && (e += "\\p{P}"); + c.control && (e += "\\p{C}"); + if (c = c.char) { + e += "object" === typeof c ? c.join("") : c; + } + this.split = new RegExp("[" + (b ? "^" : "") + e + "]+", "u"); + this.numeric = d; + } else { + this.split = y(c, ea, this.split), this.numeric = y(this.numeric, !0); + } + this.$ = y(a.$, null, this.$); + this.V = y(a.V, null, this.V); + this.rtl = a.rtl || !1; + this.C = y(a.C, !0, this.C); + this.filter = y((c = a.filter) && new Set(c), null, this.filter); + this.J = y((c = a.J) && new Map(c), null, this.J); + this.D = y((c = a.D) && new Map(c), null, this.D); + this.N = y((c = a.N) && new Map(c), null, this.N); + this.K = y(a.K, null, this.K); + this.Z = y(a.Z, 1, this.Z); + this.ha = y(a.ha, 0, this.ha); + if (this.cache = c = y(a.cache, !0, this.cache)) { + this.X = null, this.ea = "number" === typeof c ? c : 2e5, this.P = new Map(), this.U = new Map(), this.B = this.h = 128; + } + this.F = ""; + this.ba = null; + this.Y = ""; + this.ca = null; + if (this.J) { + for (const d of this.J.keys()) { + this.F += (this.F ? "|" : "") + d; + } + } + if (this.N) { + for (const d of this.N.keys()) { + this.Y += (this.Y ? "|" : "") + d; + } + } + return this; +}; +E.prototype.encode = function(a) { + if (this.cache && a.length <= this.h) { + if (this.X) { + if (this.P.has(a)) { + return this.P.get(a); } } else { - b = b[e] || (b[e] = x()), e = c[++d], Y(a, b, c, d, e); + this.X = setTimeout(la, 0, this); + } + } + this.normalize && ("function" === typeof this.normalize ? a = this.normalize(a) : ja ? a = a.normalize("NFKD").replace(ja, "").toLowerCase() : (a = a.toLowerCase(), this.D = this.D ? new Map([...ka, ...this.D]) : new Map(ka))); + this.$ && (a = this.$(a)); + this.numeric && 3 < a.length && (a = a.replace(ha, "$1 $2").replace(ia, "$1 $2").replace(fa, "$1 ")); + const b = !(this.C || this.D || this.filter || this.J || this.N || this.K); + let c = [], d = this.split || "" === this.split ? a.split(this.split) : a; + for (let f = 0, g, h; f < d.length; f++) { + if (!(g = h = d[f])) { + continue; + } + if (g.length < this.Z) { + continue; + } + if (b) { + c.push(g); + continue; + } + if (this.filter && this.filter.has(g)) { + continue; + } + if (this.cache && g.length <= this.B) { + if (this.X) { + var e = this.U.get(g); + if (e || "" === e) { + e && c.push(e); + continue; + } + } else { + this.X = setTimeout(la, 0, this); + } + } + let k; + this.N && 2 < g.length && (this.ca || (this.ca = new RegExp("(?!^)(" + this.Y + ")$")), g = g.replace(this.ca, l => this.N.get(l)), k = 1); + this.J && 1 < g.length && (this.ba || (this.ba = new RegExp("(" + this.F + ")", "g")), g = g.replace(this.ba, l => this.J.get(l)), k = 1); + g && k && (g.length < this.Z || this.filter && this.filter.has(g)) && (g = ""); + if (g && (this.D || this.C && 1 < g.length)) { + e = ""; + for (let l = 0, m = "", n, q; l < g.length; l++) { + n = g.charAt(l), n === m && this.C || ((q = this.D && this.D.get(n)) || "" === q ? q === m && this.C || !(m = q) || (e += q) : e += m = n); + } + g = e; + } + if (g && this.K) { + for (e = 0; g && e < this.K.length; e += 2) { + g = g.replace(this.K[e], this.K[e + 1]); + } + } + this.cache && h.length <= this.B && (this.U.set(h, g), this.U.size > this.ea && (this.U.clear(), this.B = this.B / 1.1 | 0)); + g && c.push(g); + } + this.V && (c = this.V(c) || c); + this.cache && a.length <= this.h && (this.P.set(a, c), this.P.size > this.ea && (this.P.clear(), this.h = this.h / 1.1 | 0)); + return c; +}; +function la(a) { + a.X = null; + a.P.clear(); + a.U.clear(); +} +;function ma(a, b, c) { + a = ("object" === typeof a ? "" + a.query : a).toLowerCase(); + let d = this.cache.get(a); + if (!d) { + d = this.search(a, b, c); + if (d instanceof Promise) { + const e = this; + d.then(function(f) { + e.cache.set(a, f); + }); + } + this.cache.set(a, d); + } + return d; +} +function F(a) { + this.limit = a && !0 !== a ? a : 1000; + this.cache = new Map(); + this.h = ""; +} +F.prototype.set = function(a, b) { + this.cache.has(a) || (this.cache.set(this.h = a, b), this.limit && this.cache.size > this.limit && this.cache.delete(this.cache.keys().next().value)); +}; +F.prototype.get = function(a) { + const b = this.cache.get(a); + b && this.limit && this.h !== a && (this.cache.delete(a), this.cache.set(this.h = a, b)); + return b; +}; +F.prototype.remove = function(a) { + for (const b of this.cache) { + const c = b[0]; + b[1].includes(a) && this.cache.delete(c); + } +}; +F.prototype.clear = function() { + this.cache.clear(); + this.h = ""; +}; +function na(a, b, c, d) { + let e = []; + for (let f = 0, g; f < a.index.length; f++) { + if (g = a.index[f], b >= g.length) { + b -= g.length; + } else { + b = g[d ? "splice" : "slice"](b, c); + const h = b.length; + if (h && (e = e.length ? e.concat(b) : b, c -= h, d && (a.length -= h), !c)) { + break; + } + b = 0; + } + } + return e; +} +function I(a) { + if (!(this instanceof I)) { + return new I(a); + } + this.index = a ? [a] : []; + this.length = a ? a.length : 0; + const b = this; + return new Proxy([], {get(c, d) { + if ("length" === d) { + return b.length; + } + if ("push" === d) { + return function(e) { + b.index[b.index.length - 1].push(e); + b.length++; + }; + } + if ("pop" === d) { + return function() { + if (b.length) { + return b.length--, b.index[b.index.length - 1].pop(); + } + }; + } + if ("indexOf" === d) { + return function(e) { + let f = 0; + for (let g = 0, h, k; g < b.index.length; g++) { + h = b.index[g]; + k = h.indexOf(e); + if (0 <= k) { + return f + k; + } + f += h.length; + } + return -1; + }; + } + if ("includes" === d) { + return function(e) { + for (let f = 0; f < b.index.length; f++) { + if (b.index[f].includes(e)) { + return !0; + } + } + return !1; + }; + } + if ("slice" === d) { + return function(e, f) { + return na(b, e || 0, f || b.length, !1); + }; + } + if ("splice" === d) { + return function(e, f) { + return na(b, e || 0, f || b.length, !0); + }; + } + if ("constructor" === d) { + return Array; + } + if ("symbol" !== typeof d) { + return (c = b.index[d / 2 ** 31 | 0]) && c[d]; + } + }, set(c, d, e) { + c = d / 2 ** 31 | 0; + (b.index[c] || (b.index[c] = []))[d] = e; + b.length++; + return !0; + }}); +} +I.prototype.clear = function() { + this.index.length = 0; +}; +I.prototype.push = function() { +}; +function L(a = 8) { + if (!(this instanceof L)) { + return new L(a); + } + this.index = z(); + this.F = []; + this.size = 0; + 32 < a ? (this.h = oa, this.B = BigInt(a)) : (this.h = pa, this.B = a); +} +L.prototype.get = function(a) { + const b = this.index[this.h(a)]; + return b && b.get(a); +}; +L.prototype.set = function(a, b) { + var c = this.h(a); + let d = this.index[c]; + d ? (c = d.size, d.set(a, b), (c -= d.size) && this.size++) : (this.index[c] = d = new Map([[a, b]]), this.F.push(d)); +}; +function M(a = 8) { + if (!(this instanceof M)) { + return new M(a); + } + this.index = z(); + this.h = []; + 32 < a ? (this.F = oa, this.B = BigInt(a)) : (this.F = pa, this.B = a); +} +M.prototype.add = function(a) { + var b = this.F(a); + let c = this.index[b]; + c ? (b = c.size, c.add(a), (b -= c.size) && this.size++) : (this.index[b] = c = new Set([a]), this.h.push(c)); +}; +u = L.prototype; +u.has = M.prototype.has = function(a) { + const b = this.index[this.F(a)]; + return b && b.has(a); +}; +u.delete = M.prototype.delete = function(a) { + const b = this.index[this.F(a)]; + b && b.delete(a) && this.size--; +}; +u.clear = M.prototype.clear = function() { + this.index = z(); + this.h = []; + this.size = 0; +}; +u.values = M.prototype.values = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let b of this.h[a].values()) { + yield b; + } + } +}; +u.keys = M.prototype.keys = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let b of this.h[a].keys()) { + yield b; + } + } +}; +u.entries = M.prototype.entries = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let b of this.h[a].entries()) { + yield b; + } + } +}; +function pa(a) { + let b = 2 ** this.B - 1; + if ("number" == typeof a) { + return a & b; + } + let c = 0, d = this.B + 1; + for (let e = 0; e < a.length; e++) { + c = (c * d ^ a.charCodeAt(e)) & b; + } + return 32 === this.B ? c + 2 ** 31 : c; +} +function oa(a) { + let b = BigInt(2) ** this.B - BigInt(1); + var c = typeof a; + if ("bigint" === c) { + return a & b; + } + if ("number" === c) { + return BigInt(a) & b; + } + c = BigInt(0); + let d = this.B + BigInt(1); + for (let e = 0; e < a.length; e++) { + c = (c * d ^ BigInt(a.charCodeAt(e))) & b; + } + return c; +} +;function qa(a, b, c, d, e, f, g, h) { + (d = a(c ? c + "." + d : d, JSON.stringify(g))) && d.then ? d.then(function() { + b.export(a, b, c, e, f + 1, h); + }) : b.export(a, b, c, e, f + 1, h); +} +;const ra = z(), N = z(); +var sa = {normalize:function(a) { + return a.toLowerCase(); +}, C:!1}; +const ta = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +function ua(a) { + O.call(a, "add"); + O.call(a, "append"); + O.call(a, "search"); + O.call(a, "update"); + O.call(a, "remove"); +} +function O(a) { + this[a + "Async"] = function() { + var b = arguments; + const c = b[b.length - 1]; + let d; + "function" === typeof c && (d = c, delete b[b.length - 1]); + this.async = !0; + b = this[a].apply(this, b); + this.async = !1; + b.then ? b.then(d) : d(b); + return b; + }; +} +;z(); +P.prototype.add = function(a, b, c, d) { + if (b && (a || 0 === a)) { + if (!d && !c && this.A.has(a)) { + return this.update(a, b); + } + b = this.encoder.encode(b); + if (d = b.length) { + const l = z(), m = z(), n = this.depth, q = this.resolution; + for (let t = 0; t < d; t++) { + let p = b[this.rtl ? d - 1 - t : t]; + var e = p.length; + if (e && (n || !m[p])) { + var f = this.score ? this.score(b, p, t, null, 0) : Q(q, d, t), g = ""; + switch(this.tokenize) { + case "full": + if (2 < e) { + for (f = 0; f < e; f++) { + for (var h = e; h > f; h--) { + g = p.substring(f, h); + var k = this.score ? this.score(b, p, t, g, f) : Q(q, d, t, e, f); + R(this, m, g, k, a, c); + } + } + break; + } + case "reverse": + if (1 < e) { + for (h = e - 1; 0 < h; h--) { + g = p[h] + g, k = this.score ? this.score(b, p, t, g, h) : Q(q, d, t, e, h), R(this, m, g, k, a, c); + } + g = ""; + } + case "forward": + if (1 < e) { + for (h = 0; h < e; h++) { + g += p[h], R(this, m, g, f, a, c); + } + break; + } + default: + if (R(this, m, p, f, a, c), n && 1 < d && t < d - 1) { + for (e = z(), g = this.da, f = p, h = Math.min(n + 1, d - t), e[f] = 1, k = 1; k < h; k++) { + if ((p = b[this.rtl ? d - 1 - t - k : t + k]) && !e[p]) { + e[p] = 1; + const r = this.score ? this.score(b, f, t, p, k) : Q(g + (d / 2 > g ? 0 : 1), d, t, h - 1, k - 1), x = this.bidirectional && p > f; + R(this, l, x ? f : p, r, a, c, x ? p : f); + } + } + } + } + } + } + this.fastupdate || this.A.add(a); + } else { + b = ""; + } + } + this.db && (b || this.R.push({del:a}), this.fa && va(this)); + return this; +}; +function R(a, b, c, d, e, f, g) { + let h = g ? a.I : a.map, k; + if (!b[c] || !g || !(k = b[c])[g]) { + if (g ? (b = k || (b[c] = z()), b[g] = 1, (k = h.get(g)) ? h = k : h.set(g, h = new Map())) : b[c] = 1, (k = h.get(c)) ? h = k : h.set(c, h = k = []), h = h[d] || (h[d] = []), !f || !h.includes(e)) { + if (h.length === 2 ** 31 - 1) { + b = new I(h); + if (a.fastupdate) { + for (let l of a.A.values()) { + l.includes(h) && (l[l.indexOf(h)] = b); + } + } + k[d] = h = b; + } + h.push(e); + a.fastupdate && ((d = a.A.get(e)) ? d.push(h) : a.A.set(e, [h])); } } } -function Z(a, b, c, d, e, f, h, g) { - if (a = a[h]) { +function Q(a, b, c, d, e) { + return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; +} +;function S(a, b, c, d) { + if (1 === a.length) { + return a = a[0], a = c || a.length > b ? b ? a.slice(c, c + b) : a.slice(c) : a, d ? wa(a) : a; + } + let e = []; + for (let f = 0, g, h; f < a.length; f++) { + if ((g = a[f]) && (h = g.length)) { + if (c) { + if (c >= h) { + c -= h; + continue; + } + c < h && (g = b ? g.slice(c, c + b) : g.slice(c), h = g.length, c = 0); + } + if (e.length) { + h > b && (g = g.slice(0, b), h = g.length), e.push(g); + } else { + if (h >= b) { + return h > b && (g = g.slice(0, b)), d ? wa(g) : g; + } + e = [g]; + } + b -= h; + if (!b) { + break; + } + } + } + if (!e.length) { + return e; + } + e = 1 < e.length ? [].concat.apply([], e) : e[0]; + return d ? wa(e) : e; +} +function wa(a) { + for (let b = 0; b < a.length; b++) { + a[b] = {score:b, id:a[b]}; + } + return a; +} +;T.prototype.or = function() { + const a = this; + let b = arguments; + var c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.or.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.or.apply(this, c); + } + let d = []; + c = []; + let e = 0, f = 0, g, h; + for (let k = 0, l; k < b.length; k++) { + if (l = b[k]) { + let m; + if (l instanceof T) { + m = l.result; + } else if (l.constructor === Array) { + m = l; + } else if (l.index) { + l.resolve = !1, m = l.index.search(l).result; + } else if (l.and) { + m = this.and(l.and); + } else if (l.xor) { + m = this.xor(l.xor); + } else if (l.G) { + m = this.G(l.G); + } else { + e = l.limit || 0; + f = l.offset || 0; + g = l.enrich; + h = l.resolve; + continue; + } + d[k] = m; + m instanceof Promise && c.push(m); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result.length && (d = [a.result].concat(d)); + a.result = xa(d, e, f, g, h, a.O); + return h ? a.result : a; + }); + } + this.result.length && (d = [this.result].concat(d)); + this.result = xa(d, e, f, g, h, a.O); + return h ? this.result : this; +}; +function xa(a, b, c, d, e, f) { + if (!a.length) { + return a; + } + "object" === typeof b && (c = b.offset || 0, d = b.enrich || !1, b = b.limit || 0); + if (2 > a.length) { + return e ? S(a[0], b, c, d) : a[0]; + } + d = []; + let g = 0, h = z(), k = da(a); + for (let l = 0, m; l < k; l++) { + for (let n = 0, q; n < a.length; n++) { + if (q = a[n]) { + if (m = q[l]) { + for (let t = 0, p; t < m.length; t++) { + if (p = m[t], !h[p]) { + if (h[p] = 1, c) { + c--; + } else { + if (e) { + d.push(p); + } else { + const r = l + (n ? f : 0); + d[r] || (d[r] = []); + d[r].push(p); + } + if (b && ++g === b) { + return d; + } + } + } + } + } + } + } + } + return d; +} +;T.prototype.and = function() { + if (this.result.length) { + const b = this; + let c = arguments; + var a = c[0]; + if (a instanceof Promise) { + return a.then(function() { + return b.and.apply(b, c); + }); + } + if (a[0] && a[0].index) { + return this.and.apply(this, a); + } + let d = []; + a = []; + let e = 0, f = 0, g; + for (let h = 0, k; h < c.length; h++) { + if (k = c[h]) { + let l; + if (k instanceof T) { + l = k.result; + } else if (k.constructor === Array) { + l = k; + } else if (k.index) { + k.resolve = !1, l = k.index.search(k).result; + } else if (k.or) { + l = this.or(k.or); + } else if (k.xor) { + l = this.xor(k.xor); + } else if (k.G) { + l = this.G(k.G); + } else { + e = k.limit || 0; + f = k.offset || 0; + g = k.resolve; + continue; + } + d[h] = l; + l instanceof Promise && a.push(l); + } + } + if (a.length) { + return Promise.all(a).then(function() { + d = [b.result].concat(d); + b.result = ya(d, e, f, g, b.O); + return g ? b.result : b; + }); + } + d = [this.result].concat(d); + this.result = ya(d, e, f, g, b.O); + return g ? this.result : this; + } + return this; +}; +function ya(a, b, c, d, e) { + if (2 > a.length) { + return []; + } + let f = [], g = 0, h = z(), k = da(a); + if (!k) { + return f; + } + for (let l = 0, m; l < a.length; l++) { + m = a[l]; + if (!m || !m.length) { + return []; + } + let n = z(), q = 0, t = l === a.length - 1; + for (let p = 0, r; p < k; p++) { + if (r = m[p]) { + for (let x = 0, A, w; x < r.length; x++) { + if (A = r[x], !l) { + n[A] = p + 1 + (l ? e : 0), q = 1; + } else if (t) { + if (w = h[A]) { + if (q = 1, c) { + c--; + } else { + if (d ? f.push(A) : (w--, p < w && (w = p), f[w] || (f[w] = []), f[w].push(A)), b && ++g === b) { + return f; + } + } + } + } else if (w = h[A]) { + p + 1 < w && (w = p + 1), n[A] = w, q = 1; + } + } + } + } + if (!q) { + return []; + } + h = n; + } + return f; +} +;T.prototype.xor = function() { + const a = this; + let b = arguments; + var c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.xor.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.xor.apply(this, c); + } + let d = []; + c = []; + let e = 0, f = 0, g, h; + for (let k = 0, l; k < b.length; k++) { + if (l = b[k]) { + let m; + if (l instanceof T) { + m = l.result; + } else if (l.constructor === Array) { + m = l; + } else if (l.index) { + l.resolve = !1, m = l.index.search(l).result; + } else if (l.or) { + m = this.or(l.or); + } else if (l.and) { + m = this.and(l.and); + } else if (l.G) { + m = this.G(l.G); + } else { + e = l.limit || 0; + f = l.offset || 0; + g = l.enrich; + h = l.resolve; + continue; + } + d[k] = m; + m instanceof Promise && c.push(m); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result.length && (d = [a.result].concat(d)); + a.result = za(d, e, f, g, !h, a.O); + return h ? a.result : a; + }); + } + this.result.length && (d = [this.result].concat(d)); + this.result = za(d, e, f, g, !h, a.O); + return h ? this.result : this; +}; +function za(a, b, c, d, e, f) { + if (!a.length) { + return a; + } + if (2 > a.length) { + return e ? S(a[0], b, c, d) : a[0]; + } + b = []; + c = z(); + for (let g = 0, h; g < a.length; g++) { + if (h = a[g]) { + for (let k = 0, l; k < h.length; k++) { + if (l = h[k]) { + for (let m = 0, n; m < l.length; m++) { + n = l[m], c[n] ? c[n]++ : c[n] = 1; + } + } + } + } + } + for (let g = 0, h; g < a.length; g++) { + if (h = a[g]) { + for (let k = 0, l; k < h.length; k++) { + if (l = h[k]) { + for (let m = 0, n; m < l.length; m++) { + n = l[m], 1 === c[n] && (e ? b.push(n) : (d = k + (g ? f : 0), b[d] || (b[d] = []), b[d].push(n))); + } + } + } + } + } + return b; +} +;T.prototype.G = function() { + const a = this; + let b = arguments; + var c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.G.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.G.apply(this, c); + } + let d = []; + c = []; + let e; + for (let f = 0, g; f < b.length; f++) { + if (g = b[f]) { + let h; + if (g instanceof T) { + h = g.result; + } else if (g.constructor === Array) { + h = g; + } else if (g.index) { + g.resolve = !1, h = g.index.search(g).result; + } else if (g.or) { + h = this.or(g.or); + } else if (g.and) { + h = this.and(g.and); + } else if (g.xor) { + h = this.xor(g.xor); + } else { + e = g.resolve; + continue; + } + d[f] = h; + h instanceof Promise && c.push(h); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result = Aa.call(a, d, e); + return e ? a.result : a; + }); + } + this.result = Aa.call(this, d, e); + return e ? this.result : this; +}; +function Aa(a, b) { + if (!a.length) { + return this.result; + } + const c = []; + a = new Set(a.flat().flat()); + for (let d = 0, e; d < this.result.length; d++) { + if (e = this.result[d]) { + for (let f = 0, g; f < e.length; f++) { + g = e[f], a.has(g) || (b ? c.push(g) : (c[d] || (c[d] = []), c[d].push(g))); + } + } + } + return c; +} +;function T(a) { + if (a && a.index) { + return a.resolve = !1, this.index = a.index, a.index.search(a); + } + if (!(this instanceof T)) { + return new T(a); + } + if (a instanceof T) { + return a; + } + this.index = null; + this.result = a || []; + this.O = 0; +} +T.prototype.limit = function(a) { + if (this.result.length) { + const b = []; + let c = 0; + for (let d = 0, e; d < this.result.length; d++) { + if (e = this.result[d], e.length + c < a) { + b[d] = e, c += e.length; + } else { + b[d] = e.slice(0, a - c); + this.result = b; + break; + } + } + } + return this; +}; +T.prototype.offset = function(a) { + if (this.result.length) { + const b = []; + let c = 0; + for (let d = 0, e; d < this.result.length; d++) { + e = this.result[d], e.length + c < a ? c += e.length : (b[d] = e.slice(a - c), c = a); + } + this.result = b; + } + return this; +}; +T.prototype.resolve = function(a, b, c) { + U = 1; + const d = this.result; + this.result = this.index = null; + return d.length ? ("object" === typeof a && (c = a.enrich, b = a.offset, a = a.limit), S(d, a || 100, b, c)) : d; +}; +function Ba(a, b, c, d) { + var e = a.length; + let f = [], g = 0, h, k, l; + d && (d = []); + for (let m = e - 1, n; 0 <= m; m--) { + l = a[m]; + e = z(); + n = !h; + for (let q = 0, t; q < l.length; q++) { + if ((t = l[q]) && t.length) { + for (let p = 0, r; p < t.length; p++) { + if (r = t[p], h) { + if (h[r]) { + if (!m) { + if (c) { + c--; + } else { + if (f[g++] = r, g === b) { + return f; + } + } + } + if (m || d) { + e[r] = 1; + } + n = !0; + } + d && !k[r] && (k[r] = 1, (d[q] || (d[q] = [])).push(r)); + } else { + e[r] = 1; + } + } + } + } + if (d) { + h || (k = e); + } else if (!n) { + return []; + } + h = e; + } + if (d) { + for (let m = d.length - 1, n, q; 0 <= m; m--) { + n = d[m]; + q = n.length; + for (let t = 0, p; t < q; t++) { + if (p = n[t], !h[p]) { + if (c) { + c--; + } else { + if (f[g++] = p, g === b) { + return f; + } + } + h[p] = 1; + } + } + } + } + return f; +} +function Ca(a, b) { + const c = z(), d = z(), e = []; + for (let f = 0; f < a.length; f++) { + c[a[f]] = 1; + } + for (let f = 0, g; f < b.length; f++) { + g = b[f]; + for (let h = 0, k; h < g.length; h++) { + k = g[h], c[k] && !d[k] && (d[k] = 1, e.push(k)); + } + } + return e; +} +;let U = 1; +P.prototype.search = function(a, b, c) { + c || (!b && C(a) ? (c = a, a = "") : C(b) && (c = b, b = 0)); + let d = [], e; + let f, g = 0, h, k, l; + if (c) { + a = c.query || a; + b = c.limit || b; + g = c.offset || 0; + var m = c.context; + f = c.suggest; + (h = U && !1 !== c.resolve) || (U = 0); + k = h && c.enrich; + l = this.db && c.tag; + } else { + h = this.resolve || U; + } + a = this.encoder.encode(a); + e = a.length; + b || !h || (b = 100); + if (1 === e) { + return Da.call(this, a[0], "", b, g, h, k, l); + } + m = this.depth && !1 !== m; + if (2 === e && m && !f) { + return Da.call(this, a[0], a[1], b, g, h, k, l); + } + let n = c = 0; + if (1 < e) { + const p = z(), r = []; + for (let x = 0, A; x < e; x++) { + if ((A = a[x]) && !p[A]) { + if (f || this.db || V(this, A)) { + r.push(A), p[A] = 1; + } else { + return h ? d : new T(d); + } + const w = A.length; + c = Math.max(c, w); + n = n ? Math.min(n, w) : w; + } + } + a = r; + e = a.length; + } + if (!e) { + return h ? d : new T(d); + } + let q = 0, t; + if (1 === e) { + return Da.call(this, a[0], "", b, g, h, k, l); + } + if (2 === e && m && !f) { + return Da.call(this, a[0], a[1], b, g, h, k, l); + } + 1 < e && (m ? (t = a[0], q = 1) : 9 < c && 3 < c / n && a.sort(aa)); + if (this.db) { + if (this.db.search && (m = this.db.search(this, a, b, g, f, h, k, l), !1 !== m)) { + return m; + } + const p = this; + return async function() { + for (let r, x; q < e; q++) { + x = a[q]; + t ? (r = await V(p, x, t), r = Ea(r, d, f, p.da, b, g, 2 === e), f && !1 === r && d.length || (t = x)) : (r = await V(p, x), r = Ea(r, d, f, p.resolution, b, g, 1 === e)); + if (r) { + return r; + } + if (f && q === e - 1) { + let A = d.length; + if (!A) { + if (t) { + t = ""; + q = -1; + continue; + } + return d; + } + if (1 === A) { + return h ? S(d[0], b, g) : new T(d[0]); + } + } + } + return h ? Ba(d, b, g, f) : new T(d[0]); + }(); + } + for (let p, r; q < e; q++) { + r = a[q]; + t ? (p = V(this, r, t), p = Ea(p, d, f, this.da, b, g, 2 === e), f && !1 === p && d.length || (t = r)) : (p = V(this, r), p = Ea(p, d, f, this.resolution, b, g, 1 === e)); + if (p) { + return p; + } + if (f && q === e - 1) { + m = d.length; + if (!m) { + if (t) { + t = ""; + q = -1; + continue; + } + return d; + } + if (1 === m) { + return h ? S(d[0], b, g) : new T(d[0]); + } + } + } + return h ? Ba(d, b, g, f) : new T(d[0]); +}; +function Da(a, b, c, d, e, f, g) { + a = V(this, a, b, c, d, e, f, g); + return this.db ? a.then(function(h) { + return e ? h : h && h.length ? e ? S(h, c, d) : new T(h) : e ? [] : new T([]); + }) : a && a.length ? e ? S(a, c, d) : new T(a) : e ? [] : new T([]); +} +function Ea(a, b, c, d, e, f, g) { + let h = []; + if (a) { + d = Math.min(a.length, d); + for (let k = 0, l = 0, m; k < d; k++) { + if (m = a[k]) { + if (f && m && g && (m.length <= f ? (f -= m.length, m = null) : (m = m.slice(f), f = 0)), m && (h[k] = m, g && (l += m.length, l >= e))) { + break; + } + } + } + if (h.length) { + if (g) { + return S(h, e, 0); + } + b.push(h); + return; + } + } + return !c && h; +} +function V(a, b, c, d, e, f, g, h) { + let k; + c && (k = a.bidirectional && b > c); + if (a.db) { + return c ? a.db.get(k ? c : b, k ? b : c, d, e, f, g, h) : a.db.get(b, "", d, e, f, g, h); + } + a = c ? (a = a.I.get(k ? b : c)) && a.get(k ? c : b) : a.map.get(b); + return a; +} +;P.prototype.remove = function(a, b) { + const c = this.A.size && (this.fastupdate ? this.A.get(a) : this.A.has(a)); + if (c) { + if (this.fastupdate) { + for (let d = 0, e; d < c.length; d++) { + if (e = c[d]) { + if (2 > e.length) { + e.pop(); + } else { + const f = e.indexOf(a); + f === c.length - 1 ? e.pop() : e.splice(f, 1); + } + } + } + } else { + Fa(this.map, a), this.depth && Fa(this.I, a); + } + b || this.A.delete(a); + } + this.db && (this.R.push({del:a}), this.fa && va(this)); + this.cache && this.cache.remove(a); + return this; +}; +function Fa(a, b) { + let c = 0; + if (a.constructor === Array) { + for (let d = 0, e, f; d < a.length; d++) { + if ((e = a[d]) && e.length) { + if (f = e.indexOf(b), 0 <= f) { + 1 < e.length ? (e.splice(f, 1), c++) : delete a[d]; + break; + } else { + c++; + } + } + } + } else { + for (let d of a) { + const e = d[0], f = Fa(d[1], b); + f ? c += f : a.delete(e); + } + } + return c; +} +;function P(a, b) { + if (!(this instanceof P)) { + return new P(a); + } + if (a) { + var c = B(a) ? a : a.preset; + c && (ta[c] || console.warn("Preset not found: " + c), a = Object.assign({}, ta[c], a)); + } else { + a = {}; + } + c = a.context || {}; + const d = a.encode || a.encoder || sa; + this.encoder = d.encode ? d : "object" === typeof d ? new E(d) : {encode:d}; + let e; + this.resolution = a.resolution || 9; + this.tokenize = e = a.tokenize || "strict"; + this.depth = "strict" === e && c.depth || 0; + this.bidirectional = !1 !== c.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + (e = a.keystore || 0) && (this.keystore = e); + this.map = e ? new L(e) : new Map(); + this.I = e ? new L(e) : new Map(); + this.A = b || (this.fastupdate ? e ? new L(e) : new Map() : e ? new M(e) : new Set()); + this.da = c.resolution || 1; + this.rtl = d.rtl || a.rtl || !1; + this.cache = (e = a.cache || null) && new F(e); + this.resolve = !1 !== a.resolve; + if (e = a.db) { + this.db = e.mount(this); + } + this.fa = !1 !== a.commit; + this.R = []; + this.h = null; +} +u = P.prototype; +u.mount = function(a) { + this.h && (clearTimeout(this.h), this.h = null); + return a.mount(this); +}; +u.commit = function(a, b) { + this.h && (clearTimeout(this.h), this.h = null); + return this.db.commit(this, a, b); +}; +function va(a) { + a.h || (a.h = setTimeout(function() { + a.h = null; + a.db.commit(a, void 0, void 0); + }, 0)); +} +u.clear = function() { + this.map.clear(); + this.I.clear(); + this.A.clear(); + this.cache && this.cache.clear(); + this.db && (this.h && clearTimeout(this.h), this.h = null, this.R = [{clear:!0}]); + return this; +}; +u.append = function(a, b) { + return this.add(a, b, !0); +}; +u.contain = function(a) { + return this.db ? this.db.has(a) : this.A.has(a); +}; +u.update = function(a, b) { + if (this.async) { + const c = this, d = this.remove(a); + return d.then ? d.then(() => c.add(a, b)) : this.add(a, b); + } + return this.remove(a).add(a, b); +}; +function Ga(a) { + let b = 0; + if (a.constructor === Array) { + for (let c = 0, d; c < a.length; c++) { + (d = a[c]) && (b += d.length); + } + } else { + for (const c of a) { + const d = c[0], e = Ga(c[1]); + e ? b += e : a.delete(d); + } + } + return b; +} +u.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + Ga(this.map); + this.depth && Ga(this.I); + return this; +}; +u.searchCache = ma; +u.export = function(a, b, c, d, e, f) { + let g = !0; + "undefined" === typeof f && (g = new Promise(l => { + f = l; + })); + let h, k; + switch(e || (e = 0)) { + case 0: + h = "reg"; + if (this.fastupdate) { + k = z(); + for (let l of this.A.keys()) { + k[l] = 1; + } + } else { + k = this.A; + } + break; + case 1: + h = "cfg"; + k = {doc:0, opt:this.B ? 1 : 0}; + break; + case 2: + h = "map"; + k = this.map; + break; + case 3: + h = "ctx"; + k = this.I; + break; + default: + "undefined" === typeof c && f && f(); + return; + } + qa(a, b || this, c, h, d, e, k, f); + return g; +}; +u.import = function(a, b) { + if (b) { + switch(B(b) && (b = JSON.parse(b)), a) { + case "cfg": + this.B = !!b.opt; + break; + case "reg": + this.fastupdate = !1; + this.A = b; + break; + case "map": + this.map = b; + break; + case "ctx": + this.I = b; + } + } +}; +ua(P.prototype); +async function Ha(a) { + a = a.data; + var b = self._index; + const c = a.args; + var d = a.task; + switch(d) { + case "init": + d = a.options || {}; + (b = d.config) && (d = b); + (b = a.factory) ? (Function("return " + b)()(self), self._index = new self.FlexSearch.Index(d), delete self.FlexSearch) : self._index = new P(d); + postMessage({id:a.id}); + break; + default: + a = a.id, b = b[d].apply(b, c), postMessage("search" === d ? {id:a, msg:b} : {id:a}); + } +} +;let Ia = 0; +function W(a) { + function b(f) { + f = f.data || f; + const g = f.id, h = g && e.h[g]; + h && (h(f.msg), delete e.h[g]); + } + if (!(this instanceof W)) { + return new W(a); + } + a || (a = {}); + let c = (self || window)._factory; + c && (c = c.toString()); + const d = "undefined" === typeof window && self.exports, e = this; + this.worker = Ja(c, d, a.worker); + this.h = z(); + if (this.worker) { + d ? this.worker.on("message", b) : this.worker.onmessage = b; + if (a.config) { + return new Promise(function(f) { + e.h[++Ia] = function() { + f(e); + }; + e.worker.postMessage({id:Ia, task:"init", factory:c, options:a}); + }); + } + this.worker.postMessage({task:"init", factory:c, options:a}); + } +} +X("add"); +X("append"); +X("search"); +X("update"); +X("remove"); +function X(a) { + W.prototype[a] = W.prototype[a + "Async"] = function() { + const b = this, c = [].slice.call(arguments); + var d = c[c.length - 1]; + let e; + "function" === typeof d && (e = d, c.splice(c.length - 1, 1)); + d = new Promise(function(f) { + b.h[++Ia] = f; + b.worker.postMessage({task:a, id:Ia, args:c}); + }); + return e ? (d.then(e), this) : d; + }; +} +function Ja(a, b, c) { + return b ? new (require("worker_threads")["Worker"])(__dirname + "/node/node.js") : a ? new window.Worker(URL.createObjectURL(new Blob(["onmessage=" + Ha.toString()], {type:"text/javascript"}))) : new window.Worker(B(c) ? c : "worker/worker.js", {type:"module"}); +} +;Y.prototype.add = function(a, b, c) { + C(a) && (b = a, a = ca(b, this.key)); + if (b && (a || 0 === a)) { + if (!c && this.A.has(a)) { + return this.update(a, b); + } + for (let h = 0, k; h < this.field.length; h++) { + k = this.M[h]; + var d = this.index.get(this.field[h]); + if ("function" === typeof k) { + var e = k(b); + e && d.add(a, e, !1, !0); + } else { + if (e = k.T, !e || e(b)) { + k instanceof String ? k = ["" + k] : B(k) && (k = [k]), Ka(b, k, this.W, 0, d, a, k[0], c); + } + } + } + if (this.tag) { + for (d = 0; d < this.L.length; d++) { + var f = this.L[d], g = this.aa[d]; + e = this.tag.get(g); + let h = z(); + if ("function" === typeof f) { + if (f = f(b), !f) { + continue; + } + } else { + const k = f.T; + if (k && !k(b)) { + continue; + } + f instanceof String && (f = "" + f); + f = ca(b, f); + } + if (e && f) { + B(f) && (f = [f]); + for (let k = 0, l, m; k < f.length; k++) { + if (l = f[k], !h[l] && (h[l] = 1, (g = e.get(l)) ? m = g : e.set(l, m = []), !c || !m.includes(a))) { + if (m.length === 2 ** 31 - 1) { + g = new I(m); + if (this.fastupdate) { + for (let n of this.A.values()) { + n.includes(m) && (n[n.indexOf(m)] = g); + } + } + e.set(l, m = g); + } + m.push(a); + this.fastupdate && ((g = this.A.get(a)) ? g.push(m) : this.A.set(a, [m])); + } + } + } else { + e || console.warn("Tag '" + g + "' was not found"); + } + } + } + if (this.store && (!c || !this.store.has(a))) { + let h; + if (this.H) { + h = z(); + for (let k = 0, l; k < this.H.length; k++) { + l = this.H[k]; + if ((c = l.T) && !c(b)) { + continue; + } + let m; + if ("function" === typeof l) { + m = l(b); + if (!m) { + continue; + } + l = [l.ia]; + } else if (B(l) || l instanceof String) { + h[l] = b[l]; + continue; + } + La(b, h, l, 0, l[0], m); + } + } + this.store.set(a, h || b); + } + } + return this; +}; +function La(a, b, c, d, e, f) { + a = a[e]; + if (d === c.length - 1) { + b[e] = f || a; + } else if (a) { + if (a.constructor === Array) { + for (b = b[e] = Array(a.length), e = 0; e < a.length; e++) { + La(a, b, c, d, e); + } + } else { + b = b[e] || (b[e] = z()), e = c[++d], La(a, b, c, d, e); + } + } +} +function Ka(a, b, c, d, e, f, g, h) { + if (a = a[g]) { if (d === b.length - 1) { if (a.constructor === Array) { if (c[d]) { @@ -683,228 +1557,455 @@ function Z(a, b, c, d, e, f, h, g) { } a = a.join(" "); } - e.add(f, a, g, !0); + e.add(f, a, h, !0); } else { if (a.constructor === Array) { - for (h = 0; h < a.length; h++) { - Z(a, b, c, d, e, f, h, g); + for (g = 0; g < a.length; g++) { + Ka(a, b, c, d, e, f, g, h); } } else { - h = b[++d], Z(a, b, c, d, e, f, h, g); + g = b[++d], Ka(a, b, c, d, e, f, g, h); } } + } else { + e.db && e.remove(f); } } -t = U.prototype; -t.add = function(a, b, c) { - D(a) && (b = a, a = X(b, this.key)); - if (b && (a || 0 === a)) { - if (!c && this.register[a]) { - return this.update(a, b); - } - for (let d = 0, e, f; d < this.h.length; d++) { - f = this.h[d], e = this.K[d], C(e) && (e = [e]), Z(b, e, this.A, 0, this.index[f], a, e[0], c); - } - if (this.I) { - let d = X(b, this.I), e = x(); - C(d) && (d = [d]); - for (let f = 0, h, g; f < d.length; f++) { - if (h = d[f], !e[h] && (e[h] = 1, g = this.l[h] || (this.l[h] = []), !c || !g.includes(a))) { - if (g[g.length] = a, this.m) { - const k = this.register[a] || (this.register[a] = []); - k[k.length] = g; - } - } - } - } - if (this.store && (!c || !this.store[a])) { - let d; - if (this.C) { - d = x(); - for (let e = 0, f; e < this.C.length; e++) { - f = this.C[e], C(f) ? d[f] = b[f] : Y(b, d, f, 0, f[0]); - } - } - this.store[a] = d || b; - } - } - return this; -}; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a) { - D(a) && (a = X(a, this.key)); - if (this.register[a]) { - for (var b = 0; b < this.h.length && (this.index[this.h[b]].remove(a, !this.o), !this.m); b++) { - } - if (this.I && !this.m) { - for (let c in this.l) { - b = this.l[c]; - const d = b.indexOf(a); - -1 !== d && (1 < b.length ? b.splice(d, 1) : delete this.l[c]); - } - } - this.store && delete this.store[a]; - delete this.register[a]; - } - return this; -}; -t.search = function(a, b, c, d) { - c || (!b && D(a) ? (c = a, a = "") : D(b) && (c = b, b = 0)); - let e = [], f = [], h, g, k, m, n, w, q = 0; +;Y.prototype.search = function(a, b, c, d) { + c || (!b && C(a) ? (c = a, a = "") : C(b) && (c = b, b = 0)); + let e = [], f = [], g; + let h; + let k; + let l, m = 0; if (c) { if (c.constructor === Array) { k = c, c = null; } else { a = c.query || a; - k = (h = c.pluck) || c.index || c.field; - m = c.tag; - g = this.store && c.enrich; - n = "and" === c.bool; - b = c.limit || b || 100; - w = c.offset || 0; - if (m && (C(m) && (m = [m]), !a)) { - for (let l = 0, p; l < m.length; l++) { - if (p = va.call(this, m[l], b, w, g)) { - e[e.length] = p, q++; + g = c.pluck; + h = c.merge; + k = g || c.field || c.index; + var n = this.tag && c.tag; + var q = this.store && c.enrich; + var t = c.suggest; + b = c.limit || b; + l = c.offset || 0; + b || (b = 100); + if (n && (!this.db || !d)) { + n.constructor !== Array && (n = [n]); + var p = []; + for (let w = 0, v; w < n.length; w++) { + v = n[w]; + if (B(v)) { + throw Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + if (v.field && v.tag) { + var r = v.tag; + if (r.constructor === Array) { + for (var x = 0; x < r.length; x++) { + p.push(v.field, r[x]); + } + } else { + p.push(v.field, r); + } + } else { + r = Object.keys(v); + for (let J = 0, K, D; J < r.length; J++) { + if (K = r[J], D = v[K], D.constructor === Array) { + for (x = 0; x < D.length; x++) { + p.push(K, D[x]); + } + } else { + p.push(K, D); + } + } } } - return q ? e : []; + if (!p.length) { + throw Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + n = p; + if (!a) { + t = []; + if (p.length) { + for (n = 0; n < p.length; n += 2) { + if (this.db) { + d = this.index.get(p[n]); + if (!d) { + console.warn("Tag '" + p[n] + ":" + p[n + 1] + "' will be skipped because there is no field '" + p[n] + "'."); + continue; + } + t.push(d = d.db.tag(p[n + 1], b, l, q)); + } else { + d = Ma.call(this, p[n], p[n + 1], b, l, q); + } + e.push({field:p[n], tag:p[n + 1], result:d}); + } + } + return t.length ? Promise.all(t).then(function(w) { + for (let v = 0; v < w.length; v++) { + e[v].result = w[v]; + } + return e; + }) : e; + } } - C(k) && (k = [k]); + B(k) && (k = [k]); } } - k || (k = this.h); - n = n && (1 < k.length || m && 1 < m.length); - const r = !d && (this.o || this.async) && []; - for (let l = 0, p, A, B; l < k.length; l++) { - let z; - A = k[l]; - C(A) || (z = A, A = z.field, a = z.query || a, b = z.limit || b, g = z.enrich || g); - if (r) { - r[l] = this.index[A].searchAsync(a, b, z || c); + k || (k = this.field); + p = !d && (this.worker || this.async) && []; + let A; + for (let w = 0, v, J, K; w < k.length; w++) { + J = k[w]; + if (this.db && this.tag && !this.M[w]) { + continue; + } + let D; + B(J) || (D = J, J = D.field, a = D.query || a, b = D.limit || b, t = D.suggest || t); + if (d) { + v = d[w]; } else { - d ? p = d[l] : p = this.index[A].search(a, b, z || c); - B = p && p.length; - if (m && B) { - const y = []; - let H = 0; - n && (y[0] = [p]); - for (let W = 0, na, R; W < m.length; W++) { - if (na = m[W], B = (R = this.l[na]) && R.length) { - H++, y[y.length] = n ? [R] : R; - } - } - H && (p = n ? ja(y, b || 100, w || 0) : ka(p, y), B = p.length); - } - if (B) { - f[q] = A, e[q++] = p; - } else if (n) { - return []; + if (r = D || c, x = this.index.get(J), n && (this.db && (r.tag = n, A = x.db.la, r.field = k), A || (r.enrich = !1)), p) { + p[w] = x.searchAsync(a, b, r); + r && q && (r.enrich = q); + continue; + } else { + v = x.search(a, b, r), r && q && (r.enrich = q); } } + K = v && v.length; + if (n && K) { + r = []; + x = 0; + if (this.db && d) { + if (!A) { + for (let G = k.length; G < d.length; G++) { + let H = d[G]; + if (H && H.length) { + x++, r.push(H); + } else if (!t) { + return e; + } + } + } + } else { + for (let G = 0, H, Xa; G < n.length; G += 2) { + H = this.tag.get(n[G]); + if (!H) { + if (console.warn("Tag '" + n[G] + ":" + n[G + 1] + "' will be skipped because there is no field '" + n[G] + "'."), t) { + continue; + } else { + return e; + } + } + if (Xa = (H = H && H.get(n[G + 1])) && H.length) { + x++, r.push(H); + } else if (!t) { + return e; + } + } + } + if (x) { + v = Ca(v, r); + K = v.length; + if (!K && !t) { + return e; + } + x--; + } + } + if (K) { + f[m] = J, e.push(v), m++; + } else if (1 === k.length) { + return e; + } } - if (r) { - const l = this; - return new Promise(function(p) { - Promise.all(r).then(function(A) { - p(l.search(a, b, c, A)); - }); + if (p) { + if (this.db && n && n.length && !A) { + for (q = 0; q < n.length; q += 2) { + d = this.index.get(n[q]); + if (!d) { + if (console.warn("Tag '" + n[q] + ":" + n[q + 1] + "' was not found because there is no field '" + n[q] + "'."), t) { + continue; + } else { + return e; + } + } + p.push(d.db.tag(n[q + 1], b, l, !1)); + } + } + const w = this; + return Promise.all(p).then(function(v) { + return v.length ? w.search(a, b, c, v) : v; }); } - if (!q) { - return []; + if (!m) { + return e; } - if (h && (!g || !this.store)) { + if (g && (!q || !this.store)) { return e[0]; } - for (let l = 0, p; l < f.length; l++) { - p = e[l]; - p.length && g && (p = wa.call(this, p)); - if (h) { - return p; + p = []; + for (let w = 0, v; w < f.length; w++) { + v = e[w]; + q && v.length && !v[0].doc && (this.db ? p.push(v = this.index.get(this.field[0]).db.enrich(v)) : v.length && (v = Na.call(this, v))); + if (g) { + return v; } - e[l] = {field:f[l], result:p}; + e[w] = {field:f[w], result:v}; } - return e; -}; -function va(a, b, c, d) { - let e = this.l[a], f = e && e.length - c; - if (f && 0 < f) { - if (f > b || c) { - e = e.slice(c, c + b); + return q && this.db && p.length ? Promise.all(p).then(function(w) { + for (let v = 0; v < w.length; v++) { + e[v].result = w[v]; } - d && (e = wa.call(this, e)); - return {tag:a, result:e}; + return h ? Oa(e, b) : e; + }) : h ? Oa(e, b) : e; +}; +function Oa(a, b) { + const c = [], d = z(); + for (let e = 0, f, g; e < a.length; e++) { + f = a[e]; + g = f.result; + for (let h = 0, k, l, m; h < g.length; h++) { + if (l = g[h], k = l.id, m = d[k]) { + m.push(f.field); + } else { + if (c.length === b) { + return c; + } + l.field = d[k] = [f.field]; + c.push(l); + } + } + } + return c; +} +function Ma(a, b, c, d, e) { + let f = this.tag.get(a); + if (!f) { + return console.warn("Tag '" + a + "' was not found"), []; + } + if ((a = (f = f && f.get(b)) && f.length - d) && 0 < a) { + if (a > c || d) { + f = f.slice(d, d + c); + } + e && (f = Na.call(this, f)); + return f; } } -function wa(a) { +function Na(a) { const b = Array(a.length); for (let c = 0, d; c < a.length; c++) { - d = a[c], b[c] = {id:d, doc:this.store[d]}; + d = a[c], b[c] = {id:d, doc:this.store.get(d)}; } return b; } -t.contain = function(a) { - return !!this.register[a]; +;function Y(a) { + if (!(this instanceof Y)) { + return new Y(a); + } + const b = a.document || a.doc || a; + var c, d; + this.M = []; + this.field = []; + this.W = []; + this.key = (c = b.key || b.id) && Pa(c, this.W) || "id"; + (d = a.keystore || 0) && (this.keystore = d); + this.A = (this.fastupdate = !!a.fastupdate) ? d ? new L(d) : new Map() : d ? new M(d) : new Set(); + this.H = (c = b.store || null) && !0 !== c && []; + this.store = c && (d ? new L(d) : new Map()); + this.cache = (c = a.cache || null) && new F(c); + a.cache = !1; + this.worker = a.worker; + this.async = !1; + c = new Map(); + d = b.index || b.field || b; + B(d) && (d = [d]); + for (let e = 0, f, g; e < d.length; e++) { + f = d[e]; + B(f) || (g = f, f = f.field); + g = C(g) ? Object.assign({}, a, g) : a; + if (this.worker) { + const h = new W(g); + c.set(f, h); + h.worker || (this.worker = !1); + } + this.worker || c.set(f, new P(g, this.A)); + g.S ? this.M[e] = g.S : (this.M[e] = Pa(f, this.W), g.filter && ("string" === typeof this.M[e] && (this.M[e] = new String(this.M[e])), this.M[e].T = g.filter)); + this.field[e] = f; + } + if (this.H) { + d = b.store; + B(d) && (d = [d]); + for (let e = 0, f, g; e < d.length; e++) { + f = d[e], g = f.field || f, f.S ? (this.H[e] = f.S, f.S.ia = g) : (this.H[e] = Pa(g, this.W), f.filter && ("string" === typeof this.H[e] && (this.H[e] = new String(this.H[e])), this.H[e].T = f.filter)); + } + } + this.index = c; + this.tag = null; + if (c = b.tag) { + if ("string" === typeof c && (c = [c]), c.length) { + this.tag = new Map(); + this.L = []; + this.aa = []; + for (let e = 0, f, g; e < c.length; e++) { + f = c[e]; + g = f.field || f; + if (!g) { + throw Error("The tag field from the document descriptor is undefined."); + } + f.S ? this.L[e] = f.S : (this.L[e] = Pa(g, this.W), f.filter && ("string" === typeof this.L[e] && (this.L[e] = new String(this.L[e])), this.L[e].T = f.filter)); + this.aa[e] = g; + this.tag.set(g, new Map()); + } + } + } + a.db && this.mount(a.db); +} +u = Y.prototype; +u.mount = function(a) { + let b = this.field; + if (this.tag) { + for (let e = 0, f; e < this.aa.length; e++) { + f = this.aa[e]; + var c = this.index.get(f); + c || (this.index.set(f, c = new P({}, this.A)), b === this.field && (b = b.slice(0)), b.push(f)); + c.tag = this.tag.get(f); + } + } + c = []; + const d = {db:a.db, type:a.type, fastupdate:a.fastupdate}; + for (let e = 0, f, g; e < b.length; e++) { + d.field = g = b[e]; + f = this.index.get(g); + const h = new a.constructor(a.id, d); + h.id = a.id; + c[e] = h.mount(f); + f.document = !0; + e ? f.ja = !0 : f.store = this.store; + } + this.db = this.async = !0; + return Promise.all(c); }; -t.get = function(a) { - return this.store[a]; +u.commit = async function(a, b) { + const c = []; + for (const d of this.index.values()) { + c.push(d.db.commit(d, a, b)); + } + await Promise.all(c); + this.A.clear(); }; -t.set = function(a, b) { - this.store[a] = b; +function Pa(a, b) { + const c = a.split(":"); + let d = 0; + for (let e = 0; e < c.length; e++) { + a = c[e], "]" === a[a.length - 1] && (a = a.substring(0, a.length - 2)) && (b[d] = !0), a && (c[d++] = a); + } + d < c.length && (c.length = d); + return 1 < d ? c : c[0]; +} +u.append = function(a, b) { + return this.add(a, b, !0); +}; +u.update = function(a, b) { + return this.remove(a).add(a, b); +}; +u.remove = function(a) { + C(a) && (a = ca(a, this.key)); + for (var b of this.index.values()) { + b.remove(a, !0); + } + if (this.A.has(a)) { + if (this.tag && !this.fastupdate) { + for (let c of this.tag.values()) { + for (let d of c) { + b = d[0]; + const e = d[1], f = e.indexOf(a); + -1 < f && (1 < e.length ? e.splice(f, 1) : c.delete(b)); + } + } + } + this.store && this.store.delete(a); + this.A.delete(a); + } + this.cache && this.cache.remove(a); return this; }; -t.searchCache = la; -t.export = function(a, b, c, d, e, f) { - let h; - "undefined" === typeof f && (h = new Promise(g => { - f = g; +u.clear = function() { + for (const a of this.index.values()) { + a.clear(); + } + if (this.tag) { + for (const a of this.tag.values()) { + a.clear(); + } + } + this.store && this.store.clear(); + return this; +}; +u.contain = function(a) { + return this.db ? this.index.get(this.field[0]).db.has(a) : this.A.has(a); +}; +u.cleanup = function() { + for (const a of this.index.values()) { + a.cleanup(); + } + return this; +}; +u.get = function(a) { + return this.db ? this.index.get(this.field[0]).db.enrich(a).then(function(b) { + return b[0] && b[0].doc; + }) : this.store.get(a); +}; +u.set = function(a, b) { + this.store.set(a, b); + return this; +}; +u.searchCache = ma; +u.export = function(a, b, c, d, e, f) { + let g; + "undefined" === typeof f && (g = new Promise(k => { + f = k; })); e || (e = 0); d || (d = 0); - if (d < this.h.length) { - const g = this.h[d], k = this.index[g]; + if (d < this.field.length) { + c = this.field[d]; + var h = this.index[c]; b = this; - setTimeout(function() { - k.export(a, b, e ? g : "", d, e++, f) || (d++, e = 1, b.export(a, b, g, d, e, f)); - }); + h.export(a, b, e ? c : "", d, e++, f) || (d++, b.export(a, b, c, d, 1, f)); } else { - let g, k; switch(e) { case 1: - g = "tag"; - k = this.l; + b = "tag"; + h = this.h; c = null; break; case 2: - g = "store"; - k = this.store; + b = "store"; + h = this.store; c = null; break; default: f(); return; } - oa(a, this, c, g, d, e, k, f); + qa(a, this, c, b, d, e, h, f); } - return h; + return g; }; -t.import = function(a, b) { +u.import = function(a, b) { if (b) { - switch(C(b) && (b = JSON.parse(b)), a) { + switch(B(b) && (b = JSON.parse(b)), a) { case "tag": - this.l = b; + this.h = b; break; case "reg": - this.m = !1; - this.register = b; - for (let d = 0, e; d < this.h.length; d++) { - e = this.index[this.h[d]], e.register = b, e.m = !1; + this.fastupdate = !1; + this.A = b; + for (let d = 0, e; d < this.field.length; d++) { + e = this.index[this.field[d]], e.A = b, e.fastupdate = !1; } break; case "store": @@ -918,56 +2019,326 @@ t.import = function(a, b) { } } }; -ia(U.prototype); -var ya = {encode:xa, F:!1, G:""}; -const za = [J("[\u00e0\u00e1\u00e2\u00e3\u00e4\u00e5]"), "a", J("[\u00e8\u00e9\u00ea\u00eb]"), "e", J("[\u00ec\u00ed\u00ee\u00ef]"), "i", J("[\u00f2\u00f3\u00f4\u00f5\u00f6\u0151]"), "o", J("[\u00f9\u00fa\u00fb\u00fc\u0171]"), "u", J("[\u00fd\u0177\u00ff]"), "y", J("\u00f1"), "n", J("[\u00e7c]"), "k", J("\u00df"), "s", J(" & "), " and "]; -function xa(a) { - var b = a = "" + a; - b.normalize && (b = b.normalize("NFD").replace(ca, "")); - return F.call(this, b.toLowerCase(), !a.normalize && za); +ua(Y.prototype); +const Qa = "undefined" !== typeof window && (window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB), Ra = ["map", "ctx", "tag", "reg", "cfg"]; +function Sa(a, b = {}) { + if (!(this instanceof Sa)) { + return new Sa(a, b); + } + "object" === typeof a && (b = a = a.name); + a || console.info("Default storage space was used, because a name was not passed."); + this.id = "flexsearch" + (a ? ":" + a.toLowerCase().replace(/[^a-z0-9_\-]/g, "") : ""); + this.field = b.field ? b.field.toLowerCase().replace(/[^a-z0-9_\-]/g, "") : ""; + this.la = !1; + this.db = null; + this.h = {}; } -;var Ba = {encode:Aa, F:!1, G:"strict"}; -const Ca = /[^a-z0-9]+/, Da = {b:"p", v:"f", w:"f", z:"s", x:"s", "\u00df":"s", d:"t", n:"m", c:"k", g:"k", j:"k", q:"k", i:"e", y:"e", u:"o"}; -function Aa(a) { - a = xa.call(this, a).join(" "); - const b = []; - if (a) { - const c = a.split(Ca), d = c.length; - for (let e = 0, f, h = 0; e < d; e++) { - if ((a = c[e]) && (!this.filter || !this.filter[a])) { - f = a[0]; - let g = Da[f] || f, k = g; - for (let m = 1; m < a.length; m++) { - f = a[m]; - const n = Da[f] || f; - n && n !== k && (g += n, k = n); +u = Sa.prototype; +u.mount = function(a) { + if (a instanceof Y) { + return a.mount(this); + } + a.db = this; + return Ta(this); +}; +function Ta(a) { + navigator.storage && navigator.storage.persist(); + return a.db || new Promise(function(b, c) { + const d = Qa.open(a.id + (a.field ? ":" + a.field : ""), 1); + d.onupgradeneeded = function() { + const e = a.db = this.result; + Ra.forEach(f => { + e.objectStoreNames.contains(f) || e.createObjectStore(f); + }); + }; + d.onblocked = function(e) { + console.error("blocked", e); + c(); + }; + d.onerror = function(e) { + console.error(this.error, e); + c(); + }; + d.onsuccess = function() { + a.db = this.result; + a.db.onversionchange = function() { + a.close(); + }; + b(a); + }; + }); +} +u.close = function() { + this.db.close(); + this.db = null; +}; +u.clear = function() { + const a = this.db.transaction(Ra, "readwrite"); + for (let b = 0; b < Ra.length; b++) { + a.objectStore(Ra[b]).clear(); + } + return Z(a); +}; +u.get = function(a, b, c = 0, d = 0, e = !0, f = !1) { + a = this.db.transaction(b ? "ctx" : "map", "readonly").objectStore(b ? "ctx" : "map").get(b ? b + ":" + a : a); + const g = this; + return Z(a).then(function(h) { + let k = []; + if (!h || !h.length) { + return k; + } + if (e) { + if (!c && !d && 1 === h.length) { + return h[0]; + } + for (let l = 0, m; l < h.length; l++) { + if ((m = h[l]) && m.length) { + if (d >= m.length) { + d -= m.length; + continue; + } + const n = c ? d + Math.min(m.length - d, c) : m.length; + for (let q = d; q < n; q++) { + k.push(m[q]); + } + d = 0; + if (k.length === c) { + break; + } } - b[h++] = g; + } + return f ? g.enrich(k) : k; + } + return h; + }); +}; +u.tag = function(a, b = 0, c = 0, d = !1) { + a = this.db.transaction("tag", "readonly").objectStore("tag").get(a); + const e = this; + return Z(a).then(function(f) { + if (!f || !f.length || c >= f.length) { + return []; + } + if (!b && !c) { + return f; + } + f = f.slice(c, c + b); + return d ? e.enrich(f) : f; + }); +}; +u.enrich = function(a) { + "object" !== typeof a && (a = [a]); + const b = this.db.transaction("reg", "readonly").objectStore("reg"), c = []; + for (let d = 0; d < a.length; d++) { + c[d] = Z(b.get(a[d])); + } + return Promise.all(c).then(function(d) { + for (let e = 0; e < d.length; e++) { + d[e] = {id:a[e], doc:d[e] ? JSON.parse(d[e]) : null}; + } + return d; + }); +}; +u.has = function(a) { + a = this.db.transaction("reg", "readonly").objectStore("reg").getKey(a); + return Z(a); +}; +u.search = null; +u.info = function() { +}; +u.transaction = function(a, b, c) { + let d = this.h[a + ":" + b]; + if (d) { + return c.call(this, d); + } + let e = this.db.transaction(a, b); + this.h[a + ":" + b] = d = e.objectStore(a); + return new Promise((f, g) => { + e.onerror = h => { + this.h[a + ":" + b] = null; + e.abort(); + e = d = null; + g(h); + }; + e.oncomplete = h => { + e = d = this.h[a + ":" + b] = null; + f(h || !0); + }; + return c.call(this, d); + }); +}; +u.commit = async function(a, b, c) { + if (b) { + await this.clear(), a.R = []; + } else { + let d = a.R; + a.R = []; + for (let e = 0, f; e < d.length; e++) { + if (f = d[e], f.clear) { + await this.clear(); + b = !0; + break; + } else { + d[e] = f.ma; } } + b || (c || (d = d.concat(ba(a.A))), d.length && await this.remove(d)); } - return b; + a.A.size && (await this.transaction("map", "readwrite", function(d) { + for (const e of a.map) { + const f = e[0], g = e[1]; + g.length && (b ? d.put(g, f) : d.get(f).onsuccess = function() { + let h = this.result; + var k; + if (h && h.length) { + const l = Math.max(h.length, g.length); + for (let m = 0, n, q; m < l; m++) { + if ((q = g[m]) && q.length) { + if ((n = h[m]) && n.length) { + for (k = 0; k < q.length; k++) { + n.push(q[k]); + } + } else { + h[m] = q; + } + k = 1; + } + } + } else { + h = g, k = 1; + } + k && d.put(h, f); + }); + } + }), await this.transaction("ctx", "readwrite", function(d) { + for (const e of a.I) { + const f = e[0], g = e[1]; + for (const h of g) { + const k = h[0], l = h[1]; + l.length && (b ? d.put(l, f + ":" + k) : d.get(f + ":" + k).onsuccess = function() { + let m = this.result; + var n; + if (m && m.length) { + const q = Math.max(m.length, l.length); + for (let t = 0, p, r; t < q; t++) { + if ((r = l[t]) && r.length) { + if ((p = m[t]) && p.length) { + for (n = 0; n < r.length; n++) { + p.push(r[n]); + } + } else { + m[t] = r; + } + n = 1; + } + } + } else { + m = l, n = 1; + } + n && d.put(m, f + ":" + k); + }); + } + } + }), a.store ? await this.transaction("reg", "readwrite", function(d) { + for (const e of a.store) { + const f = e[0], g = e[1]; + d.put("object" === typeof g ? JSON.stringify(g) : 1, f); + } + }) : a.ja || await this.transaction("reg", "readwrite", function(d) { + for (const e of a.A.keys()) { + d.put(1, e); + } + }), a.tag && await this.transaction("tag", "readwrite", function(d) { + for (const e of a.tag) { + const f = e[0], g = e[1]; + g.length && (d.get(f).onsuccess = function() { + let h = this.result; + h = h && h.length ? h.concat(g) : g; + d.put(h, f); + }); + } + }), a.map.clear(), a.I.clear(), a.tag && a.tag.clear(), a.store && a.store.clear(), a.document || a.A.clear()); +}; +function Ua(a, b, c) { + const d = a.value; + let e, f, g = 0; + for (let h = 0, k; h < d.length; h++) { + if (k = c ? d : d[h]) { + for (let l = 0, m, n; l < b.length; l++) { + if (n = b[l], m = k.indexOf(f ? parseInt(n, 10) : n), 0 > m && !f && "string" === typeof n && !isNaN(n) && (m = k.indexOf(parseInt(n, 10))) && (f = 1), 0 <= m) { + if (e = 1, 1 < k.length) { + k.splice(m, 1); + } else { + d[h] = []; + break; + } + } + } + g += k.length; + } + if (c) { + break; + } + } + g ? e && a.update(d) : a.delete(); + a.continue(); } -;var Fa = {encode:Ea, F:!1, G:""}; -const Ga = [J("ae"), "a", J("oe"), "o", J("sh"), "s", J("th"), "t", J("ph"), "f", J("pf"), "f", J("(?![aeo])h(?![aeo])"), "", J("(?!^[aeo])h(?!^[aeo])"), ""]; -function Ea(a, b) { - a && (a = Aa.call(this, a).join(" "), 2 < a.length && (a = G(a, Ga)), b || (1 < a.length && (a = da(a)), a && (a = a.split(" ")))); - return a || []; +u.remove = function(a) { + "object" !== typeof a && (a = [a]); + return Promise.all([this.transaction("map", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + const c = this.result; + c && Ua(c, a); + }; + }), this.transaction("ctx", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + const c = this.result; + c && Ua(c, a); + }; + }), this.transaction("tag", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + const c = this.result; + c && Ua(c, a, !0); + }; + }), this.transaction("reg", "readwrite", function(b) { + for (let c = 0; c < a.length; c++) { + b.delete(a[c]); + } + })]); +}; +function Z(a) { + return new Promise((b, c) => { + a.onsuccess = function() { + b(this.result); + }; + a.oncomplete = function() { + b(this.result); + }; + a.onerror = c; + a = null; + }); } -;var Ia = {encode:Ha, F:!1, G:""}; -const Ja = J("(?!\\b)[aeo]"); -function Ha(a) { - a && (a = Ea.call(this, a, !0), 1 < a.length && (a = a.replace(Ja, "")), 1 < a.length && (a = da(a)), a && (a = a.split(" "))); - return a || []; -} -;K["latin:default"] = fa; -K["latin:simple"] = ya; -K["latin:balance"] = Ba; -K["latin:advanced"] = Fa; -K["latin:extra"] = Ia; -export default {Index:N, Document:U, Worker:S, registerCharset:function(a, b) { - K[a] = b; -}, registerLanguage:function(a, b) { - ha[a] = b; +;const Va = new Map([["b", "p"], ["v", "f"], ["w", "f"], ["z", "s"], ["x", "s"], ["d", "t"], ["n", "m"], ["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], ["i", "e"], ["y", "e"], ["u", "o"]]); +var Wa = {normalize:!0, C:!0, D:Va}; +const Ya = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]), Za = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"]; +var $a = {normalize:!0, C:!0, D:Va, K:Za, J:Ya}; +var ab = {normalize:!0, C:!0, D:Va, K:Za.concat([/(?!^)[aeoy]/g, ""]), J:Ya}; +const bb = {a:"", e:"", i:"", o:"", u:"", y:"", b:1, f:1, p:1, v:1, c:2, g:2, j:2, k:2, q:2, s:2, x:2, z:2, "\u00df":2, d:3, t:3, l:4, m:5, n:5, r:6}; +N["latin:exact"] = {normalize:!1, C:!1}; +N["latin:default"] = sa; +N["latin:simple"] = {normalize:!0, C:!0}; +N["latin:balance"] = Wa; +N["latin:advanced"] = $a; +N["latin:extra"] = ab; +N["latin:soundex"] = {normalize:!0, C:!1, ga:{ka:!0}, V:function(a) { + for (let c = 0; c < a.length; c++) { + var b = a[c]; + let d = b.charAt(0), e = bb[d]; + for (let f = 1, g; f < b.length && (g = b.charAt(f), "h" === g || "w" === g || !(g = bb[g]) || g === e || (d += g, e = g, 4 !== d.length)); f++) { + } + a[c] = d; + } }}; +export default {Index:P, Encoder:E, Charset:N, Language:ra, Document:Y, Worker:W, Resolver:T, IndexedDB:Sa}; +export const Index=P;export const Encoder=E;export const Charset=N;export const Language=ra;export const Document=Y;export const Worker=W;export const Resolver=T;export const IndexedDB=Sa; \ No newline at end of file diff --git a/dist/flexsearch.bundle.module.min.js b/dist/flexsearch.bundle.module.min.js index f5b31a2..9b04762 100644 --- a/dist/flexsearch.bundle.module.min.js +++ b/dist/flexsearch.bundle.module.min.js @@ -1,34 +1,89 @@ /**! - * FlexSearch.js v0.7.41 (Bundle.module) + * FlexSearch.js v0.8.0 (Bundle/Module) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -var t;function u(a){return"undefined"!==typeof a?a:!0}function v(a){const b=Array(a);for(let c=0;c=this.B&&(w||!n[l])){var f=O(q,d,r),h="";switch(this.G){case "full":if(2f;g--)if(g-f>=this.B){var k=O(q,d,r,e,f);h=l.substring(f,g);P(this,n,h,k,a,c)}break}case "reverse":if(1=this.B&&P(this,n, -h,O(q,d,r,e,g),a,c);h=""}case "forward":if(1=this.B&&P(this,n,h,f,a,c);break}default:if(this.C&&(f=Math.min(f/this.C(b,l,r)|0,q-1)),P(this,n,l,f,a,c),w&&1=this.B&&!e[l]){e[l]=1;const p=this.l&&l>f;P(this,m,p?f:l,O(h+(d/2>h?0:1),d,r,g-1,k-1),a,c,p?l:f)}}}}this.m||(this.register[a]=1)}}return this}; -function O(a,b,c,d,e){return c&&1=this.B&&!c[q])if(this.s||f||this.map[q])k[w++]=q,c[q]=1;else return d;a=k;e=a.length}if(!e)return d;b||(b=100);g=this.depth&&1=d)))break;if(n){if(f)return qa(k,d,0);b[b.length]=k;return}}return!c&&k}function qa(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function ra(a,b,c,d){c?(d=d&&b>c,a=(a=a[d?b:c])&&a[d?c:b]):a=a[b];return a}t.contain=function(a){return!!this.register[a]};t.update=function(a,b){return this.remove(a).add(a,b)}; -t.remove=function(a,b){const c=this.register[a];if(c){if(this.m)for(let d=0,e;d{f=m}));let g,k;switch(e||(e=0)){case 0:g="reg";if(this.m){k=x();for(let m in this.register)k[m]=1}else k=this.register;break;case 1:g="cfg";k={doc:0,opt:this.s?1:0};break;case 2:g="map";k=this.map;break;case 3:g="ctx";k=this.h;break;default:"undefined"===typeof c&&f&&f();return}oa(a,b||this,c,g,d,e,k,f);return h}; -t.import=function(a,b){if(b)switch(C(b)&&(b=JSON.parse(b)),a){case "cfg":this.s=!!b.opt;break;case "reg":this.m=!1;this.register=b;break;case "map":this.map=b;break;case "ctx":this.h=b}};ia(N.prototype);function sa(a){a=a.data;var b=self._index;const c=a.args;var d=a.task;switch(d){case "init":d=a.options||{};a=a.factory;b=d.encode;d.cache=!1;b&&0===b.indexOf("function")&&(d.encode=Function("return "+b)());a?(Function("return "+a)()(self),self._index=new self.FlexSearch.Index(d),delete self.FlexSearch):self._index=new N(d);break;default:a=a.id,b=b[d].apply(b,c),postMessage("search"===d?{id:a,msg:b}:{id:a})}};let ta=0;function S(a){if(!(this instanceof S))return new S(a);var b;a?E(b=a.encode)&&(a.encode=b.toString()):a={};(b=(self||window)._factory)&&(b=b.toString());const c="undefined"===typeof window&&self.exports,d=this;this.o=ua(b,c,a.worker);this.h=x();if(this.o){if(c)this.o.on("message",function(e){d.h[e.id](e.msg);delete d.h[e.id]});else this.o.onmessage=function(e){e=e.data;d.h[e.id](e.msg);delete d.h[e.id]};this.o.postMessage({task:"init",factory:b,options:a})}}T("add");T("append");T("search"); -T("update");T("remove");function T(a){S.prototype[a]=S.prototype[a+"Async"]=function(){const b=this,c=[].slice.call(arguments);var d=c[c.length-1];let e;E(d)&&(e=d,c.splice(c.length-1,1));d=new Promise(function(f){setTimeout(function(){b.h[++ta]=f;b.o.postMessage({task:a,id:ta,args:c})})});return e?(d.then(e),this):d}} -function ua(a,b,c){let d;try{d=b?new (require("worker_threads")["Worker"])(__dirname + "/node/node.js"):a?new Worker(URL.createObjectURL(new Blob(["onmessage="+sa.toString()],{type:"text/javascript"}))):new Worker(C(c)?c:"worker/worker.js",{type:"module"})}catch(e){}return d};function U(a){if(!(this instanceof U))return new U(a);var b=a.document||a.doc||a,c;this.K=[];this.h=[];this.A=[];this.register=x();this.key=(c=b.key||b.id)&&V(c,this.A)||"id";this.m=u(a.fastupdate);this.C=(c=b.store)&&!0!==c&&[];this.store=c&&x();this.I=(c=b.tag)&&V(c,this.A);this.l=c&&x();this.cache=(c=a.cache)&&new M(c);a.cache=!1;this.o=a.worker;this.async=!1;c=x();let d=b.index||b.field||b;C(d)&&(d=[d]);for(let e=0,f,h;eb||c)e=e.slice(c,c+b);d&&(e=wa.call(this,e));return{tag:a,result:e}}}function wa(a){const b=Array(a.length);for(let c=0,d;c{f=g}));e||(e=0);d||(d=0);if(dthis.N.get(l)),k=1);this.J&&1this.J.get(l)),k=1);g&&k&&(g.lengththis.ea&&(this.U.clear(), +this.B=this.B/1.1|0));g&&c.push(g)}this.V&&(c=this.V(c)||c);this.cache&&a.length<=this.h&&(this.P.set(a,c),this.P.size>this.ea&&(this.P.clear(),this.h=this.h/1.1|0));return c};function la(a){a.X=null;a.P.clear();a.U.clear()};function ma(a,b,c){a=("object"===typeof a?""+a.query:a).toLowerCase();let d=this.cache.get(a);if(!d){d=this.search(a,b,c);if(d instanceof Promise){const e=this;d.then(function(f){e.cache.set(a,f)})}this.cache.set(a,d)}return d}function F(a){this.limit=a&&!0!==a?a:1E3;this.cache=new Map;this.h=""}F.prototype.set=function(a,b){this.cache.has(a)||(this.cache.set(this.h=a,b),this.limit&&this.cache.size>this.limit&&this.cache.delete(this.cache.keys().next().value))}; +F.prototype.get=function(a){const b=this.cache.get(a);b&&this.limit&&this.h!==a&&(this.cache.delete(a),this.cache.set(this.h=a,b));return b};F.prototype.remove=function(a){for(const b of this.cache){const c=b[0];b[1].includes(a)&&this.cache.delete(c)}};F.prototype.clear=function(){this.cache.clear();this.h=""};function na(a,b,c,d){let e=[];for(let f=0,g;f=g.length)b-=g.length;else{b=g[d?"splice":"slice"](b,c);const h=b.length;if(h&&(e=e.length?e.concat(b):b,c-=h,d&&(a.length-=h),!c))break;b=0}return e} +function H(a){if(!(this instanceof H))return new H(a);this.index=a?[a]:[];this.length=a?a.length:0;const b=this;return new Proxy([],{get(c,d){if("length"===d)return b.length;if("push"===d)return function(e){b.index[b.index.length-1].push(e);b.length++};if("pop"===d)return function(){if(b.length)return b.length--,b.index[b.index.length-1].pop()};if("indexOf"===d)return function(e){let f=0;for(let g=0,h,k;gf;h--){g=p.substring(f,h);var k=this.score?this.score(b,p,t,g,f):Q(q,d,t,e,f);R(this,m,g,k,a,c)}break}case "reverse":if(1< +e){for(h=e-1;0g?0:1),d,t,h-1,k-1),x=this.bidirectional&&p>f;R(this,l,x?f:p,r,a,c,x?p:f)}}}}this.fastupdate||this.A.add(a)}else b=""}this.db&& +(b||this.R.push({del:a}),this.fa&&va(this));return this};function R(a,b,c,d,e,f,g){let h=g?a.I:a.map,k;if(!b[c]||!g||!(k=b[c])[g])if(g?(b=k||(b[c]=z()),b[g]=1,(k=h.get(g))?h=k:h.set(g,h=new Map)):b[c]=1,(k=h.get(c))?h=k:h.set(c,h=k=[]),h=h[d]||(h[d]=[]),!f||!h.includes(e)){if(h.length===2**31-1){b=new H(h);if(a.fastupdate)for(let l of a.A.values())l.includes(h)&&(l[l.indexOf(h)]=b);k[d]=h=b}h.push(e);a.fastupdate&&((d=a.A.get(e))?d.push(h):a.A.set(e,[h]))}} +function Q(a,b,c,d,e){return c&&1b?b?a.slice(c,c+b):a.slice(c):a,d?wa(a):a;let e=[];for(let f=0,g,h;f=h){c-=h;continue}cb&&(g=g.slice(0,b),h=g.length),e.push(g);else{if(h>=b)return h>b&&(g=g.slice(0,b)),d?wa(g):g;e=[g]}b-=h;if(!b)break}if(!e.length)return e;e=1a.length)return e?S(a[0],b,c,d):a[0];d=[];let g=0,h=z(),k=da(a);for(let l=0,m;la.length)return[];let f=[],g=0,h=z(),k=da(a);if(!k)return f;for(let l=0,m;la.length)return e?S(a[0],b,c,d):a[0];b=[];c=z();for(let g=0,h;g=e)))break;if(h.length){if(g)return S(h,e,0);b.push(h);return}}return!c&&h}function V(a,b,c,d,e,f,g,h){let k;c&&(k=a.bidirectional&&b>c);if(a.db)return c?a.db.get(k?c:b,k?b:c,d,e,f,g,h):a.db.get(b,"",d,e,f,g,h);a=c?(a=a.I.get(k?b:c))&&a.get(k?c:b):a.map.get(b);return a};P.prototype.remove=function(a,b){const c=this.A.size&&(this.fastupdate?this.A.get(a):this.A.has(a));if(c){if(this.fastupdate)for(let d=0,e;de.length)e.pop();else{const f=e.indexOf(a);f===c.length-1?e.pop():e.splice(f,1)}}else Fa(this.map,a),this.depth&&Fa(this.I,a);b||this.A.delete(a)}this.db&&(this.R.push({del:a}),this.fa&&va(this));this.cache&&this.cache.remove(a);return this}; +function Fa(a,b){let c=0;if(a.constructor===Array)for(let d=0,e,f;dc.add(a,b)):this.add(a,b)}return this.remove(a).add(a,b)};function Ga(a){let b=0;if(a.constructor===Array)for(let c=0,d;c{f=l}));let h,k;switch(e||(e=0)){case 0:h="reg";if(this.fastupdate){k=z();for(let l of this.A.keys())k[l]=1}else k=this.A;break;case 1:h="cfg";k={doc:0,opt:this.B?1:0};break;case 2:h="map";k=this.map;break;case 3:h="ctx";k=this.I;break;default:"undefined"===typeof c&&f&&f();return}qa(a,b||this,c,h,d,e,k,f);return g}; +u.import=function(a,b){if(b)switch(B(b)&&(b=JSON.parse(b)),a){case "cfg":this.B=!!b.opt;break;case "reg":this.fastupdate=!1;this.A=b;break;case "map":this.map=b;break;case "ctx":this.I=b}};ua(P.prototype);async function Ha(a){a=a.data;var b=self._index;const c=a.args;var d=a.task;switch(d){case "init":d=a.options||{};(b=d.config)&&(d=await import(b));(b=a.factory)?(Function("return "+b)()(self),self._index=new self.FlexSearch.Index(d),delete self.FlexSearch):self._index=new P(d);postMessage({id:a.id});break;default:a=a.id,b=b[d].apply(b,c),postMessage("search"===d?{id:a,msg:b}:{id:a})}};let Ia=0; +function W(a){function b(f){f=f.data||f;const g=f.id,h=g&&e.h[g];h&&(h(f.msg),delete e.h[g])}if(!(this instanceof W))return new W(a);a||(a={});let c=(self||window)._factory;c&&(c=c.toString());const d="undefined"===typeof window&&self.exports,e=this;this.worker=Ja(c,d,a.worker);this.h=z();if(this.worker){d?this.worker.on("message",b):this.worker.onmessage=b;if(a.config)return new Promise(function(f){e.h[++Ia]=function(){f(e)};e.worker.postMessage({id:Ia,task:"init",factory:c,options:a})});this.worker.postMessage({task:"init", +factory:c,options:a})}}X("add");X("append");X("search");X("update");X("remove");function X(a){W.prototype[a]=W.prototype[a+"Async"]=function(){const b=this,c=[].slice.call(arguments);var d=c[c.length-1];let e;"function"===typeof d&&(e=d,c.splice(c.length-1,1));d=new Promise(function(f){b.h[++Ia]=f;b.worker.postMessage({task:a,id:Ia,args:c})});return e?(d.then(e),this):d}} +function Ja(a,b,c){return b?new (require("worker_threads")["Worker"])(__dirname + "/node/node.js"):a?new window.Worker(URL.createObjectURL(new Blob(["onmessage="+Ha.toString()],{type:"text/javascript"}))):new window.Worker(B(c)?c:"worker/worker.js",{type:"module"})};Y.prototype.add=function(a,b,c){C(a)&&(b=a,a=ca(b,this.key));if(b&&(a||0===a)){if(!c&&this.A.has(a))return this.update(a,b);for(let h=0,k;hc||d)a=a.slice(d,d+c);e&&(a=Na.call(this,a));return a}} +function Na(a){const b=Array(a.length);for(let c=0,d;c{f=k}));e||(e=0);d||(d=0);if(d{e.objectStoreNames.contains(f)||e.createObjectStore(f)})};d.onblocked=function(e){console.error("blocked",e);c()};d.onerror=function(e){console.error(this.error,e);c()};d.onsuccess=function(){a.db=this.result;a.db.onversionchange=function(){a.close()};b(a)}})} +u.close=function(){this.db.close();this.db=null};u.clear=function(){const a=this.db.transaction(Ra,"readwrite");for(let b=0;b=m.length){d-=m.length;continue}const n=c?d+Math.min(m.length-d,c):m.length;for(let q=d;q=f.length)return[];if(!b&&!c)return f;f=f.slice(c,c+b);return d?e.enrich(f):f})}; +u.enrich=function(a){"object"!==typeof a&&(a=[a]);const b=this.db.transaction("reg","readonly").objectStore("reg"),c=[];for(let d=0;d{e.onerror=h=>{this.h[a+":"+b]=null;e.abort();e=d=null;g(h)};e.oncomplete=h=>{e=d=this.h[a+":"+b]=null;f(h||!0)};return c.call(this,d)})}; +u.commit=async function(a,b,c){if(b)await this.clear(),a.R=[];else{let d=a.R;a.R=[];for(let e=0,f;em&&!f&&"string"===typeof n&&!isNaN(n)&&(m=k.indexOf(parseInt(n,10)))&&(f=1),0<=m)if(e=1,1{a.onsuccess=function(){b(this.result)};a.oncomplete=function(){b(this.result)};a.onerror=c;a=null})};const Va=new Map([["b","p"],["v","f"],["w","f"],["z","s"],["x","s"],["d","t"],["n","m"],["c","k"],["g","k"],["j","k"],["q","k"],["i","e"],["y","e"],["u","o"]]);var Wa={normalize:!0,C:!0,D:Va};const Ya=new Map([["ai","ei"],["ae","a"],["oe","o"],["ue","u"],["sh","s"],["ch","c"],["th","t"],["ph","f"],["pf","f"]]),Za=[/([^aeo])h([aeo$])/g,"$1$2",/([aeo])h([^aeo]|$)/g,"$1$2"];var $a={normalize:!0,C:!0,D:Va,K:Za,J:Ya};var ab={normalize:!0,C:!0,D:Va,K:Za.concat([/(?!^)[aeoy]/g,""]),J:Ya};const bb={a:"",e:"",i:"",o:"",u:"",y:"",b:1,f:1,p:1,v:1,c:2,g:2,j:2,k:2,q:2,s:2,x:2,z:2,"\u00df":2,d:3,t:3,l:4,m:5,n:5,r:6};M["latin:exact"]={normalize:!1,C:!1};M["latin:default"]=sa;M["latin:simple"]={normalize:!0,C:!0};M["latin:balance"]=Wa;M["latin:advanced"]=$a;M["latin:extra"]=ab;M["latin:soundex"]={normalize:!0,C:!1,ga:{ka:!0},V:function(a){for(let c=0;c this.L.get(m)), k = 1); + this.H && 1 < g.length && (this.Y || (this.Y = new RegExp("(" + this.F + ")", "g")), g = g.replace(this.Y, m => this.H.get(m)), k = 1); + g && k && (g.length < this.W || this.filter && this.filter.has(g)) && (g = ""); + if (g && (this.D || this.C && 1 < g.length)) { + d = ""; + for (let m = 0, l = "", r, t; m < g.length; m++) { + r = g.charAt(m), r === l && this.C || ((t = this.D && this.D.get(r)) || "" === t ? t === l && this.C || !(l = t) || (d += t) : d += l = r); + } + g = d; + } + if (g && this.I) { + for (d = 0; g && d < this.I.length; d += 2) { + g = g.replace(this.I[d], this.I[d + 1]); + } + } + this.cache && h.length <= this.B && (this.R.set(h, g), this.R.size > this.$ && (this.R.clear(), this.B = this.B / 1.1 | 0)); + g && b.push(g); + } + this.S && (b = this.S(b) || b); + this.cache && a.length <= this.h && (this.N.set(a, b), this.N.size > this.$ && (this.N.clear(), this.h = this.h / 1.1 | 0)); + return b; +}; +function ha(a) { + a.U = null; + a.N.clear(); + a.R.clear(); +} +;function ia(a, c, b) { + a = ("object" === typeof a ? "" + a.query : a).toLowerCase(); + let e = this.cache.get(a); + if (!e) { + e = this.search(a, c, b); + if (e instanceof Promise) { + const d = this; + e.then(function(f) { + d.cache.set(a, f); + }); + } + this.cache.set(a, e); + } + return e; } function I(a) { - return new RegExp(a, "g"); + this.limit = a && !0 !== a ? a : 1000; + this.cache = new Map(); + this.h = ""; } -function J(a) { - let b = "", c = ""; - for (let e = 0, d = a.length, f; e < d; e++) { - (f = a[e]) !== c && (b += c = f); +I.prototype.set = function(a, c) { + this.cache.has(a) || (this.cache.set(this.h = a, c), this.limit && this.cache.size > this.limit && this.cache.delete(this.cache.keys().next().value)); +}; +I.prototype.get = function(a) { + const c = this.cache.get(a); + c && this.limit && this.h !== a && (this.cache.delete(a), this.cache.set(this.h = a, c)); + return c; +}; +I.prototype.remove = function(a) { + for (const c of this.cache) { + const b = c[0]; + c[1].includes(a) && this.cache.delete(b); + } +}; +I.prototype.clear = function() { + this.cache.clear(); + this.h = ""; +}; +function J(a = 8) { + if (!(this instanceof J)) { + return new J(a); + } + this.index = y(); + this.F = []; + this.size = 0; + 32 < a ? (this.h = ja, this.B = BigInt(a)) : (this.h = ka, this.B = a); +} +J.prototype.get = function(a) { + const c = this.index[this.h(a)]; + return c && c.get(a); +}; +J.prototype.set = function(a, c) { + var b = this.h(a); + let e = this.index[b]; + e ? (b = e.size, e.set(a, c), (b -= e.size) && this.size++) : (this.index[b] = e = new Map([[a, c]]), this.F.push(e)); +}; +function M(a = 8) { + if (!(this instanceof M)) { + return new M(a); + } + this.index = y(); + this.h = []; + 32 < a ? (this.F = ja, this.B = BigInt(a)) : (this.F = ka, this.B = a); +} +M.prototype.add = function(a) { + var c = this.F(a); + let b = this.index[c]; + b ? (c = b.size, b.add(a), (c -= b.size) && this.size++) : (this.index[c] = b = new Set([a]), this.h.push(b)); +}; +u = J.prototype; +u.has = M.prototype.has = function(a) { + const c = this.index[this.F(a)]; + return c && c.has(a); +}; +u.delete = M.prototype.delete = function(a) { + const c = this.index[this.F(a)]; + c && c.delete(a) && this.size--; +}; +u.clear = M.prototype.clear = function() { + this.index = y(); + this.h = []; + this.size = 0; +}; +u.values = M.prototype.values = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let c of this.h[a].values()) { + yield c; + } + } +}; +u.keys = M.prototype.keys = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let c of this.h[a].keys()) { + yield c; + } + } +}; +u.entries = M.prototype.entries = function*() { + for (let a = 0; a < this.h.length; a++) { + for (let c of this.h[a].entries()) { + yield c; + } + } +}; +function ka(a) { + let c = 2 ** this.B - 1; + if ("number" == typeof a) { + return a & c; + } + let b = 0, e = this.B + 1; + for (let d = 0; d < a.length; d++) { + b = (b * e ^ a.charCodeAt(d)) & c; + } + return 32 === this.B ? b + 2 ** 31 : b; +} +function ja(a) { + let c = BigInt(2) ** this.B - BigInt(1); + var b = typeof a; + if ("bigint" === b) { + return a & c; + } + if ("number" === b) { + return BigInt(a) & c; + } + b = BigInt(0); + let e = this.B + BigInt(1); + for (let d = 0; d < a.length; d++) { + b = (b * e ^ BigInt(a.charCodeAt(d))) & c; } return b; } -;var da = {encode:K, B:!1, C:""}; -function K(a) { - return E.call(this, ("" + a).toLowerCase(), !1); +;function la(a, c, b, e, d, f, g, h) { + (e = a(b ? b + "." + e : e, JSON.stringify(g))) && e.then ? e.then(function() { + c.export(a, c, b, d, f + 1, h); + }) : c.export(a, c, b, d, f + 1, h); } -;const L = {}, M = {}; -function ea(a) { - O(a, "add"); - O(a, "append"); - O(a, "search"); - O(a, "update"); - O(a, "remove"); +;const ma = y(), N = y(); +var na = {normalize:function(a) { + return a.toLowerCase(); +}, C:!1}; +const oa = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +function pa(a) { + O.call(a, "add"); + O.call(a, "append"); + O.call(a, "search"); + O.call(a, "update"); + O.call(a, "remove"); } -function O(a, b) { - a[b + "Async"] = function() { - const c = this, e = arguments; - var d = e[e.length - 1]; - let f; - "function" === typeof d && (f = d, delete e[e.length - 1]); - d = new Promise(function(g) { - setTimeout(function() { - c.async = !0; - const h = c[b].apply(c, e); - c.async = !1; - g(h); - }); - }); - return f ? (d.then(f), this) : d; +function O(a) { + this[a + "Async"] = function() { + var c = arguments; + const b = c[c.length - 1]; + let e; + "function" === typeof b && (e = b, delete c[c.length - 1]); + this.async = !0; + c = this[a].apply(this, c); + this.async = !1; + c.then ? c.then(e) : e(c); + return c; }; } -;function fa(a, b, c, e) { - const d = a.length; - let f = [], g, h, k = 0; - e && (e = []); - for (let m = d - 1; 0 <= m; m--) { - const n = a[m], u = n.length, q = z(); - let r = !g; - for (let l = 0; l < u; l++) { - const p = n[l], A = p.length; - if (A) { - for (let B = 0, y, x; B < A; B++) { - if (x = p[B], g) { - if (g[x]) { - if (!m) { - if (c) { - c--; - } else { - if (f[k++] = x, k === b) { - return f; - } - } - } - if (m || e) { - q[x] = 1; - } - r = !0; - } - if (e && (y = (h[x] || 0) + 1, h[x] = y, y < d)) { - const G = e[y - 2] || (e[y - 2] = []); - G[G.length] = x; - } - } else { - q[x] = 1; - } - } - } +;y(); +P.prototype.add = function(a, c, b, e) { + if (c && (a || 0 === a)) { + if (!e && !b && this.A.has(a)) { + return this.update(a, c); } - if (e) { - g || (h = q); - } else if (!r) { - return []; - } - g = q; - } - if (e) { - for (let m = e.length - 1, n, u; 0 <= m; m--) { - n = e[m]; - u = n.length; - for (let q = 0, r; q < u; q++) { - if (r = n[q], !g[r]) { - if (c) { - c--; - } else { - if (f[k++] = r, k === b) { - return f; - } - } - g[r] = 1; - } - } - } - } - return f; -} -function ha(a, b) { - const c = z(), e = z(), d = []; - for (let f = 0; f < a.length; f++) { - c[a[f]] = 1; - } - for (let f = 0, g; f < b.length; f++) { - g = b[f]; - for (let h = 0, k; h < g.length; h++) { - k = g[h], c[k] && !e[k] && (e[k] = 1, d[d.length] = k); - } - } - return d; -} -;const ja = {memory:{charset:"latin:extra", A:3, m:4, D:!1}, performance:{A:3, m:3, s:!1, context:{depth:2, A:1}}, match:{charset:"latin:extra", C:"reverse"}, score:{charset:"latin:advanced", A:20, m:3, context:{depth:3, A:9}}, "default":{}}; -function P(a, b) { - if (!(this instanceof P)) { - return new P(a); - } - var c; - let e; - if (a) { - if (C(a)) { - ja[a] || console.warn("Preset not found: " + a), a = ja[a]; - } else { - if (c = a.preset) { - c[c] || console.warn("Preset not found: " + c), a = Object.assign({}, c[c], a); - } - } - c = a.charset; - e = a.lang; - C(c) && (-1 === c.indexOf(":") && (c += ":default"), c = M[c]); - C(e) && (e = L[e]); - } else { - a = {}; - } - let d, f, g = a.context || {}; - this.encode = a.encode || c && c.encode || K; - this.register = b || z(); - this.A = d = a.resolution || 9; - this.C = b = c && c.C || a.tokenize || "strict"; - this.depth = "strict" === b && g.depth; - this.h = v(g.bidirectional); - this.s = f = v(a.optimize); - this.D = v(a.fastupdate); - this.m = a.minlength || 1; - this.F = a.boost; - this.map = f ? w(d) : z(); - this.o = d = g.resolution || 1; - this.l = f ? w(d) : z(); - this.B = c && c.B || a.rtl; - this.G = (b = a.matcher || e && e.G) && H(b, !1); - this.H = (b = a.stemmer || e && e.H) && H(b, !0); - if (a = b = a.filter || e && e.filter) { - a = b; - c = z(); - for (let h = 0, k = a.length; h < k; h++) { - c[a[h]] = 1; - } - a = c; - } - this.filter = a; -} -t = P.prototype; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.add = function(a, b, c, e) { - if (b && (a || 0 === a)) { - if (!e && !c && this.register[a]) { - return this.update(a, b); - } - b = this.encode(b); - if (e = b.length) { - const m = z(), n = z(), u = this.depth, q = this.A; - for (let r = 0; r < e; r++) { - let l = b[this.B ? e - 1 - r : r]; - var d = l.length; - if (l && d >= this.m && (u || !n[l])) { - var f = Q(q, e, r), g = ""; - switch(this.C) { + c = this.encoder.encode(c); + if (e = c.length) { + const m = y(), l = y(), r = this.depth, t = this.resolution; + for (let p = 0; p < e; p++) { + let n = c[this.rtl ? e - 1 - p : p]; + var d = n.length; + if (d && (r || !l[n])) { + var f = this.score ? this.score(c, n, p, null, 0) : Q(t, e, p), g = ""; + switch(this.tokenize) { case "full": if (2 < d) { for (f = 0; f < d; f++) { for (var h = d; h > f; h--) { - if (h - f >= this.m) { - var k = Q(q, e, r, d, f); - g = l.substring(f, h); - S(this, n, g, k, a, c); - } + g = n.substring(f, h); + var k = this.score ? this.score(c, n, p, g, f) : Q(t, e, p, d, f); + R(this, l, g, k, a, b); } } break; @@ -263,240 +389,515 @@ t.add = function(a, b, c, e) { case "reverse": if (1 < d) { for (h = d - 1; 0 < h; h--) { - g = l[h] + g, g.length >= this.m && S(this, n, g, Q(q, e, r, d, h), a, c); + g = n[h] + g, k = this.score ? this.score(c, n, p, g, h) : Q(t, e, p, d, h), R(this, l, g, k, a, b); } g = ""; } case "forward": if (1 < d) { for (h = 0; h < d; h++) { - g += l[h], g.length >= this.m && S(this, n, g, f, a, c); + g += n[h], R(this, l, g, f, a, b); } break; } default: - if (this.F && (f = Math.min(f / this.F(b, l, r) | 0, q - 1)), S(this, n, l, f, a, c), u && 1 < e && r < e - 1) { - for (d = z(), g = this.o, f = l, h = Math.min(u + 1, e - r), d[f] = 1, k = 1; k < h; k++) { - if ((l = b[this.B ? e - 1 - r - k : r + k]) && l.length >= this.m && !d[l]) { - d[l] = 1; - const p = this.h && l > f; - S(this, m, p ? f : l, Q(g + (e / 2 > g ? 0 : 1), e, r, h - 1, k - 1), a, c, p ? l : f); + if (R(this, l, n, f, a, b), r && 1 < e && p < e - 1) { + for (d = y(), g = this.ca, f = n, h = Math.min(r + 1, e - p), d[f] = 1, k = 1; k < h; k++) { + if ((n = c[this.rtl ? e - 1 - p - k : p + k]) && !d[n]) { + d[n] = 1; + const v = this.score ? this.score(c, f, p, n, k) : Q(g + (e / 2 > g ? 0 : 1), e, p, h - 1, k - 1), C = this.bidirectional && n > f; + R(this, m, C ? f : n, v, a, b, C ? n : f); } } } } } } - this.D || (this.register[a] = 1); + this.fastupdate || this.A.add(a); } } return this; }; -function Q(a, b, c, e, d) { - return c && 1 < a ? b + (e || 0) <= a ? c + (d || 0) : (a - 1) / (b + (e || 0)) * (c + (d || 0)) + 1 | 0 : 0; +function R(a, c, b, e, d, f, g) { + let h = g ? a.K : a.map, k; + c[b] && g && (k = c[b])[g] || (g ? (c = k || (c[b] = y()), c[g] = 1, (k = h.get(g)) ? h = k : h.set(g, h = new Map())) : c[b] = 1, (k = h.get(b)) ? h = k : h.set(b, h = []), h = h[e] || (h[e] = []), f && h.includes(d) || (h.push(d), a.fastupdate && ((c = a.A.get(d)) ? c.push(h) : a.A.set(d, [h])))); } -function S(a, b, c, e, d, f, g) { - let h = g ? a.l : a.map; - if (!b[c] || g && !b[c][g]) { - a.s && (h = h[e]), g ? (b = b[c] || (b[c] = z()), b[g] = 1, h = h[g] || (h[g] = z())) : b[c] = 1, h = h[c] || (h[c] = []), a.s || (h = h[e] || (h[e] = [])), f && h.includes(d) || (h[h.length] = d, a.D && (a = a.register[d] || (a.register[d] = []), a[a.length] = h)); - } +function Q(a, c, b, e, d) { + return b && 1 < a ? c + (e || 0) <= a ? b + (d || 0) : (a - 1) / (c + (e || 0)) * (b + (d || 0)) + 1 | 0 : 0; } -t.search = function(a, b, c) { - c || (!b && D(a) ? (c = a, a = c.query) : D(b) && (c = b)); - let e = [], d; - let f, g = 0; - if (c) { - a = c.query || a; - b = c.limit; - g = c.offset || 0; - var h = c.context; - f = c.suggest; +;function S(a, c, b) { + if (1 === a.length) { + return a = a[0], a = b || a.length > c ? c ? a.slice(b, b + c) : a.slice(b) : a; } - if (a && (a = this.encode("" + a), d = a.length, 1 < d)) { - c = z(); - var k = []; - for (let n = 0, u = 0, q; n < d; n++) { - if ((q = a[n]) && q.length >= this.m && !c[q]) { - if (this.s || f || this.map[q]) { - k[u++] = q, c[q] = 1; - } else { - return e; - } - } - } - a = k; - d = a.length; - } - if (!d) { - return e; - } - b || (b = 100); - h = this.depth && 1 < d && !1 !== h; - c = 0; - let m; - h ? (m = a[0], c = 1) : 1 < d && a.sort(aa); - for (let n, u; c < d; c++) { - u = a[c]; - h ? (n = ka(this, e, f, b, g, 2 === d, u, m), f && !1 === n && e.length || (m = u)) : n = ka(this, e, f, b, g, 1 === d, u); - if (n) { - return n; - } - if (f && c === d - 1) { - k = e.length; - if (!k) { - if (h) { - h = 0; - c = -1; + let e = []; + for (let d = 0, f, g; d < a.length; d++) { + if ((f = a[d]) && (g = f.length)) { + if (b) { + if (b >= g) { + b -= g; continue; } - return e; + b < g && (f = c ? f.slice(b, b + c) : f.slice(b), g = f.length, b = 0); } - if (1 === k) { - return la(e[0], b, g); + if (e.length) { + g > c && (f = f.slice(0, c), g = f.length), e.push(f); + } else { + if (g >= c) { + return g > c && (f = f.slice(0, c)), f; + } + e = [f]; + } + c -= g; + if (!c) { + break; } } } - return fa(e, b, g, f); -}; -function ka(a, b, c, e, d, f, g, h) { - let k = [], m = h ? a.l : a.map; - a.s || (m = ma(m, g, h, a.h)); - if (m) { - let n = 0; - const u = Math.min(m.length, h ? a.o : a.A); - for (let q = 0, r = 0, l, p; q < u; q++) { - if (l = m[q]) { - if (a.s && (l = ma(l, g, h, a.h)), d && l && f && (p = l.length, p <= d ? (d -= p, l = null) : (l = l.slice(d), d = 0)), l && (k[n++] = l, f && (r += l.length, r >= e))) { - break; + return e.length ? e = 1 < e.length ? [].concat.apply([], e) : e[0] : e; +} +;function qa(a, c, b, e) { + var d = a.length; + let f = [], g = 0, h, k, m; + e && (e = []); + for (let l = d - 1, r; 0 <= l; l--) { + m = a[l]; + d = y(); + r = !h; + for (let t = 0, p; t < m.length; t++) { + if ((p = m[t]) && p.length) { + for (let n = 0, v; n < p.length; n++) { + if (v = p[n], h) { + if (h[v]) { + if (!l) { + if (b) { + b--; + } else { + if (f[g++] = v, g === c) { + return f; + } + } + } + if (l || e) { + d[v] = 1; + } + r = !0; + } + e && !k[v] && (k[v] = 1, (e[t] || (e[t] = [])).push(v)); + } else { + d[v] = 1; + } } } } - if (n) { - if (f) { - return la(k, e, 0); - } - b[b.length] = k; - return; + if (e) { + h || (k = d); + } else if (!r) { + return []; } + h = d; } - return !c && k; -} -function la(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function ma(a, b, c, e) { - c ? (e = e && b > c, a = (a = a[e ? b : c]) && a[e ? c : b]) : a = a[b]; - return a; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a, b) { - const c = this.register[a]; - if (c) { - if (this.D) { - for (let e = 0, d; e < c.length; e++) { - d = c[e], d.splice(d.indexOf(a), 1); - } - } else { - T(this.map, a, this.A, this.s), this.depth && T(this.l, a, this.o, this.s); - } - b || delete this.register[a]; - } - return this; -}; -function T(a, b, c, e, d) { - let f = 0; - if (a.constructor === Array) { - if (d) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), f++) : f++; - } else { - d = Math.min(a.length, c); - for (let g = 0, h; g < d; g++) { - if (h = a[g]) { - f = T(h, b, c, e, d), e || f || delete a[g]; + if (e) { + for (let l = e.length - 1, r, t; 0 <= l; l--) { + r = e[l]; + t = r.length; + for (let p = 0, n; p < t; p++) { + if (n = r[p], !h[n]) { + if (b) { + b--; + } else { + if (f[g++] = n, g === c) { + return f; + } + } + h[n] = 1; } } } - } else { - for (let g in a) { - (f = T(a[g], b, c, e, d)) || delete a[g]; - } } return f; } -ea(P.prototype); -function U(a) { - if (!(this instanceof U)) { - return new U(a); +function ra(a, c) { + const b = y(), e = y(), d = []; + for (let f = 0; f < a.length; f++) { + b[a[f]] = 1; } - var b = a.document || a.doc || a, c; - this.F = []; - this.h = []; - this.o = []; - this.register = z(); - this.key = (c = b.key || b.id) && V(c, this.o) || "id"; - this.D = v(a.fastupdate); - this.l = (c = b.store) && !0 !== c && []; - this.store = c && z(); - this.async = !1; - c = z(); - let e = b.index || b.field || b; - C(e) && (e = [e]); - for (let d = 0, f, g; d < e.length; d++) { - f = e[d], C(f) || (g = f, f = f.field), g = D(g) ? Object.assign({}, a, g) : a, this.I || (c[f] = new P(g, this.register)), this.F[d] = V(f, this.o), this.h[d] = f; - } - if (this.l) { - for (a = b.store, C(a) && (a = [a]), b = 0; b < a.length; b++) { - this.l[b] = V(a[b], this.o); + for (let f = 0, g; f < c.length; f++) { + g = c[f]; + for (let h = 0, k; h < g.length; h++) { + k = g[h], b[k] && !e[k] && (e[k] = 1, d.push(k)); } } - this.index = c; + return d; } -function V(a, b) { - const c = a.split(":"); - let e = 0; - for (let d = 0; d < c.length; d++) { - a = c[d], 0 <= a.indexOf("[]") && (a = a.substring(0, a.length - 2)) && (b[e] = !0), a && (c[e++] = a); +;P.prototype.search = function(a, c, b) { + b || (!c && B(a) ? (b = a, a = "") : B(c) && (b = c, c = 0)); + let e = []; + let d, f = 0; + if (b) { + a = b.query || a; + c = b.limit || c; + f = b.offset || 0; + var g = b.context; + d = b.suggest; } - e < c.length && (c.length = e); - return 1 < e ? c : c[0]; -} -function na(a, b) { - if (C(b)) { - a = a[b]; - } else { - for (let c = 0; a && c < b.length; c++) { - a = a[b[c]]; + a = this.encoder.encode(a); + b = a.length; + c || (c = 100); + if (1 === b) { + return T.call(this, a[0], "", c, f); + } + g = this.depth && !1 !== g; + if (2 === b && g && !d) { + return T.call(this, a[0], a[1], c, f); + } + let h = 0, k = 0; + if (1 < b) { + var m = y(); + const r = []; + for (let t = 0, p; t < b; t++) { + if ((p = a[t]) && !m[p]) { + if (d || U(this, p)) { + r.push(p), m[p] = 1; + } else { + return e; + } + const n = p.length; + h = Math.max(h, n); + k = k ? Math.min(k, n) : n; + } + } + a = r; + b = a.length; + } + if (!b) { + return e; + } + m = 0; + let l; + if (1 === b) { + return T.call(this, a[0], "", c, f); + } + if (2 === b && g && !d) { + return T.call(this, a[0], a[1], c, f); + } + 1 < b && (g ? (l = a[0], m = 1) : 9 < h && 3 < h / k && a.sort(aa)); + for (let r, t; m < b; m++) { + t = a[m]; + l ? (r = U(this, t, l), r = sa(r, e, d, this.ca, c, f, 2 === b), d && !1 === r && e.length || (l = t)) : (r = U(this, t), r = sa(r, e, d, this.resolution, c, f, 1 === b)); + if (r) { + return r; + } + if (d && m === b - 1) { + g = e.length; + if (!g) { + if (l) { + l = ""; + m = -1; + continue; + } + return e; + } + if (1 === g) { + return S(e[0], c, f); + } } } + return qa(e, c, f, d); +}; +function T(a, c, b, e) { + return (a = U(this, a, c)) && a.length ? S(a, b, e) : []; +} +function sa(a, c, b, e, d, f, g) { + let h = []; + if (a) { + e = Math.min(a.length, e); + for (let k = 0, m = 0, l; k < e; k++) { + if (l = a[k]) { + if (f && l && g && (l.length <= f ? (f -= l.length, l = null) : (l = l.slice(f), f = 0)), l && (h[k] = l, g && (m += l.length, m >= d))) { + break; + } + } + } + if (h.length) { + if (g) { + return S(h, d, 0); + } + c.push(h); + return; + } + } + return !b && h; +} +function U(a, c, b) { + let e; + b && (e = a.bidirectional && c > b); + a = b ? (a = a.K.get(e ? c : b)) && a.get(e ? b : c) : a.map.get(c); return a; } -function W(a, b, c, e, d) { - a = a[d]; - if (e === c.length - 1) { - b[d] = a; - } else if (a) { - if (a.constructor === Array) { - for (b = b[d] = Array(a.length), d = 0; d < a.length; d++) { - W(a, b, c, e, d); +;P.prototype.remove = function(a, c) { + const b = this.A.size && (this.fastupdate ? this.A.get(a) : this.A.has(a)); + if (b) { + if (this.fastupdate) { + for (let e = 0, d; e < b.length; e++) { + if (d = b[e]) { + if (2 > d.length) { + d.pop(); + } else { + const f = d.indexOf(a); + f === b.length - 1 ? d.pop() : d.splice(f, 1); + } + } } } else { - b = b[d] || (b[d] = z()), d = c[++e], W(a, b, c, e, d); + V(this.map, a), this.depth && V(this.K, a); + } + c || this.A.delete(a); + } + this.cache && this.cache.remove(a); + return this; +}; +function V(a, c) { + let b = 0; + if (a.constructor === Array) { + for (let e = 0, d, f; e < a.length; e++) { + if ((d = a[e]) && d.length) { + if (f = d.indexOf(c), 0 <= f) { + 1 < d.length ? (d.splice(f, 1), b++) : delete a[e]; + break; + } else { + b++; + } + } + } + } else { + for (let e of a) { + const d = e[0], f = V(e[1], c); + f ? b += f : a.delete(d); + } + } + return b; +} +;function P(a, c) { + if (!(this instanceof P)) { + return new P(a); + } + if (a) { + var b = z(a) ? a : a.preset; + b && (oa[b] || console.warn("Preset not found: " + b), a = Object.assign({}, oa[b], a)); + } else { + a = {}; + } + b = a.context || {}; + const e = a.encode || a.encoder || na; + this.encoder = e.encode ? e : "object" === typeof e ? new H(e) : {encode:e}; + let d; + this.resolution = a.resolution || 9; + this.tokenize = d = a.tokenize || "strict"; + this.depth = "strict" === d && b.depth || 0; + this.bidirectional = !1 !== b.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + this.map = (d = !1, new Map()); + this.K = d ? new J(d) : new Map(); + this.A = c || (this.fastupdate ? d ? new J(d) : new Map() : d ? new M(d) : new Set()); + this.ca = b.resolution || 1; + this.rtl = e.rtl || a.rtl || !1; + this.cache = (d = a.cache || null) && new I(d); +} +u = P.prototype; +u.clear = function() { + this.map.clear(); + this.K.clear(); + this.A.clear(); + this.cache && this.cache.clear(); + return this; +}; +u.append = function(a, c) { + return this.add(a, c, !0); +}; +u.contain = function(a) { + return this.A.has(a); +}; +u.update = function(a, c) { + if (this.async) { + const b = this, e = this.remove(a); + return e.then ? e.then(() => b.add(a, c)) : this.add(a, c); + } + return this.remove(a).add(a, c); +}; +function W(a) { + let c = 0; + if (a.constructor === Array) { + for (let b = 0, e; b < a.length; b++) { + (e = a[b]) && (c += e.length); + } + } else { + for (const b of a) { + const e = b[0], d = W(b[1]); + d ? c += d : a.delete(e); + } + } + return c; +} +u.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + W(this.map); + this.depth && W(this.K); + return this; +}; +u.searchCache = ia; +u.export = function(a, c, b, e, d, f) { + let g = !0; + "undefined" === typeof f && (g = new Promise(m => { + f = m; + })); + let h, k; + switch(d || (d = 0)) { + case 0: + h = "reg"; + if (this.fastupdate) { + k = y(); + for (let m of this.A.keys()) { + k[m] = 1; + } + } else { + k = this.A; + } + break; + case 1: + h = "cfg"; + k = {doc:0, opt:this.h ? 1 : 0}; + break; + case 2: + h = "map"; + k = this.map; + break; + case 3: + h = "ctx"; + k = this.K; + break; + default: + "undefined" === typeof b && f && f(); + return; + } + la(a, c || this, b, h, e, d, k, f); + return g; +}; +u.import = function(a, c) { + if (c) { + switch(z(c) && (c = JSON.parse(c)), a) { + case "cfg": + this.h = !!c.opt; + break; + case "reg": + this.fastupdate = !1; + this.A = c; + break; + case "map": + this.map = c; + break; + case "ctx": + this.K = c; + } + } +}; +pa(P.prototype); +X.prototype.add = function(a, c, b) { + B(a) && (c = a, a = F(c, this.key)); + if (c && (a || 0 === a)) { + if (!b && this.A.has(a)) { + return this.update(a, c); + } + for (let h = 0, k; h < this.field.length; h++) { + k = this.M[h]; + var e = this.index.get(this.field[h]); + if ("function" === typeof k) { + var d = k(c); + d && e.add(a, d, !1, !0); + } else { + if (d = k.P, !d || d(c)) { + k instanceof String ? k = ["" + k] : z(k) && (k = [k]), ta(c, k, this.T, 0, e, a, k[0], b); + } + } + } + if (this.tag) { + for (e = 0; e < this.J.length; e++) { + var f = this.J[e], g = this.da[e]; + d = this.tag.get(g); + let h = y(); + if ("function" === typeof f) { + if (f = f(c), !f) { + continue; + } + } else { + const k = f.P; + if (k && !k(c)) { + continue; + } + f instanceof String && (f = "" + f); + f = F(c, f); + } + if (d && f) { + z(f) && (f = [f]); + for (let k = 0, m, l; k < f.length; k++) { + m = f[k], h[m] || (h[m] = 1, (g = d.get(m)) ? l = g : d.set(m, l = []), b && l.includes(a) || (l.push(a), this.fastupdate && ((g = this.A.get(a)) ? g.push(l) : this.A.set(a, [l])))); + } + } else { + d || console.warn("Tag '" + g + "' was not found"); + } + } + } + if (this.store && (!b || !this.store.has(a))) { + let h; + if (this.G) { + h = y(); + for (let k = 0, m; k < this.G.length; k++) { + m = this.G[k]; + if ((b = m.P) && !b(c)) { + continue; + } + let l; + if ("function" === typeof m) { + l = m(c); + if (!l) { + continue; + } + m = [m.ea]; + } else if (z(m) || m instanceof String) { + h[m] = c[m]; + continue; + } + ua(c, h, m, 0, m[0], l); + } + } + this.store.set(a, h || c); + } + } + return this; +}; +function ua(a, c, b, e, d, f) { + a = a[d]; + if (e === b.length - 1) { + c[d] = f || a; + } else if (a) { + if (a.constructor === Array) { + for (c = c[d] = Array(a.length), d = 0; d < a.length; d++) { + ua(a, c, b, e, d); + } + } else { + c = c[d] || (c[d] = y()), d = b[++e], ua(a, c, b, e, d); } } } -function X(a, b, c, e, d, f, g, h) { +function ta(a, c, b, e, d, f, g, h) { if (a = a[g]) { - if (e === b.length - 1) { + if (e === c.length - 1) { if (a.constructor === Array) { - if (c[e]) { - for (b = 0; b < a.length; b++) { - d.add(f, a[b], !0, !0); + if (b[e]) { + for (c = 0; c < a.length; c++) { + d.add(f, a[c], !0, !0); } return; } @@ -506,213 +907,394 @@ function X(a, b, c, e, d, f, g, h) { } else { if (a.constructor === Array) { for (g = 0; g < a.length; g++) { - X(a, b, c, e, d, f, g, h); + ta(a, c, b, e, d, f, g, h); } } else { - g = b[++e], X(a, b, c, e, d, f, g, h); + g = c[++e], ta(a, c, b, e, d, f, g, h); } } } } -t = U.prototype; -t.add = function(a, b, c) { - D(a) && (b = a, a = na(b, this.key)); - if (b && (a || 0 === a)) { - if (!c && this.register[a]) { - return this.update(a, b); - } - for (let e = 0, d, f; e < this.h.length; e++) { - f = this.h[e], d = this.F[e], C(d) && (d = [d]), X(b, d, this.o, 0, this.index[f], a, d[0], c); - } - if (this.store && (!c || !this.store[a])) { - let e; - if (this.l) { - e = z(); - for (let d = 0, f; d < this.l.length; d++) { - f = this.l[d], C(f) ? e[f] = b[f] : W(b, e, f, 0, f[0]); - } - } - this.store[a] = e || b; - } - } - return this; -}; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a) { - D(a) && (a = na(a, this.key)); - if (this.register[a]) { - for (let b = 0; b < this.h.length && (this.index[this.h[b]].remove(a, !this.I), !this.D); b++) { - } - this.store && delete this.store[a]; - delete this.register[a]; - } - return this; -}; -t.search = function(a, b, c, e) { - c || (!b && D(a) ? (c = a, a = "") : D(b) && (c = b, b = 0)); - let d = [], f = [], g, h, k, m, n, u, q = 0; - if (c) { - if (c.constructor === Array) { - k = c, c = null; +;X.prototype.search = function(a, c, b, e) { + b || (!c && B(a) ? (b = a, a = "") : B(c) && (b = c, c = 0)); + let d = []; + var f = []; + let g, h, k, m, l; + let r = 0; + if (b) { + if (b.constructor === Array) { + m = b, b = null; } else { - a = c.query || a; - k = (g = c.pluck) || c.index || c.field; - m = !1; - h = this.store && c.enrich; - n = "and" === c.bool; - b = c.limit || b || 100; - u = c.offset || 0; - if (m && (C(m) && (m = [m]), !a)) { - for (let l = 0, p; l < m.length; l++) { - if (p = oa.call(this, m[l], b, u, h)) { - d[d.length] = p, q++; + a = b.query || a; + var t = b.pluck; + h = b.merge; + m = t || b.field || b.index; + l = this.tag && b.tag; + g = this.store && b.enrich; + k = b.suggest; + c = b.limit || c; + var p = b.offset || 0; + c || (c = 100); + if (l) { + l.constructor !== Array && (l = [l]); + var n = []; + for (let w = 0, q; w < l.length; w++) { + q = l[w]; + if (z(q)) { + throw Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + if (q.field && q.tag) { + var v = q.tag; + if (v.constructor === Array) { + for (var C = 0; C < v.length; C++) { + n.push(q.field, v[C]); + } + } else { + n.push(q.field, v); + } + } else { + v = Object.keys(q); + for (let D = 0, E, A; D < v.length; D++) { + if (E = v[D], A = q[E], A.constructor === Array) { + for (C = 0; C < A.length; C++) { + n.push(E, A[C]); + } + } else { + n.push(E, A); + } + } } } - return q ? d : []; + if (!n.length) { + throw Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + l = n; + if (!a) { + e = []; + if (n.length) { + for (f = 0; f < n.length; f += 2) { + t = va.call(this, n[f], n[f + 1], c, p, g), d.push({field:n[f], tag:n[f + 1], result:t}); + } + } + return e.length ? Promise.all(e).then(function(w) { + for (let q = 0; q < w.length; q++) { + d[q].result = w[q]; + } + return d; + }) : d; + } } - C(k) && (k = [k]); + z(m) && (m = [m]); } } - k || (k = this.h); - n = n && (1 < k.length || m && 1 < m.length); - const r = !e && (this.I || this.async) && []; - for (let l = 0, p, A, B; l < k.length; l++) { - let y; - A = k[l]; - C(A) || (y = A, A = y.field, a = y.query || a, b = y.limit || b, h = y.enrich || h); - if (r) { - r[l] = this.index[A].searchAsync(a, b, y || c); + m || (m = this.field); + p = !e && (this.worker || this.async) && []; + for (let w = 0, q, D, E; w < m.length; w++) { + D = m[w]; + let A; + z(D) || (A = D, D = A.field, a = A.query || a, c = A.limit || c, k = A.suggest || k); + if (e) { + q = e[w]; } else { - e ? p = e[l] : p = this.index[A].search(a, b, y || c); - B = p && p.length; - if (m && B) { - const x = []; - let G = 0; - n && (x[0] = [p]); - for (let R = 0, ia, N; R < m.length; R++) { - if (ia = m[R], B = (N = this.J[ia]) && N.length) { - G++, x[x.length] = n ? [N] : N; + if (n = A || b, v = this.index.get(D), l && (n.enrich = !1), p) { + p[w] = v.searchAsync(a, c, n); + n && g && (n.enrich = g); + continue; + } else { + q = v.search(a, c, n), n && g && (n.enrich = g); + } + } + E = q && q.length; + if (l && E) { + n = []; + v = 0; + for (let K = 0, L, Da; K < l.length; K += 2) { + L = this.tag.get(l[K]); + if (!L) { + if (console.warn("Tag '" + l[K] + ":" + l[K + 1] + "' will be skipped because there is no field '" + l[K] + "'."), k) { + continue; + } else { + return d; } } - G && (p = n ? fa(x, b || 100, u || 0) : ha(p, x), B = p.length); + if (Da = (L = L && L.get(l[K + 1])) && L.length) { + v++, n.push(L); + } else if (!k) { + return d; + } } - if (B) { - f[q] = A, d[q++] = p; - } else if (n) { - return []; + if (v) { + q = ra(q, n); + E = q.length; + if (!E && !k) { + return d; + } + v--; } } + if (E) { + f[r] = D, d.push(q), r++; + } else if (1 === m.length) { + return d; + } } - if (r) { - const l = this; - return new Promise(function(p) { - Promise.all(r).then(function(A) { - p(l.search(a, b, c, A)); - }); + if (p) { + const w = this; + return Promise.all(p).then(function(q) { + return q.length ? w.search(a, c, b, q) : q; }); } - if (!q) { - return []; + if (!r) { + return d; } - if (g && (!h || !this.store)) { + if (t && (!g || !this.store)) { return d[0]; } - for (let l = 0, p; l < f.length; l++) { - p = d[l]; - p.length && h && (p = pa.call(this, p)); - if (g) { - return p; + p = []; + for (let w = 0, q; w < f.length; w++) { + q = d[w]; + g && q.length && !q[0].doc && q.length && (q = wa.call(this, q)); + if (t) { + return q; } - d[l] = {field:f[l], result:p}; + d[w] = {field:f[w], result:q}; } - return d; + return h ? xa(d, c) : d; }; -function oa(a, b, c, e) { - let d = this.J[a], f = d && d.length - c; - if (f && 0 < f) { - if (f > b || c) { - d = d.slice(c, c + b); - } - e && (d = pa.call(this, d)); - return {tag:a, result:d}; - } -} -function pa(a) { - const b = Array(a.length); - for (let c = 0, e; c < a.length; c++) { - e = a[c], b[c] = {id:e, doc:this.store[e]}; - } - return b; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.get = function(a) { - return this.store[a]; -}; -t.set = function(a, b) { - this.store[a] = b; - return this; -}; -ea(U.prototype); -var ra = {encode:qa, B:!1, C:""}; -const sa = [I("[\u00e0\u00e1\u00e2\u00e3\u00e4\u00e5]"), "a", I("[\u00e8\u00e9\u00ea\u00eb]"), "e", I("[\u00ec\u00ed\u00ee\u00ef]"), "i", I("[\u00f2\u00f3\u00f4\u00f5\u00f6\u0151]"), "o", I("[\u00f9\u00fa\u00fb\u00fc\u0171]"), "u", I("[\u00fd\u0177\u00ff]"), "y", I("\u00f1"), "n", I("[\u00e7c]"), "k", I("\u00df"), "s", I(" & "), " and "]; -function qa(a) { - var b = a = "" + a; - b.normalize && (b = b.normalize("NFD").replace(ca, "")); - return E.call(this, b.toLowerCase(), !a.normalize && sa); -} -;var ua = {encode:ta, B:!1, C:"strict"}; -const va = /[^a-z0-9]+/, wa = {b:"p", v:"f", w:"f", z:"s", x:"s", "\u00df":"s", d:"t", n:"m", c:"k", g:"k", j:"k", q:"k", i:"e", y:"e", u:"o"}; -function ta(a) { - a = qa.call(this, a).join(" "); - const b = []; - if (a) { - const c = a.split(va), e = c.length; - for (let d = 0, f, g = 0; d < e; d++) { - if ((a = c[d]) && (!this.filter || !this.filter[a])) { - f = a[0]; - let h = wa[f] || f, k = h; - for (let m = 1; m < a.length; m++) { - f = a[m]; - const n = wa[f] || f; - n && n !== k && (h += n, k = n); +function xa(a, c) { + const b = [], e = y(); + for (let d = 0, f, g; d < a.length; d++) { + f = a[d]; + g = f.result; + for (let h = 0, k, m, l; h < g.length; h++) { + if (m = g[h], k = m.id, l = e[k]) { + l.push(f.field); + } else { + if (b.length === c) { + return b; } - b[g++] = h; + m.field = e[k] = [f.field]; + b.push(m); } } } return b; } -;var ya = {encode:xa, B:!1, C:""}; -const za = [I("ae"), "a", I("oe"), "o", I("sh"), "s", I("th"), "t", I("ph"), "f", I("pf"), "f", I("(?![aeo])h(?![aeo])"), "", I("(?!^[aeo])h(?!^[aeo])"), ""]; -function xa(a, b) { - a && (a = ta.call(this, a).join(" "), 2 < a.length && (a = F(a, za)), b || (1 < a.length && (a = J(a)), a && (a = a.split(" ")))); - return a || []; +function va(a, c, b, e, d) { + let f = this.tag.get(a); + if (!f) { + return console.warn("Tag '" + a + "' was not found"), []; + } + if ((a = (f = f && f.get(c)) && f.length - e) && 0 < a) { + if (a > b || e) { + f = f.slice(e, e + b); + } + d && (f = wa.call(this, f)); + return f; + } } -;var Ba = {encode:Aa, B:!1, C:""}; -const Ca = I("(?!\\b)[aeo]"); -function Aa(a) { - a && (a = xa.call(this, a, !0), 1 < a.length && (a = a.replace(Ca, "")), 1 < a.length && (a = J(a)), a && (a = a.split(" "))); - return a || []; +function wa(a) { + const c = Array(a.length); + for (let b = 0, e; b < a.length; b++) { + e = a[b], c[b] = {id:e, doc:this.store.get(e)}; + } + return c; } -;M["latin:default"] = da; -M["latin:simple"] = ra; -M["latin:balance"] = ua; -M["latin:advanced"] = ya; -M["latin:extra"] = Ba; -const Y = {Index:P, Document:U, Worker:null, registerCharset:function(a, b) { - M[a] = b; -}, registerLanguage:function(a, b) { - L[a] = b; +;function X(a) { + if (!(this instanceof X)) { + return new X(a); + } + const c = a.document || a.doc || a; + var b; + this.M = []; + this.field = []; + this.T = []; + this.key = (b = c.key || c.id) && Y(b, this.T) || "id"; + this.A = (this.fastupdate = !!a.fastupdate) ? new Map() : new Set(); + this.G = (b = c.store || null) && !0 !== b && []; + this.store = b && new Map(); + this.cache = (b = a.cache || null) && new I(b); + this.async = a.cache = !1; + b = new Map(); + let e = c.index || c.field || c; + z(e) && (e = [e]); + for (let d = 0, f, g; d < e.length; d++) { + f = e[d], z(f) || (g = f, f = f.field), g = B(g) ? Object.assign({}, a, g) : a, b.set(f, new P(g, this.A)), g.O ? this.M[d] = g.O : (this.M[d] = Y(f, this.T), g.filter && ("string" === typeof this.M[d] && (this.M[d] = new String(this.M[d])), this.M[d].P = g.filter)), this.field[d] = f; + } + if (this.G) { + a = c.store; + z(a) && (a = [a]); + for (let d = 0, f, g; d < a.length; d++) { + f = a[d], g = f.field || f, f.O ? (this.G[d] = f.O, f.O.ea = g) : (this.G[d] = Y(g, this.T), f.filter && ("string" === typeof this.G[d] && (this.G[d] = new String(this.G[d])), this.G[d].P = f.filter)); + } + } + this.index = b; + this.tag = null; + if (b = c.tag) { + if ("string" === typeof b && (b = [b]), b.length) { + this.tag = new Map(); + this.J = []; + this.da = []; + for (let d = 0, f, g; d < b.length; d++) { + f = b[d]; + g = f.field || f; + if (!g) { + throw Error("The tag field from the document descriptor is undefined."); + } + f.O ? this.J[d] = f.O : (this.J[d] = Y(g, this.T), f.filter && ("string" === typeof this.J[d] && (this.J[d] = new String(this.J[d])), this.J[d].P = f.filter)); + this.da[d] = g; + this.tag.set(g, new Map()); + } + } + } +} +function Y(a, c) { + const b = a.split(":"); + let e = 0; + for (let d = 0; d < b.length; d++) { + a = b[d], "]" === a[a.length - 1] && (a = a.substring(0, a.length - 2)) && (c[e] = !0), a && (b[e++] = a); + } + e < b.length && (b.length = e); + return 1 < e ? b : b[0]; +} +u = X.prototype; +u.append = function(a, c) { + return this.add(a, c, !0); +}; +u.update = function(a, c) { + return this.remove(a).add(a, c); +}; +u.remove = function(a) { + B(a) && (a = F(a, this.key)); + for (var c of this.index.values()) { + c.remove(a, !0); + } + if (this.A.has(a)) { + if (this.tag && !this.fastupdate) { + for (let b of this.tag.values()) { + for (let e of b) { + c = e[0]; + const d = e[1], f = d.indexOf(a); + -1 < f && (1 < d.length ? d.splice(f, 1) : b.delete(c)); + } + } + } + this.store && this.store.delete(a); + this.A.delete(a); + } + this.cache && this.cache.remove(a); + return this; +}; +u.clear = function() { + for (const a of this.index.values()) { + a.clear(); + } + if (this.tag) { + for (const a of this.tag.values()) { + a.clear(); + } + } + this.store && this.store.clear(); + return this; +}; +u.contain = function(a) { + return this.A.has(a); +}; +u.cleanup = function() { + for (const a of this.index.values()) { + a.cleanup(); + } + return this; +}; +u.get = function(a) { + return this.store.get(a); +}; +u.set = function(a, c) { + this.store.set(a, c); + return this; +}; +u.searchCache = ia; +u.export = function(a, c, b, e, d, f) { + let g; + "undefined" === typeof f && (g = new Promise(k => { + f = k; + })); + d || (d = 0); + e || (e = 0); + if (e < this.field.length) { + b = this.field[e]; + var h = this.index[b]; + c = this; + h.export(a, c, d ? b : "", e, d++, f) || (e++, c.export(a, c, b, e, 1, f)); + } else { + switch(d) { + case 1: + c = "tag"; + h = this.B; + b = null; + break; + case 2: + c = "store"; + h = this.store; + b = null; + break; + default: + f(); + return; + } + la(a, this, b, c, e, d, h, f); + } + return g; +}; +u.import = function(a, c) { + if (c) { + switch(z(c) && (c = JSON.parse(c)), a) { + case "tag": + this.B = c; + break; + case "reg": + this.fastupdate = !1; + this.A = c; + for (let e = 0, d; e < this.field.length; e++) { + d = this.index[this.field[e]], d.A = c, d.fastupdate = !1; + } + break; + case "store": + this.store = c; + break; + default: + a = a.split("."); + const b = a[0]; + a = a[1]; + b && a && this.index[b].import(a, c); + } + } +}; +pa(X.prototype); +const ya = new Map([["b", "p"], ["v", "f"], ["w", "f"], ["z", "s"], ["x", "s"], ["d", "t"], ["n", "m"], ["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], ["i", "e"], ["y", "e"], ["u", "o"]]); +var za = {normalize:!0, C:!0, D:ya}; +const Aa = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]), Ba = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"]; +var Ca = {normalize:!0, C:!0, D:ya, I:Ba, H:Aa}; +var Ea = {normalize:!0, C:!0, D:ya, I:Ba.concat([/(?!^)[aeoy]/g, ""]), H:Aa}; +const Fa = {a:"", e:"", i:"", o:"", u:"", y:"", b:1, f:1, p:1, v:1, c:2, g:2, j:2, k:2, q:2, s:2, x:2, z:2, "\u00df":2, d:3, t:3, l:4, m:5, n:5, r:6}; +N["latin:exact"] = {normalize:!1, C:!1}; +N["latin:default"] = na; +N["latin:simple"] = {normalize:!0, C:!0}; +N["latin:balance"] = za; +N["latin:advanced"] = Ca; +N["latin:extra"] = Ea; +N["latin:soundex"] = {normalize:!0, C:!1, aa:{fa:!0}, S:function(a) { + for (let b = 0; b < a.length; b++) { + var c = a[b]; + let e = c.charAt(0), d = Fa[e]; + for (let f = 1, g; f < c.length && (g = c.charAt(f), "h" === g || "w" === g || !(g = Fa[g]) || g === d || (e += g, d = g, 4 !== e.length)); f++) { + } + a[b] = e; + } }}; -let Z; -(Z = self.define) && Z.amd ? Z([], function() { - return Y; -}) : self.exports ? self.exports = Y : self.FlexSearch = Y; +const Ga = {Index:P, Encoder:H, Charset:N, Language:ma, Document:X, Worker:null, Resolver:null, IndexedDB:null}, Z = self; +let Ha; +(Ha = Z.define) && Ha.amd ? Ha([], function() { + return Ga; +}) : "object" === typeof Z.exports ? Z.exports = Ga : Z.FlexSearch = Ga; }(this)); diff --git a/dist/flexsearch.compact.min.js b/dist/flexsearch.compact.min.js index b3879af..bbd3e24 100644 --- a/dist/flexsearch.compact.min.js +++ b/dist/flexsearch.compact.min.js @@ -1,27 +1,53 @@ /**! - * FlexSearch.js v0.7.41 (Compact) + * FlexSearch.js v0.8.0 (Bundle) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -(function(self){'use strict';var t;function v(a){return"undefined"!==typeof a?a:!0}function w(a){const b=Array(a);for(let c=0;c=this.m&&(u||!n[l])){var f=Q(q,e,r),g="";switch(this.C){case "full":if(2f;h--)if(h-f>=this.m){var k=Q(q,e,r,d,f);g=l.substring(f,h);S(this,n,g,k,a,c)}break}case "reverse":if(1=this.m&&S(this,n, -g,Q(q,e,r,d,h),a,c);g=""}case "forward":if(1=this.m&&S(this,n,g,f,a,c);break}default:if(this.F&&(f=Math.min(f/this.F(b,l,r)|0,q-1)),S(this,n,l,f,a,c),u&&1=this.m&&!d[l]){d[l]=1;const p=this.h&&l>f;S(this,m,p?f:l,Q(g+(e/2>g?0:1),e,r,h-1,k-1),a,c,p?l:f)}}}}this.D||(this.register[a]=1)}}return this}; -function Q(a,b,c,e,d){return c&&1=this.m&&!c[q])if(this.s||f||this.map[q])k[u++]=q,c[q]=1;else return e;a=k;d=a.length}if(!d)return e;b||(b=100);h=this.depth&&1=e)))break;if(n){if(f)return la(k,e,0);b[b.length]=k;return}}return!c&&k}function la(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function ma(a,b,c,e){c?(e=e&&b>c,a=(a=a[e?b:c])&&a[e?c:b]):a=a[b];return a}t.contain=function(a){return!!this.register[a]};t.update=function(a,b){return this.remove(a).add(a,b)};t.remove=function(a,b){const c=this.register[a];if(c){if(this.D)for(let e=0,d;eb||c)d=d.slice(c,c+b);e&&(d=pa.call(this,d));return{tag:a,result:d}}}function pa(a){const b=Array(a.length);for(let c=0,e;cthis.L.get(m)),k=1);this.H&&1this.H.get(m)),k=1);g&&k&&(g.lengththis.$&&(this.R.clear(), +this.B=this.B/1.1|0));g&&c.push(g)}this.S&&(c=this.S(c)||c);this.cache&&a.length<=this.h&&(this.N.set(a,c),this.N.size>this.$&&(this.N.clear(),this.h=this.h/1.1|0));return c};function ha(a){a.U=null;a.N.clear();a.R.clear()};function ia(a,b,c){a=("object"===typeof a?""+a.query:a).toLowerCase();let e=this.cache.get(a);if(!e){e=this.search(a,b,c);if(e instanceof Promise){const d=this;e.then(function(f){d.cache.set(a,f)})}this.cache.set(a,e)}return e}function I(a){this.limit=a&&!0!==a?a:1E3;this.cache=new Map;this.h=""}I.prototype.set=function(a,b){this.cache.has(a)||(this.cache.set(this.h=a,b),this.limit&&this.cache.size>this.limit&&this.cache.delete(this.cache.keys().next().value))}; +I.prototype.get=function(a){const b=this.cache.get(a);b&&this.limit&&this.h!==a&&(this.cache.delete(a),this.cache.set(this.h=a,b));return b};I.prototype.remove=function(a){for(const b of this.cache){const c=b[0];b[1].includes(a)&&this.cache.delete(c)}};I.prototype.clear=function(){this.cache.clear();this.h=""};function J(a=8){if(!(this instanceof J))return new J(a);this.index=y();this.F=[];this.size=0;32f;h--){g=n.substring(f,h);var k=this.score?this.score(b,n,p,g,f):P(t,e,p,d,f);Q(this,l,g,k,a,c)}break}case "reverse":if(1< +d){for(h=d-1;0g?0:1),e,p,h-1,k-1),C=this.bidirectional&&n>f;Q(this,m,C?f:n,v,a,c,C?n:f)}}}}this.fastupdate||this.A.add(a)}}return this}; +function Q(a,b,c,e,d,f,g){let h=g?a.K:a.map,k;b[c]&&g&&(k=b[c])[g]||(g?(b=k||(b[c]=y()),b[g]=1,(k=h.get(g))?h=k:h.set(g,h=new Map)):b[c]=1,(k=h.get(c))?h=k:h.set(c,h=[]),h=h[e]||(h[e]=[]),f&&h.includes(d)||(h.push(d),a.fastupdate&&((b=a.A.get(d))?b.push(h):a.A.set(d,[h]))))}function P(a,b,c,e,d){return c&&1b?b?a.slice(c,c+b):a.slice(c):a;let e=[];for(let d=0,f,g;d=g){c-=g;continue}cb&&(f=f.slice(0,b),g=f.length),e.push(f);else{if(g>=b)return g>b&&(f=f.slice(0,b)),f;e=[f]}b-=g;if(!b)break}return e.length?e=1=d)))break;if(h.length){if(g)return R(h,d,0);b.push(h);return}}return!c&&h}function U(a,b,c){let e;c&&(e=a.bidirectional&&b>c);a=c?(a=a.K.get(e?b:c))&&a.get(e?c:b):a.map.get(b);return a};O.prototype.remove=function(a,b){const c=this.A.size&&(this.fastupdate?this.A.get(a):this.A.has(a));if(c){if(this.fastupdate)for(let e=0,d;ed.length)d.pop();else{const f=d.indexOf(a);f===c.length-1?d.pop():d.splice(f,1)}}else V(this.map,a),this.depth&&V(this.K,a);b||this.A.delete(a)}this.cache&&this.cache.remove(a);return this}; +function V(a,b){let c=0;if(a.constructor===Array)for(let e=0,d,f;ec.add(a,b)):this.add(a,b)}return this.remove(a).add(a,b)};function W(a){let b=0;if(a.constructor===Array)for(let c=0,e;c{f=m}));let h,k;switch(d||(d=0)){case 0:h="reg";if(this.fastupdate){k=y();for(let m of this.A.keys())k[m]=1}else k=this.A;break;case 1:h="cfg";k={doc:0,opt:this.h?1:0};break;case 2:h="map";k=this.map;break;case 3:h="ctx";k=this.K;break;default:"undefined"===typeof c&&f&&f();return}la(a,b||this,c,h,e,d,k,f);return g}; +u.import=function(a,b){if(b)switch(z(b)&&(b=JSON.parse(b)),a){case "cfg":this.h=!!b.opt;break;case "reg":this.fastupdate=!1;this.A=b;break;case "map":this.map=b;break;case "ctx":this.K=b}};pa(O.prototype);X.prototype.add=function(a,b,c){B(a)&&(b=a,a=F(b,this.key));if(b&&(a||0===a)){if(!c&&this.A.has(a))return this.update(a,b);for(let h=0,k;hc||e)a=a.slice(e,e+c);d&&(a=wa.call(this,a));return a}} +function wa(a){const b=Array(a.length);for(let c=0,e;c{f=k}));d||(d=0);e||(e=0);if(e= this.m && (u || !n[l])) { - var f = T(q, e, r), g = ""; - switch(this.C) { - case "full": - if (2 < d) { - for (f = 0; f < d; f++) { - for (var h = d; h > f; h--) { - if (h - f >= this.m) { - var k = T(q, e, r, d, f); - g = l.substring(f, h); - U(this, n, g, k, a, c); - } - } - } - break; - } - case "reverse": - if (1 < d) { - for (h = d - 1; 0 < h; h--) { - g = l[h] + g, g.length >= this.m && U(this, n, g, T(q, e, r, d, h), a, c); - } - g = ""; - } - case "forward": - if (1 < d) { - for (h = 0; h < d; h++) { - g += l[h], g.length >= this.m && U(this, n, g, f, a, c); - } - break; - } - default: - if (this.F && (f = Math.min(f / this.F(b, l, r) | 0, q - 1)), U(this, n, l, f, a, c), u && 1 < e && r < e - 1) { - for (d = z(), g = this.o, f = l, h = Math.min(u + 1, e - r), d[f] = 1, k = 1; k < h; k++) { - if ((l = b[this.B ? e - 1 - r - k : r + k]) && l.length >= this.m && !d[l]) { - d[l] = 1; - const p = this.h && l > f; - U(this, m, p ? f : l, T(g + (e / 2 > g ? 0 : 1), e, r, h - 1, k - 1), a, c, p ? l : f); - } - } - } - } - } - } - this.D || (this.register[a] = 1); - } - } - return this; -}; -function T(a, b, c, e, d) { - return c && 1 < a ? b + (e || 0) <= a ? c + (d || 0) : (a - 1) / (b + (e || 0)) * (c + (d || 0)) + 1 | 0 : 0; -} -function U(a, b, c, e, d, f, g) { - let h = g ? a.l : a.map; - if (!b[c] || g && !b[c][g]) { - a.s && (h = h[e]), g ? (b = b[c] || (b[c] = z()), b[g] = 1, h = h[g] || (h[g] = z())) : b[c] = 1, h = h[c] || (h[c] = []), a.s || (h = h[e] || (h[e] = [])), f && h.includes(d) || (h[h.length] = d, a.D && (a = a.register[d] || (a.register[d] = []), a[a.length] = h)); - } -} -t.search = function(a, b, c) { - c || (!b && D(a) ? (c = a, a = c.query) : D(b) && (c = b)); - let e = [], d; - let f, g = 0; - if (c) { - a = c.query || a; - b = c.limit; - g = c.offset || 0; - var h = c.context; - f = c.suggest; - } - if (a && (a = this.encode("" + a), d = a.length, 1 < d)) { - c = z(); - var k = []; - for (let n = 0, u = 0, q; n < d; n++) { - if ((q = a[n]) && q.length >= this.m && !c[q]) { - if (this.s || f || this.map[q]) { - k[u++] = q, c[q] = 1; - } else { - return e; - } - } - } - a = k; - d = a.length; - } - if (!d) { - return e; - } - b || (b = 100); - h = this.depth && 1 < d && !1 !== h; - c = 0; - let m; - h ? (m = a[0], c = 1) : 1 < d && a.sort(aa); - for (let n, u; c < d; c++) { - u = a[c]; - h ? (n = ia(this, e, f, b, g, 2 === d, u, m), f && !1 === n && e.length || (m = u)) : n = ia(this, e, f, b, g, 1 === d, u); - if (n) { - return n; - } - if (f && c === d - 1) { - k = e.length; - if (!k) { - if (h) { - h = 0; - c = -1; - continue; - } - return e; - } - if (1 === k) { - return ja(e[0], b, g); - } - } - } - return Q(e, b, g, f); -}; -function ia(a, b, c, e, d, f, g, h) { - let k = [], m = h ? a.l : a.map; - a.s || (m = ka(m, g, h, a.h)); - if (m) { - let n = 0; - const u = Math.min(m.length, h ? a.o : a.A); - for (let q = 0, r = 0, l, p; q < u; q++) { - if (l = m[q]) { - if (a.s && (l = ka(l, g, h, a.h)), d && l && f && (p = l.length, p <= d ? (d -= p, l = null) : (l = l.slice(d), d = 0)), l && (k[n++] = l, f && (r += l.length, r >= e))) { - break; - } - } - } - if (n) { - if (f) { - return ja(k, e, 0); - } - b[b.length] = k; - return; - } - } - return !c && k; -} -function ja(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function ka(a, b, c, e) { - c ? (e = e && b > c, a = (a = a[e ? b : c]) && a[e ? c : b]) : a = a[b]; - return a; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a, b) { - const c = this.register[a]; - if (c) { - if (this.D) { - for (let e = 0, d; e < c.length; e++) { - d = c[e], d.splice(d.indexOf(a), 1); - } - } else { - V(this.map, a, this.A, this.s), this.depth && V(this.l, a, this.o, this.s); - } - b || delete this.register[a]; - } - return this; -}; -function V(a, b, c, e, d) { - let f = 0; - if (a.constructor === Array) { - if (d) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), f++) : f++; - } else { - d = Math.min(a.length, c); - for (let g = 0, h; g < d; g++) { - if (h = a[g]) { - f = V(h, b, c, e, d), e || f || delete a[g]; - } - } - } - } else { - for (let g in a) { - (f = V(a[g], b, c, e, d)) || delete a[g]; - } - } - return f; -} -O(S.prototype); -function W(a) { - if (!(this instanceof W)) { - return new W(a); - } - var b = a.document || a.doc || a, c; - this.F = []; - this.h = []; - this.o = []; - this.register = z(); - this.key = (c = b.key || b.id) && X(c, this.o) || "id"; - this.D = v(a.fastupdate); - this.l = (c = b.store) && !0 !== c && []; - this.store = c && z(); - this.async = !1; - c = z(); - let e = b.index || b.field || b; - C(e) && (e = [e]); - for (let d = 0, f, g; d < e.length; d++) { - f = e[d], C(f) || (g = f, f = f.field), g = D(g) ? Object.assign({}, a, g) : a, this.I || (c[f] = new S(g, this.register)), this.F[d] = X(f, this.o), this.h[d] = f; - } - if (this.l) { - for (a = b.store, C(a) && (a = [a]), b = 0; b < a.length; b++) { - this.l[b] = X(a[b], this.o); - } - } - this.index = c; -} -function X(a, b) { - const c = a.split(":"); - let e = 0; - for (let d = 0; d < c.length; d++) { - a = c[d], 0 <= a.indexOf("[]") && (a = a.substring(0, a.length - 2)) && (b[e] = !0), a && (c[e++] = a); - } - e < c.length && (c.length = e); - return 1 < e ? c : c[0]; -} -function la(a, b) { - if (C(b)) { +function ba(a, b) { + if (z(b)) { a = a[b]; } else { for (let c = 0; a && c < b.length; c++) { @@ -476,21 +65,734 @@ function la(a, b) { } return a; } -function Y(a, b, c, e, d) { +;const ca = /[^\p{L}\p{N}]+/u, da = /(\d{3})/g, ea = /(\D)(\d{3})/g, fa = /(\d{3})(\D)/g, F = "".normalize && /[\u0300-\u036f]/g, ha = !F && new Map([["\u00aa", "a"], ["\u00b2", "2"], ["\u00b3", "3"], ["\u00b9", "1"], ["\u00ba", "o"], ["\u00bc", "1\u20444"], ["\u00bd", "1\u20442"], ["\u00be", "3\u20444"], ["\u00e0", "a"], ["\u00e1", "a"], ["\u00e2", "a"], ["\u00e3", "a"], ["\u00e4", "a"], ["\u00e5", "a"], ["\u00e7", "c"], ["\u00e8", "e"], ["\u00e9", "e"], ["\u00ea", "e"], ["\u00eb", "e"], ["\u00ec", +"i"], ["\u00ed", "i"], ["\u00ee", "i"], ["\u00ef", "i"], ["\u00f1", "n"], ["\u00f2", "o"], ["\u00f3", "o"], ["\u00f4", "o"], ["\u00f5", "o"], ["\u00f6", "o"], ["\u00f9", "u"], ["\u00fa", "u"], ["\u00fb", "u"], ["\u00fc", "u"], ["\u00fd", "y"], ["\u00ff", "y"], ["\u0101", "a"], ["\u0103", "a"], ["\u0105", "a"], ["\u0107", "c"], ["\u0109", "c"], ["\u010b", "c"], ["\u010d", "c"], ["\u010f", "d"], ["\u0113", "e"], ["\u0115", "e"], ["\u0117", "e"], ["\u0119", "e"], ["\u011b", "e"], ["\u011d", "g"], ["\u011f", +"g"], ["\u0121", "g"], ["\u0123", "g"], ["\u0125", "h"], ["\u0129", "i"], ["\u012b", "i"], ["\u012d", "i"], ["\u012f", "i"], ["\u0133", "ij"], ["\u0135", "j"], ["\u0137", "k"], ["\u013a", "l"], ["\u013c", "l"], ["\u013e", "l"], ["\u0140", "l"], ["\u0144", "n"], ["\u0146", "n"], ["\u0148", "n"], ["\u0149", "n"], ["\u014d", "o"], ["\u014f", "o"], ["\u0151", "o"], ["\u0155", "r"], ["\u0157", "r"], ["\u0159", "r"], ["\u015b", "s"], ["\u015d", "s"], ["\u015f", "s"], ["\u0161", "s"], ["\u0163", "t"], ["\u0165", +"t"], ["\u0169", "u"], ["\u016b", "u"], ["\u016d", "u"], ["\u016f", "u"], ["\u0171", "u"], ["\u0173", "u"], ["\u0175", "w"], ["\u0177", "y"], ["\u017a", "z"], ["\u017c", "z"], ["\u017e", "z"], ["\u017f", "s"], ["\u01a1", "o"], ["\u01b0", "u"], ["\u01c6", "dz"], ["\u01c9", "lj"], ["\u01cc", "nj"], ["\u01ce", "a"], ["\u01d0", "i"], ["\u01d2", "o"], ["\u01d4", "u"], ["\u01d6", "u"], ["\u01d8", "u"], ["\u01da", "u"], ["\u01dc", "u"], ["\u01df", "a"], ["\u01e1", "a"], ["\u01e3", "ae"], ["\u00e6", "ae"], +["\u01fd", "ae"], ["\u01e7", "g"], ["\u01e9", "k"], ["\u01eb", "o"], ["\u01ed", "o"], ["\u01ef", "\u0292"], ["\u01f0", "j"], ["\u01f3", "dz"], ["\u01f5", "g"], ["\u01f9", "n"], ["\u01fb", "a"], ["\u01ff", "\u00f8"], ["\u0201", "a"], ["\u0203", "a"], ["\u0205", "e"], ["\u0207", "e"], ["\u0209", "i"], ["\u020b", "i"], ["\u020d", "o"], ["\u020f", "o"], ["\u0211", "r"], ["\u0213", "r"], ["\u0215", "u"], ["\u0217", "u"], ["\u0219", "s"], ["\u021b", "t"], ["\u021f", "h"], ["\u0227", "a"], ["\u0229", "e"], +["\u022b", "o"], ["\u022d", "o"], ["\u022f", "o"], ["\u0231", "o"], ["\u0233", "y"], ["\u02b0", "h"], ["\u02b1", "h"], ["\u0266", "h"], ["\u02b2", "j"], ["\u02b3", "r"], ["\u02b4", "\u0279"], ["\u02b5", "\u027b"], ["\u02b6", "\u0281"], ["\u02b7", "w"], ["\u02b8", "y"], ["\u02e0", "\u0263"], ["\u02e1", "l"], ["\u02e2", "s"], ["\u02e3", "x"], ["\u02e4", "\u0295"], ["\u0390", "\u03b9"], ["\u03ac", "\u03b1"], ["\u03ad", "\u03b5"], ["\u03ae", "\u03b7"], ["\u03af", "\u03b9"], ["\u03b0", "\u03c5"], ["\u03ca", +"\u03b9"], ["\u03cb", "\u03c5"], ["\u03cc", "\u03bf"], ["\u03cd", "\u03c5"], ["\u03ce", "\u03c9"], ["\u03d0", "\u03b2"], ["\u03d1", "\u03b8"], ["\u03d2", "\u03a5"], ["\u03d3", "\u03a5"], ["\u03d4", "\u03a5"], ["\u03d5", "\u03c6"], ["\u03d6", "\u03c0"], ["\u03f0", "\u03ba"], ["\u03f1", "\u03c1"], ["\u03f2", "\u03c2"], ["\u03f5", "\u03b5"], ["\u0439", "\u0438"], ["\u0450", "\u0435"], ["\u0451", "\u0435"], ["\u0453", "\u0433"], ["\u0457", "\u0456"], ["\u045c", "\u043a"], ["\u045d", "\u0438"], ["\u045e", +"\u0443"], ["\u0477", "\u0475"], ["\u04c2", "\u0436"], ["\u04d1", "\u0430"], ["\u04d3", "\u0430"], ["\u04d7", "\u0435"], ["\u04db", "\u04d9"], ["\u04dd", "\u0436"], ["\u04df", "\u0437"], ["\u04e3", "\u0438"], ["\u04e5", "\u0438"], ["\u04e7", "\u043e"], ["\u04eb", "\u04e9"], ["\u04ed", "\u044d"], ["\u04ef", "\u0443"], ["\u04f1", "\u0443"], ["\u04f3", "\u0443"], ["\u04f5", "\u0447"]]); +function G(a = {}) { + if (!(this instanceof G)) { + return new G(...arguments); + } + for (a = 0; a < arguments.length; a++) { + this.assign(arguments[a]); + } +} +G.prototype.assign = function(a) { + this.normalize = x(a.normalize, !0, this.normalize); + let b = a.U, c = b || a.Z || a.split; + if ("object" === typeof c) { + let e = !b, d = ""; + a.U || (d += "\\p{Z}"); + c.Y && (d += "\\p{L}"); + c.$ && (d += "\\p{N}", e = !!b); + c.ba && (d += "\\p{S}"); + c.aa && (d += "\\p{P}"); + c.control && (d += "\\p{C}"); + if (c = c.char) { + d += "object" === typeof c ? c.join("") : c; + } + this.split = new RegExp("[" + (b ? "^" : "") + d + "]+", "u"); + this.numeric = e; + } else { + this.split = x(c, ca, this.split), this.numeric = x(this.numeric, !0); + } + this.R = x(a.R, null, this.R); + this.M = x(a.M, null, this.M); + this.rtl = a.rtl || !1; + this.C = x(a.C, !0, this.C); + this.filter = x((c = a.filter) && new Set(c), null, this.filter); + this.H = x((c = a.H) && new Map(c), null, this.H); + this.D = x((c = a.D) && new Map(c), null, this.D); + this.K = x((c = a.K) && new Map(c), null, this.K); + this.I = x(a.I, null, this.I); + this.P = x(a.P, 1, this.P); + this.V = x(a.V, 0, this.V); + this.A = ""; + this.F = null; + this.B = ""; + this.S = null; + if (this.H) { + for (const e of this.H.keys()) { + this.A += (this.A ? "|" : "") + e; + } + } + if (this.K) { + for (const e of this.K.keys()) { + this.B += (this.B ? "|" : "") + e; + } + } + return this; +}; +G.prototype.encode = function(a) { + this.normalize && ("function" === typeof this.normalize ? a = this.normalize(a) : F ? a = a.normalize("NFKD").replace(F, "").toLowerCase() : (a = a.toLowerCase(), this.D = this.D ? new Map([...ha, ...this.D]) : new Map(ha))); + this.R && (a = this.R(a)); + this.numeric && 3 < a.length && (a = a.replace(ea, "$1 $2").replace(fa, "$1 $2").replace(da, "$1 ")); + const b = !(this.C || this.D || this.filter || this.H || this.K || this.I); + let c = []; + a = this.split || "" === this.split ? a.split(this.split) : a; + for (let d = 0, f; d < a.length; d++) { + if (!(f = a[d])) { + continue; + } + if (f.length < this.P) { + continue; + } + if (b) { + c.push(f); + continue; + } + if (this.filter && this.filter.has(f)) { + continue; + } + let g; + this.K && 2 < f.length && (this.S || (this.S = new RegExp("(?!^)(" + this.B + ")$")), f = f.replace(this.S, h => this.K.get(h)), g = 1); + this.H && 1 < f.length && (this.F || (this.F = new RegExp("(" + this.A + ")", "g")), f = f.replace(this.F, h => this.H.get(h)), g = 1); + f && g && (f.length < this.P || this.filter && this.filter.has(f)) && (f = ""); + if (f && (this.D || this.C && 1 < f.length)) { + var e = ""; + for (let h = 0, k = "", n, l; h < f.length; h++) { + n = f.charAt(h), n === k && this.C || ((l = this.D && this.D.get(n)) || "" === l ? l === k && this.C || !(k = l) || (e += l) : e += k = n); + } + f = e; + } + if (f && this.I) { + for (e = 0; f && e < this.I.length; e += 2) { + f = f.replace(this.I[e], this.I[e + 1]); + } + } + f && c.push(f); + } + this.M && (c = this.M(c) || c); + return c; +}; +function H(a = 8) { + if (!(this instanceof H)) { + return new H(a); + } + this.index = y(); + this.F = []; + this.size = 0; + 32 < a ? (this.A = ia, this.B = BigInt(a)) : (this.A = ja, this.B = a); +} +H.prototype.get = function(a) { + const b = this.index[this.A(a)]; + return b && b.get(a); +}; +H.prototype.set = function(a, b) { + var c = this.A(a); + let e = this.index[c]; + e ? (c = e.size, e.set(a, b), (c -= e.size) && this.size++) : (this.index[c] = e = new Map([[a, b]]), this.F.push(e)); +}; +function I(a = 8) { + if (!(this instanceof I)) { + return new I(a); + } + this.index = y(); + this.A = []; + 32 < a ? (this.F = ia, this.B = BigInt(a)) : (this.F = ja, this.B = a); +} +I.prototype.add = function(a) { + var b = this.F(a); + let c = this.index[b]; + c ? (b = c.size, c.add(a), (b -= c.size) && this.size++) : (this.index[b] = c = new Set([a]), this.A.push(c)); +}; +u = H.prototype; +u.has = I.prototype.has = function(a) { + const b = this.index[this.F(a)]; + return b && b.has(a); +}; +u.delete = I.prototype.delete = function(a) { + const b = this.index[this.F(a)]; + b && b.delete(a) && this.size--; +}; +u.clear = I.prototype.clear = function() { + this.index = y(); + this.A = []; + this.size = 0; +}; +u.values = I.prototype.values = function*() { + for (let a = 0; a < this.A.length; a++) { + for (let b of this.A[a].values()) { + yield b; + } + } +}; +u.keys = I.prototype.keys = function*() { + for (let a = 0; a < this.A.length; a++) { + for (let b of this.A[a].keys()) { + yield b; + } + } +}; +u.entries = I.prototype.entries = function*() { + for (let a = 0; a < this.A.length; a++) { + for (let b of this.A[a].entries()) { + yield b; + } + } +}; +function ja(a) { + let b = 2 ** this.B - 1; + if ("number" == typeof a) { + return a & b; + } + let c = 0, e = this.B + 1; + for (let d = 0; d < a.length; d++) { + c = (c * e ^ a.charCodeAt(d)) & b; + } + return 32 === this.B ? c + 2 ** 31 : c; +} +function ia(a) { + let b = BigInt(2) ** this.B - BigInt(1); + var c = typeof a; + if ("bigint" === c) { + return a & b; + } + if ("number" === c) { + return BigInt(a) & b; + } + c = BigInt(0); + let e = this.B + BigInt(1); + for (let d = 0; d < a.length; d++) { + c = (c * e ^ BigInt(a.charCodeAt(d))) & b; + } + return c; +} +;function ka(a, b, c, e, d, f, g, h) { + (e = a(c ? c + "." + e : e, JSON.stringify(g))) && e.then ? e.then(function() { + b.export(a, b, c, d, f + 1, h); + }) : b.export(a, b, c, d, f + 1, h); +} +;const la = y(), J = y(); +var ma = {normalize:function(a) { + return a.toLowerCase(); +}, C:!1}; +const na = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +function oa(a) { + M.call(a, "add"); + M.call(a, "append"); + M.call(a, "search"); + M.call(a, "update"); + M.call(a, "remove"); +} +function M(a) { + this[a + "Async"] = function() { + var b = arguments; + const c = b[b.length - 1]; + let e; + "function" === typeof c && (e = c, delete b[b.length - 1]); + this.async = !0; + b = this[a].apply(this, b); + this.async = !1; + b.then ? b.then(e) : e(b); + return b; + }; +} +;y(); +N.prototype.add = function(a, b, c, e) { + if (b && (a || 0 === a)) { + if (!e && !c && this.h.has(a)) { + return this.update(a, b); + } + b = this.encoder.encode(b); + if (e = b.length) { + const n = y(), l = y(), r = this.depth, t = this.resolution; + for (let p = 0; p < e; p++) { + let m = b[this.rtl ? e - 1 - p : p]; + var d = m.length; + if (d && (r || !l[m])) { + var f = this.score ? this.score(b, m, p, null, 0) : O(t, e, p), g = ""; + switch(this.tokenize) { + case "full": + if (2 < d) { + for (f = 0; f < d; f++) { + for (var h = d; h > f; h--) { + g = m.substring(f, h); + var k = this.score ? this.score(b, m, p, g, f) : O(t, e, p, d, f); + P(this, l, g, k, a, c); + } + } + break; + } + case "reverse": + if (1 < d) { + for (h = d - 1; 0 < h; h--) { + g = m[h] + g, k = this.score ? this.score(b, m, p, g, h) : O(t, e, p, d, h), P(this, l, g, k, a, c); + } + g = ""; + } + case "forward": + if (1 < d) { + for (h = 0; h < d; h++) { + g += m[h], P(this, l, g, f, a, c); + } + break; + } + default: + if (P(this, l, m, f, a, c), r && 1 < e && p < e - 1) { + for (d = y(), g = this.W, f = m, h = Math.min(r + 1, e - p), d[f] = 1, k = 1; k < h; k++) { + if ((m = b[this.rtl ? e - 1 - p - k : p + k]) && !d[m]) { + d[m] = 1; + const v = this.score ? this.score(b, f, p, m, k) : O(g + (e / 2 > g ? 0 : 1), e, p, h - 1, k - 1), C = this.bidirectional && m > f; + P(this, n, C ? f : m, v, a, c, C ? m : f); + } + } + } + } + } + } + this.fastupdate || this.h.add(a); + } + } + return this; +}; +function P(a, b, c, e, d, f, g) { + let h = g ? a.J : a.map, k; + b[c] && g && (k = b[c])[g] || (g ? (b = k || (b[c] = y()), b[g] = 1, (k = h.get(g)) ? h = k : h.set(g, h = new Map())) : b[c] = 1, (k = h.get(c)) ? h = k : h.set(c, h = []), h = h[e] || (h[e] = []), f && h.includes(d) || (h.push(d), a.fastupdate && ((b = a.h.get(d)) ? b.push(h) : a.h.set(d, [h])))); +} +function O(a, b, c, e, d) { + return c && 1 < a ? b + (e || 0) <= a ? c + (d || 0) : (a - 1) / (b + (e || 0)) * (c + (d || 0)) + 1 | 0 : 0; +} +;function Q(a, b, c) { + if (1 === a.length) { + return a = a[0], a = c || a.length > b ? b ? a.slice(c, c + b) : a.slice(c) : a; + } + let e = []; + for (let d = 0, f, g; d < a.length; d++) { + if ((f = a[d]) && (g = f.length)) { + if (c) { + if (c >= g) { + c -= g; + continue; + } + c < g && (f = b ? f.slice(c, c + b) : f.slice(c), g = f.length, c = 0); + } + if (e.length) { + g > b && (f = f.slice(0, b), g = f.length), e.push(f); + } else { + if (g >= b) { + return g > b && (f = f.slice(0, b)), f; + } + e = [f]; + } + b -= g; + if (!b) { + break; + } + } + } + return e.length ? e = 1 < e.length ? [].concat.apply([], e) : e[0] : e; +} +;function pa(a, b, c, e) { + var d = a.length; + let f = [], g = 0, h, k, n; + e && (e = []); + for (let l = d - 1, r; 0 <= l; l--) { + n = a[l]; + d = y(); + r = !h; + for (let t = 0, p; t < n.length; t++) { + if ((p = n[t]) && p.length) { + for (let m = 0, v; m < p.length; m++) { + if (v = p[m], h) { + if (h[v]) { + if (!l) { + if (c) { + c--; + } else { + if (f[g++] = v, g === b) { + return f; + } + } + } + if (l || e) { + d[v] = 1; + } + r = !0; + } + e && !k[v] && (k[v] = 1, (e[t] || (e[t] = [])).push(v)); + } else { + d[v] = 1; + } + } + } + } + if (e) { + h || (k = d); + } else if (!r) { + return []; + } + h = d; + } + if (e) { + for (let l = e.length - 1, r, t; 0 <= l; l--) { + r = e[l]; + t = r.length; + for (let p = 0, m; p < t; p++) { + if (m = r[p], !h[m]) { + if (c) { + c--; + } else { + if (f[g++] = m, g === b) { + return f; + } + } + h[m] = 1; + } + } + } + } + return f; +} +function qa(a, b) { + const c = y(), e = y(), d = []; + for (let f = 0; f < a.length; f++) { + c[a[f]] = 1; + } + for (let f = 0, g; f < b.length; f++) { + g = b[f]; + for (let h = 0, k; h < g.length; h++) { + k = g[h], c[k] && !e[k] && (e[k] = 1, d.push(k)); + } + } + return d; +} +;N.prototype.search = function(a, b, c) { + c || (!b && B(a) ? (c = a, a = "") : B(b) && (c = b, b = 0)); + let e = []; + let d, f = 0; + if (c) { + a = c.query || a; + b = c.limit || b; + f = c.offset || 0; + var g = c.context; + d = c.suggest; + } + a = this.encoder.encode(a); + c = a.length; + b || (b = 100); + if (1 === c) { + return R.call(this, a[0], "", b, f); + } + g = this.depth && !1 !== g; + if (2 === c && g && !d) { + return R.call(this, a[0], a[1], b, f); + } + let h = 0, k = 0; + if (1 < c) { + var n = y(); + const r = []; + for (let t = 0, p; t < c; t++) { + if ((p = a[t]) && !n[p]) { + if (d || S(this, p)) { + r.push(p), n[p] = 1; + } else { + return e; + } + const m = p.length; + h = Math.max(h, m); + k = k ? Math.min(k, m) : m; + } + } + a = r; + c = a.length; + } + if (!c) { + return e; + } + n = 0; + let l; + if (1 === c) { + return R.call(this, a[0], "", b, f); + } + if (2 === c && g && !d) { + return R.call(this, a[0], a[1], b, f); + } + 1 < c && (g ? (l = a[0], n = 1) : 9 < h && 3 < h / k && a.sort(aa)); + for (let r, t; n < c; n++) { + t = a[n]; + l ? (r = S(this, t, l), r = ra(r, e, d, this.W, b, f, 2 === c), d && !1 === r && e.length || (l = t)) : (r = S(this, t), r = ra(r, e, d, this.resolution, b, f, 1 === c)); + if (r) { + return r; + } + if (d && n === c - 1) { + g = e.length; + if (!g) { + if (l) { + l = ""; + n = -1; + continue; + } + return e; + } + if (1 === g) { + return Q(e[0], b, f); + } + } + } + return pa(e, b, f, d); +}; +function R(a, b, c, e) { + return (a = S(this, a, b)) && a.length ? Q(a, c, e) : []; +} +function ra(a, b, c, e, d, f, g) { + let h = []; + if (a) { + e = Math.min(a.length, e); + for (let k = 0, n = 0, l; k < e; k++) { + if (l = a[k]) { + if (f && l && g && (l.length <= f ? (f -= l.length, l = null) : (l = l.slice(f), f = 0)), l && (h[k] = l, g && (n += l.length, n >= d))) { + break; + } + } + } + if (h.length) { + if (g) { + return Q(h, d, 0); + } + b.push(h); + return; + } + } + return !c && h; +} +function S(a, b, c) { + let e; + c && (e = a.bidirectional && b > c); + a = c ? (a = a.J.get(e ? b : c)) && a.get(e ? c : b) : a.map.get(b); + return a; +} +;N.prototype.remove = function(a, b) { + const c = this.h.size && (this.fastupdate ? this.h.get(a) : this.h.has(a)); + if (c) { + if (this.fastupdate) { + for (let e = 0, d; e < c.length; e++) { + if (d = c[e]) { + if (2 > d.length) { + d.pop(); + } else { + const f = d.indexOf(a); + f === c.length - 1 ? d.pop() : d.splice(f, 1); + } + } + } + } else { + T(this.map, a), this.depth && T(this.J, a); + } + b || this.h.delete(a); + } + return this; +}; +function T(a, b) { + let c = 0; + if (a.constructor === Array) { + for (let e = 0, d, f; e < a.length; e++) { + if ((d = a[e]) && d.length) { + if (f = d.indexOf(b), 0 <= f) { + 1 < d.length ? (d.splice(f, 1), c++) : delete a[e]; + break; + } else { + c++; + } + } + } + } else { + for (let e of a) { + const d = e[0], f = T(e[1], b); + f ? c += f : a.delete(d); + } + } + return c; +} +;function N(a, b) { + if (!(this instanceof N)) { + return new N(a); + } + if (a) { + var c = z(a) ? a : a.preset; + c && (na[c] || console.warn("Preset not found: " + c), a = Object.assign({}, na[c], a)); + } else { + a = {}; + } + c = a.context || {}; + const e = a.encode || a.encoder || ma; + this.encoder = e.encode ? e : "object" === typeof e ? new G(e) : {encode:e}; + let d; + this.resolution = a.resolution || 9; + this.tokenize = d = a.tokenize || "strict"; + this.depth = "strict" === d && c.depth || 0; + this.bidirectional = !1 !== c.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + this.map = (d = !1, new Map()); + this.J = d ? new H(d) : new Map(); + this.h = b || (this.fastupdate ? d ? new H(d) : new Map() : d ? new I(d) : new Set()); + this.W = c.resolution || 1; + this.rtl = e.rtl || a.rtl || !1; +} +u = N.prototype; +u.clear = function() { + this.map.clear(); + this.J.clear(); + this.h.clear(); + return this; +}; +u.append = function(a, b) { + return this.add(a, b, !0); +}; +u.contain = function(a) { + return this.h.has(a); +}; +u.update = function(a, b) { + if (this.async) { + const c = this, e = this.remove(a); + return e.then ? e.then(() => c.add(a, b)) : this.add(a, b); + } + return this.remove(a).add(a, b); +}; +function U(a) { + let b = 0; + if (a.constructor === Array) { + for (let c = 0, e; c < a.length; c++) { + (e = a[c]) && (b += e.length); + } + } else { + for (const c of a) { + const e = c[0], d = U(c[1]); + d ? b += d : a.delete(e); + } + } + return b; +} +u.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + U(this.map); + this.depth && U(this.J); + return this; +}; +u.export = function(a, b, c, e, d, f) { + let g = !0; + "undefined" === typeof f && (g = new Promise(n => { + f = n; + })); + let h, k; + switch(d || (d = 0)) { + case 0: + h = "reg"; + if (this.fastupdate) { + k = y(); + for (let n of this.h.keys()) { + k[n] = 1; + } + } else { + k = this.h; + } + break; + case 1: + h = "cfg"; + k = {doc:0, opt:this.A ? 1 : 0}; + break; + case 2: + h = "map"; + k = this.map; + break; + case 3: + h = "ctx"; + k = this.J; + break; + default: + "undefined" === typeof c && f && f(); + return; + } + ka(a, b || this, c, h, e, d, k, f); + return g; +}; +u.import = function(a, b) { + if (b) { + switch(z(b) && (b = JSON.parse(b)), a) { + case "cfg": + this.A = !!b.opt; + break; + case "reg": + this.fastupdate = !1; + this.h = b; + break; + case "map": + this.map = b; + break; + case "ctx": + this.J = b; + } + } +}; +oa(N.prototype); +V.prototype.add = function(a, b, c) { + B(a) && (b = a, a = ba(b, this.key)); + if (b && (a || 0 === a)) { + if (!c && this.h.has(a)) { + return this.update(a, b); + } + for (let d = 0, f; d < this.field.length; d++) { + f = this.L[d]; + const g = this.index.get(this.field[d]); + if ("function" === typeof f) { + var e = f(b); + e && g.add(a, e, !1, !0); + } else { + if (e = f.T, !e || e(b)) { + f instanceof String ? f = ["" + f] : z(f) && (f = [f]), W(b, f, this.O, 0, g, a, f[0], c); + } + } + } + if (this.store && (!c || !this.store.has(a))) { + let d; + if (this.G) { + d = y(); + for (let f = 0, g; f < this.G.length; f++) { + g = this.G[f]; + if ((c = g.T) && !c(b)) { + continue; + } + let h; + if ("function" === typeof g) { + h = g(b); + if (!h) { + continue; + } + g = [g.X]; + } else if (z(g) || g instanceof String) { + d[g] = b[g]; + continue; + } + X(b, d, g, 0, g[0], h); + } + } + this.store.set(a, d || b); + } + } + return this; +}; +function X(a, b, c, e, d, f) { a = a[d]; if (e === c.length - 1) { - b[d] = a; + b[d] = f || a; } else if (a) { if (a.constructor === Array) { for (b = b[d] = Array(a.length), d = 0; d < a.length; d++) { - Y(a, b, c, e, d); + X(a, b, c, e, d); } } else { - b = b[d] || (b[d] = z()), d = c[++e], Y(a, b, c, e, d); + b = b[d] || (b[d] = y()), d = c[++e], X(a, b, c, e, d); } } } -function Z(a, b, c, e, d, f, g, h) { +function W(a, b, c, e, d, f, g, h) { if (a = a[g]) { if (e === b.length - 1) { if (a.constructor === Array) { @@ -506,209 +808,353 @@ function Z(a, b, c, e, d, f, g, h) { } else { if (a.constructor === Array) { for (g = 0; g < a.length; g++) { - Z(a, b, c, e, d, f, g, h); + W(a, b, c, e, d, f, g, h); } } else { - g = b[++e], Z(a, b, c, e, d, f, g, h); + g = b[++e], W(a, b, c, e, d, f, g, h); } } } } -t = W.prototype; -t.add = function(a, b, c) { - D(a) && (b = a, a = la(b, this.key)); - if (b && (a || 0 === a)) { - if (!c && this.register[a]) { - return this.update(a, b); - } - for (let e = 0, d, f; e < this.h.length; e++) { - f = this.h[e], d = this.F[e], C(d) && (d = [d]), Z(b, d, this.o, 0, this.index[f], a, d[0], c); - } - if (this.store && (!c || !this.store[a])) { - let e; - if (this.l) { - e = z(); - for (let d = 0, f; d < this.l.length; d++) { - f = this.l[d], C(f) ? e[f] = b[f] : Y(b, e, f, 0, f[0]); - } - } - this.store[a] = e || b; - } - } - return this; -}; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a) { - D(a) && (a = la(a, this.key)); - if (this.register[a]) { - for (let b = 0; b < this.h.length && (this.index[this.h[b]].remove(a, !this.I), !this.D); b++) { - } - this.store && delete this.store[a]; - delete this.register[a]; - } - return this; -}; -t.search = function(a, b, c, e) { - c || (!b && D(a) ? (c = a, a = "") : D(b) && (c = b, b = 0)); - let d = [], f = [], g, h, k, m, n, u, q = 0; +;V.prototype.search = function(a, b, c, e) { + c || (!b && B(a) ? (c = a, a = "") : B(b) && (c = b, b = 0)); + let d = []; + var f = []; + let g, h, k, n, l; + let r = 0; if (c) { if (c.constructor === Array) { - k = c, c = null; + n = c, c = null; } else { a = c.query || a; - k = (g = c.pluck) || c.index || c.field; - m = !1; - h = this.store && c.enrich; - n = "and" === c.bool; - b = c.limit || b || 100; - u = c.offset || 0; - if (m && (C(m) && (m = [m]), !a)) { - for (let l = 0, p; l < m.length; l++) { - if (p = ma.call(this, m[l], b, u, h)) { - d[d.length] = p, q++; + var t = c.pluck; + h = c.merge; + n = t || c.field || c.index; + l = !1; + g = this.store && c.enrich; + k = c.suggest; + b = c.limit || b; + var p = c.offset || 0; + b || (b = 100); + if (l) { + l.constructor !== Array && (l = [l]); + var m = []; + for (let w = 0, q; w < l.length; w++) { + q = l[w]; + if (z(q)) { + throw Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + if (q.field && q.tag) { + var v = q.tag; + if (v.constructor === Array) { + for (var C = 0; C < v.length; C++) { + m.push(q.field, v[C]); + } + } else { + m.push(q.field, v); + } + } else { + v = Object.keys(q); + for (let D = 0, E, A; D < v.length; D++) { + if (E = v[D], A = q[E], A.constructor === Array) { + for (C = 0; C < A.length; C++) { + m.push(E, A[C]); + } + } else { + m.push(E, A); + } + } } } - return q ? d : []; + if (!m.length) { + throw Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + l = m; + if (!a) { + e = []; + if (m.length) { + for (f = 0; f < m.length; f += 2) { + t = sa.call(this, m[f], m[f + 1], b, p, g), d.push({field:m[f], tag:m[f + 1], result:t}); + } + } + return e.length ? Promise.all(e).then(function(w) { + for (let q = 0; q < w.length; q++) { + d[q].result = w[q]; + } + return d; + }) : d; + } } - C(k) && (k = [k]); + z(n) && (n = [n]); } } - k || (k = this.h); - n = n && (1 < k.length || m && 1 < m.length); - const r = !e && (this.I || this.async) && []; - for (let l = 0, p, A, B; l < k.length; l++) { - let y; - A = k[l]; - C(A) || (y = A, A = y.field, a = y.query || a, b = y.limit || b, h = y.enrich || h); - if (r) { - r[l] = this.index[A].searchAsync(a, b, y || c); + n || (n = this.field); + p = !e && (this.worker || this.async) && []; + for (let w = 0, q, D, E; w < n.length; w++) { + D = n[w]; + let A; + z(D) || (A = D, D = A.field, a = A.query || a, b = A.limit || b, k = A.suggest || k); + if (e) { + q = e[w]; } else { - e ? p = e[l] : p = this.index[A].search(a, b, y || c); - B = p && p.length; - if (m && B) { - const x = []; - let G = 0; - n && (x[0] = [p]); - for (let R = 0, fa, N; R < m.length; R++) { - if (fa = m[R], B = (N = this.J[fa]) && N.length) { - G++, x[x.length] = n ? [N] : N; - } - } - G && (p = n ? Q(x, b || 100, u || 0) : ea(p, x), B = p.length); - } - if (B) { - f[q] = A, d[q++] = p; - } else if (n) { - return []; + if (m = A || c, v = this.index.get(D), l && (m.enrich = !1), p) { + p[w] = v.searchAsync(a, b, m); + m && g && (m.enrich = g); + continue; + } else { + q = v.search(a, b, m), m && g && (m.enrich = g); } } + E = q && q.length; + if (l && E) { + m = []; + v = 0; + for (let K = 0, L, xa; K < l.length; K += 2) { + L = this.tag.get(l[K]); + if (!L) { + if (console.warn("Tag '" + l[K] + ":" + l[K + 1] + "' will be skipped because there is no field '" + l[K] + "'."), k) { + continue; + } else { + return d; + } + } + if (xa = (L = L && L.get(l[K + 1])) && L.length) { + v++, m.push(L); + } else if (!k) { + return d; + } + } + if (v) { + q = qa(q, m); + E = q.length; + if (!E && !k) { + return d; + } + v--; + } + } + if (E) { + f[r] = D, d.push(q), r++; + } else if (1 === n.length) { + return d; + } } - if (r) { - const l = this; - return new Promise(function(p) { - Promise.all(r).then(function(A) { - p(l.search(a, b, c, A)); - }); + if (p) { + const w = this; + return Promise.all(p).then(function(q) { + return q.length ? w.search(a, b, c, q) : q; }); } - if (!q) { - return []; + if (!r) { + return d; } - if (g && (!h || !this.store)) { + if (t && (!g || !this.store)) { return d[0]; } - for (let l = 0, p; l < f.length; l++) { - p = d[l]; - p.length && h && (p = na.call(this, p)); - if (g) { - return p; + p = []; + for (let w = 0, q; w < f.length; w++) { + q = d[w]; + g && q.length && !q[0].doc && q.length && (q = ta.call(this, q)); + if (t) { + return q; } - d[l] = {field:f[l], result:p}; + d[w] = {field:f[w], result:q}; } - return d; + return h ? ua(d, b) : d; }; -function ma(a, b, c, e) { - let d = this.J[a], f = d && d.length - c; - if (f && 0 < f) { - if (f > b || c) { - d = d.slice(c, c + b); - } - e && (d = na.call(this, d)); - return {tag:a, result:d}; - } -} -function na(a) { - const b = Array(a.length); - for (let c = 0, e; c < a.length; c++) { - e = a[c], b[c] = {id:e, doc:this.store[e]}; - } - return b; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.get = function(a) { - return this.store[a]; -}; -t.set = function(a, b) { - this.store[a] = b; - return this; -}; -O(W.prototype); -var pa = {encode:oa, B:!1, C:""}; -const qa = [I("[\u00e0\u00e1\u00e2\u00e3\u00e4\u00e5]"), "a", I("[\u00e8\u00e9\u00ea\u00eb]"), "e", I("[\u00ec\u00ed\u00ee\u00ef]"), "i", I("[\u00f2\u00f3\u00f4\u00f5\u00f6\u0151]"), "o", I("[\u00f9\u00fa\u00fb\u00fc\u0171]"), "u", I("[\u00fd\u0177\u00ff]"), "y", I("\u00f1"), "n", I("[\u00e7c]"), "k", I("\u00df"), "s", I(" & "), " and "]; -function oa(a) { - var b = a = "" + a; - b.normalize && (b = b.normalize("NFD").replace(ca, "")); - return E.call(this, b.toLowerCase(), !a.normalize && qa); -} -;var sa = {encode:ra, B:!1, C:"strict"}; -const ta = /[^a-z0-9]+/, ua = {b:"p", v:"f", w:"f", z:"s", x:"s", "\u00df":"s", d:"t", n:"m", c:"k", g:"k", j:"k", q:"k", i:"e", y:"e", u:"o"}; -function ra(a) { - a = oa.call(this, a).join(" "); - const b = []; - if (a) { - const c = a.split(ta), e = c.length; - for (let d = 0, f, g = 0; d < e; d++) { - if ((a = c[d]) && (!this.filter || !this.filter[a])) { - f = a[0]; - let h = ua[f] || f, k = h; - for (let m = 1; m < a.length; m++) { - f = a[m]; - const n = ua[f] || f; - n && n !== k && (h += n, k = n); +function ua(a, b) { + const c = [], e = y(); + for (let d = 0, f, g; d < a.length; d++) { + f = a[d]; + g = f.result; + for (let h = 0, k, n, l; h < g.length; h++) { + if (n = g[h], k = n.id, l = e[k]) { + l.push(f.field); + } else { + if (c.length === b) { + return c; } - b[g++] = h; + n.field = e[k] = [f.field]; + c.push(n); } } } + return c; +} +function sa(a, b, c, e, d) { + let f = this.tag.get(a); + if (!f) { + return console.warn("Tag '" + a + "' was not found"), []; + } + if ((a = (f = f && f.get(b)) && f.length - e) && 0 < a) { + if (a > c || e) { + f = f.slice(e, e + c); + } + d && (f = ta.call(this, f)); + return f; + } +} +function ta(a) { + const b = Array(a.length); + for (let c = 0, e; c < a.length; c++) { + e = a[c], b[c] = {id:e, doc:this.store.get(e)}; + } return b; } -;var wa = {encode:va, B:!1, C:""}; -const xa = [I("ae"), "a", I("oe"), "o", I("sh"), "s", I("th"), "t", I("ph"), "f", I("pf"), "f", I("(?![aeo])h(?![aeo])"), "", I("(?!^[aeo])h(?!^[aeo])"), ""]; -function va(a, b) { - a && (a = ra.call(this, a).join(" "), 2 < a.length && (a = F(a, xa)), b || (1 < a.length && (a = J(a)), a && (a = a.split(" ")))); - return a || []; +;function V(a) { + if (!(this instanceof V)) { + return new V(a); + } + const b = a.document || a.doc || a; + var c; + this.L = []; + this.field = []; + this.O = []; + this.key = (c = b.key || b.id) && Y(c, this.O) || "id"; + this.h = (this.fastupdate = !!a.fastupdate) ? new Map() : new Set(); + this.G = (c = b.store || null) && !0 !== c && []; + this.store = c && new Map(); + this.async = !1; + c = new Map(); + let e = b.index || b.field || b; + z(e) && (e = [e]); + for (let d = 0, f, g; d < e.length; d++) { + f = e[d], z(f) || (g = f, f = f.field), g = B(g) ? Object.assign({}, a, g) : a, c.set(f, new N(g, this.h)), g.N ? this.L[d] = g.N : (this.L[d] = Y(f, this.O), g.filter && ("string" === typeof this.L[d] && (this.L[d] = new String(this.L[d])), this.L[d].T = g.filter)), this.field[d] = f; + } + if (this.G) { + a = b.store; + z(a) && (a = [a]); + for (let d = 0, f, g; d < a.length; d++) { + f = a[d], g = f.field || f, f.N ? (this.G[d] = f.N, f.N.X = g) : (this.G[d] = Y(g, this.O), f.filter && ("string" === typeof this.G[d] && (this.G[d] = new String(this.G[d])), this.G[d].T = f.filter)); + } + } + this.index = c; } -;var za = {encode:ya, B:!1, C:""}; -const Aa = I("(?!\\b)[aeo]"); -function ya(a) { - a && (a = va.call(this, a, !0), 1 < a.length && (a = a.replace(Aa, "")), 1 < a.length && (a = J(a)), a && (a = a.split(" "))); - return a || []; +function Y(a, b) { + const c = a.split(":"); + let e = 0; + for (let d = 0; d < c.length; d++) { + a = c[d], "]" === a[a.length - 1] && (a = a.substring(0, a.length - 2)) && (b[e] = !0), a && (c[e++] = a); + } + e < c.length && (c.length = e); + return 1 < e ? c : c[0]; } -;M["latin:default"] = da; -M["latin:simple"] = pa; -M["latin:balance"] = sa; -M["latin:advanced"] = wa; -M["latin:extra"] = za; -export default {Index:S, Document:W, Worker:null, registerCharset:function(a, b) { - M[a] = b; -}, registerLanguage:function(a, b) { - L[a] = b; +u = V.prototype; +u.append = function(a, b) { + return this.add(a, b, !0); +}; +u.update = function(a, b) { + return this.remove(a).add(a, b); +}; +u.remove = function(a) { + B(a) && (a = ba(a, this.key)); + for (const b of this.index.values()) { + b.remove(a, !0); + } + this.h.has(a) && (this.store && this.store.delete(a), this.h.delete(a)); + return this; +}; +u.clear = function() { + for (const a of this.index.values()) { + a.clear(); + } + this.store && this.store.clear(); + return this; +}; +u.contain = function(a) { + return this.h.has(a); +}; +u.cleanup = function() { + for (const a of this.index.values()) { + a.cleanup(); + } + return this; +}; +u.get = function(a) { + return this.store.get(a); +}; +u.set = function(a, b) { + this.store.set(a, b); + return this; +}; +u.export = function(a, b, c, e, d, f) { + let g; + "undefined" === typeof f && (g = new Promise(k => { + f = k; + })); + d || (d = 0); + e || (e = 0); + if (e < this.field.length) { + c = this.field[e]; + var h = this.index[c]; + b = this; + h.export(a, b, d ? c : "", e, d++, f) || (e++, b.export(a, b, c, e, 1, f)); + } else { + switch(d) { + case 1: + b = "tag"; + h = this.B; + c = null; + break; + case 2: + b = "store"; + h = this.store; + c = null; + break; + default: + f(); + return; + } + ka(a, this, c, b, e, d, h, f); + } + return g; +}; +u.import = function(a, b) { + if (b) { + switch(z(b) && (b = JSON.parse(b)), a) { + case "tag": + this.B = b; + break; + case "reg": + this.fastupdate = !1; + this.h = b; + for (let e = 0, d; e < this.field.length; e++) { + d = this.index[this.field[e]], d.h = b, d.fastupdate = !1; + } + break; + case "store": + this.store = b; + break; + default: + a = a.split("."); + const c = a[0]; + a = a[1]; + c && a && this.index[c].import(a, b); + } + } +}; +oa(V.prototype); +const Z = new Map([["b", "p"], ["v", "f"], ["w", "f"], ["z", "s"], ["x", "s"], ["d", "t"], ["n", "m"], ["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], ["i", "e"], ["y", "e"], ["u", "o"]]); +var va = {normalize:!0, C:!0, D:Z}; +const wa = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]), ya = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"]; +var za = {normalize:!0, C:!0, D:Z, I:ya, H:wa}; +var Aa = {normalize:!0, C:!0, D:Z, I:ya.concat([/(?!^)[aeoy]/g, ""]), H:wa}; +const Ba = {a:"", e:"", i:"", o:"", u:"", y:"", b:1, f:1, p:1, v:1, c:2, g:2, j:2, k:2, q:2, s:2, x:2, z:2, "\u00df":2, d:3, t:3, l:4, m:5, n:5, r:6}; +J["latin:exact"] = {normalize:!1, C:!1}; +J["latin:default"] = ma; +J["latin:simple"] = {normalize:!0, C:!0}; +J["latin:balance"] = va; +J["latin:advanced"] = za; +J["latin:extra"] = Aa; +J["latin:soundex"] = {normalize:!0, C:!1, U:{Y:!0}, M:function(a) { + for (let c = 0; c < a.length; c++) { + var b = a[c]; + let e = b.charAt(0), d = Ba[e]; + for (let f = 1, g; f < b.length && (g = b.charAt(f), "h" === g || "w" === g || !(g = Ba[g]) || g === d || (e += g, d = g, 4 !== e.length)); f++) { + } + a[c] = e; + } }}; -}(this)); +export default {Index:N, Encoder:G, Charset:J, Language:la, Document:V, Worker:null, Resolver:null, IndexedDB:null}; + +export const Index=N;export const Encoder=G;export const Charset=J;export const Language=la;export const Document=V;export const Worker=null;export const Resolver=null;export const IndexedDB=null; \ No newline at end of file diff --git a/dist/flexsearch.compact.module.min.js b/dist/flexsearch.compact.module.min.js index 8b69f69..0446e87 100644 --- a/dist/flexsearch.compact.module.min.js +++ b/dist/flexsearch.compact.module.min.js @@ -1,27 +1,48 @@ /**! - * FlexSearch.js v0.7.41 (Compact.module) + * FlexSearch.js v0.8.0 (Bundle) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -(function(self){'use strict';var t;function v(a){return"undefined"!==typeof a?a:!0}function w(a){const b=Array(a);for(let c=0;c=this.m&&(u||!n[l])){var f=T(q,e,r),g="";switch(this.C){case "full":if(2f;h--)if(h-f>=this.m){var k=T(q,e,r,d,f);g=l.substring(f,h);U(this,n,g,k,a,c)}break}case "reverse":if(1=this.m&&U(this,n, -g,T(q,e,r,d,h),a,c);g=""}case "forward":if(1=this.m&&U(this,n,g,f,a,c);break}default:if(this.F&&(f=Math.min(f/this.F(b,l,r)|0,q-1)),U(this,n,l,f,a,c),u&&1=this.m&&!d[l]){d[l]=1;const p=this.h&&l>f;U(this,m,p?f:l,T(g+(e/2>g?0:1),e,r,h-1,k-1),a,c,p?l:f)}}}}this.D||(this.register[a]=1)}}return this}; -function T(a,b,c,e,d){return c&&1=this.m&&!c[q])if(this.s||f||this.map[q])k[u++]=q,c[q]=1;else return e;a=k;d=a.length}if(!d)return e;b||(b=100);h=this.depth&&1=e)))break;if(n){if(f)return ja(k,e,0);b[b.length]=k;return}}return!c&&k}function ja(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function ka(a,b,c,e){c?(e=e&&b>c,a=(a=a[e?b:c])&&a[e?c:b]):a=a[b];return a}t.contain=function(a){return!!this.register[a]};t.update=function(a,b){return this.remove(a).add(a,b)};t.remove=function(a,b){const c=this.register[a];if(c){if(this.D)for(let e=0,d;eb||c)d=d.slice(c,c+b);e&&(d=na.call(this,d));return{tag:a,result:d}}}function na(a){const b=Array(a.length);for(let c=0,e;cthis.K.get(h)),g=1);this.H&&1this.H.get(h)),g=1);f&&g&&(f.lengthf;h--){g=m.substring(f,h);var k=this.score?this.score(b,m,p,g,f):N(t,e,p,d,f);O(this,l,g,k,a,c)}break}case "reverse":if(1< +d){for(h=d-1;0g?0:1),e,p,h-1,k-1),C=this.bidirectional&&m>f;O(this,n,C?f:m,v,a,c,C?m:f)}}}}this.fastupdate||this.h.add(a)}}return this}; +function O(a,b,c,e,d,f,g){let h=g?a.J:a.map,k;b[c]&&g&&(k=b[c])[g]||(g?(b=k||(b[c]=y()),b[g]=1,(k=h.get(g))?h=k:h.set(g,h=new Map)):b[c]=1,(k=h.get(c))?h=k:h.set(c,h=[]),h=h[e]||(h[e]=[]),f&&h.includes(d)||(h.push(d),a.fastupdate&&((b=a.h.get(d))?b.push(h):a.h.set(d,[h]))))}function N(a,b,c,e,d){return c&&1b?b?a.slice(c,c+b):a.slice(c):a;let e=[];for(let d=0,f,g;d=g){c-=g;continue}cb&&(f=f.slice(0,b),g=f.length),e.push(f);else{if(g>=b)return g>b&&(f=f.slice(0,b)),f;e=[f]}b-=g;if(!b)break}return e.length?e=1=d)))break;if(h.length){if(g)return P(h,d,0);b.push(h);return}}return!c&&h}function S(a,b,c){let e;c&&(e=a.bidirectional&&b>c);a=c?(a=a.J.get(e?b:c))&&a.get(e?c:b):a.map.get(b);return a};M.prototype.remove=function(a,b){const c=this.h.size&&(this.fastupdate?this.h.get(a):this.h.has(a));if(c){if(this.fastupdate)for(let e=0,d;ed.length)d.pop();else{const f=d.indexOf(a);f===c.length-1?d.pop():d.splice(f,1)}}else T(this.map,a),this.depth&&T(this.J,a);b||this.h.delete(a)}return this}; +function T(a,b){let c=0;if(a.constructor===Array)for(let e=0,d,f;ec.add(a,b)):this.add(a,b)}return this.remove(a).add(a,b)}; +function U(a){let b=0;if(a.constructor===Array)for(let c=0,e;c{f=n}));let h,k;switch(d||(d=0)){case 0:h="reg";if(this.fastupdate){k=y();for(let n of this.h.keys())k[n]=1}else k=this.h;break;case 1:h="cfg";k={doc:0,opt:this.A?1:0};break;case 2:h="map";k=this.map;break;case 3:h="ctx";k=this.J;break;default:"undefined"===typeof c&&f&&f();return}ka(a,b||this,c,h,e,d,k,f);return g}; +u.import=function(a,b){if(b)switch(A(b)&&(b=JSON.parse(b)),a){case "cfg":this.A=!!b.opt;break;case "reg":this.fastupdate=!1;this.h=b;break;case "map":this.map=b;break;case "ctx":this.J=b}};oa(M.prototype);V.prototype.add=function(a,b,c){B(a)&&(b=a,a=ba(b,this.key));if(b&&(a||0===a)){if(!c&&this.h.has(a))return this.update(a,b);for(let d=0,f;dc||e)a=a.slice(e,e+c);d&&(a=ta.call(this,a));return a}} +function ta(a){const b=Array(a.length);for(let c=0,e;c{f=k}));d||(d=0);e||(e=0);if(e>> 0) + "_", e = 0; return b; }); -y("Symbol.iterator", function(a) { +B("Symbol.iterator", function(a) { if (a) { return a; } a = Symbol("Symbol.iterator"); for (var b = "Array Int8Array Uint8Array Uint8ClampedArray Int16Array Uint16Array Int32Array Uint32Array Float32Array Float64Array".split(" "), c = 0; c < b.length; c++) { - var d = x[b[c]]; - "function" === typeof d && "function" != typeof d.prototype[a] && v(d.prototype, a, {configurable:!0, writable:!0, value:function() { - return ca(aa(this)); + var d = z[b[c]]; + "function" === typeof d && "function" != typeof d.prototype[a] && ca(d.prototype, a, {configurable:!0, writable:!0, value:function() { + return va(aa(this)); }}); } return a; }); -function ca(a) { +function va(a) { a = {next:a}; a[Symbol.iterator] = function() { return this; }; return a; } -function da(a, b) { - a instanceof String && (a += ""); - var c = 0, d = !1, e = {next:function() { - if (!d && c < a.length) { - var g = c++; - return {value:b(g, a[g]), done:!1}; - } - d = !0; - return {done:!0, value:void 0}; - }}; - e[Symbol.iterator] = function() { - return e; - }; - return e; -} -y("Array.prototype.keys", function(a) { - return a ? a : function() { - return da(this, function(b) { - return b; - }); - }; -}); -function ea(a) { - var b = "undefined" != typeof Symbol && Symbol.iterator && a[Symbol.iterator]; - if (b) { - return b.call(a); - } - if ("number" == typeof a.length) { - return {next:aa(a)}; - } - throw Error(String(a) + " is not an iterable or ArrayLike"); -} -y("Promise", function(a) { +B("Promise", function(a) { function b(f) { - this.l = 0; - this.m = void 0; + this.A = 0; + this.C = void 0; this.h = []; - this.M = !1; - var h = this.o(); + this.R = !1; + var h = this.D(); try { f(h.resolve, h.reject); } catch (k) { @@ -146,21 +281,21 @@ y("Promise", function(a) { if (a) { return a; } - c.prototype.l = function(f) { + c.prototype.A = function(f) { if (null == this.h) { this.h = []; var h = this; - this.m(function() { - h.D(); + this.C(function() { + h.G(); }); } this.h.push(f); }; - var e = x.setTimeout; - c.prototype.m = function(f) { + var e = z.setTimeout; + c.prototype.C = function(f) { e(f, 0); }; - c.prototype.D = function() { + c.prototype.G = function() { for (; this.h && this.h.length;) { var f = this.h; this.h = []; @@ -170,32 +305,32 @@ y("Promise", function(a) { try { k(); } catch (l) { - this.o(l); + this.D(l); } } } this.h = null; }; - c.prototype.o = function(f) { - this.m(function() { + c.prototype.D = function(f) { + this.C(function() { throw f; }); }; - b.prototype.o = function() { + b.prototype.D = function() { function f(l) { return function(m) { k || (k = !0, l.call(h, m)); }; } var h = this, k = !1; - return {resolve:f(this.S), reject:f(this.D)}; + return {resolve:f(this.wa), reject:f(this.G)}; }; - b.prototype.S = function(f) { + b.prototype.wa = function(f) { if (f === this) { - this.D(new TypeError("A Promise cannot resolve to itself")); + this.G(new TypeError("A Promise cannot resolve to itself")); } else { if (f instanceof b) { - this.U(f); + this.ya(f); } else { a: { switch(typeof f) { @@ -209,72 +344,72 @@ y("Promise", function(a) { h = !1; } } - h ? this.R(f) : this.I(f); + h ? this.va(f) : this.K(f); } } }; - b.prototype.R = function(f) { + b.prototype.va = function(f) { var h = void 0; try { h = f.then; } catch (k) { - this.D(k); + this.G(k); return; } - "function" == typeof h ? this.V(h, f) : this.I(f); + "function" == typeof h ? this.za(h, f) : this.K(f); }; - b.prototype.D = function(f) { - this.N(2, f); + b.prototype.G = function(f) { + this.pa(2, f); }; - b.prototype.I = function(f) { - this.N(1, f); + b.prototype.K = function(f) { + this.pa(1, f); }; - b.prototype.N = function(f, h) { - if (0 != this.l) { - throw Error("Cannot settle(" + f + ", " + h + "): Promise already settled in state" + this.l); + b.prototype.pa = function(f, h) { + if (0 != this.A) { + throw Error("Cannot settle(" + f + ", " + h + "): Promise already settled in state" + this.A); } - this.l = f; - this.m = h; - 2 === this.l && this.T(); - this.O(); + this.A = f; + this.C = h; + 2 === this.A && this.xa(); + this.ta(); }; - b.prototype.T = function() { + b.prototype.xa = function() { var f = this; e(function() { - if (f.P()) { - var h = x.console; - "undefined" !== typeof h && h.error(f.m); + if (f.ua()) { + var h = z.console; + "undefined" !== typeof h && h.error(f.C); } }, 1); }; - b.prototype.P = function() { - if (this.M) { + b.prototype.ua = function() { + if (this.R) { return !1; } - var f = x.CustomEvent, h = x.Event, k = x.dispatchEvent; + var f = z.CustomEvent, h = z.Event, k = z.dispatchEvent; if ("undefined" === typeof k) { return !0; } - "function" === typeof f ? f = new f("unhandledrejection", {cancelable:!0}) : "function" === typeof h ? f = new h("unhandledrejection", {cancelable:!0}) : (f = x.document.createEvent("CustomEvent"), f.initCustomEvent("unhandledrejection", !1, !0, f)); + "function" === typeof f ? f = new f("unhandledrejection", {cancelable:!0}) : "function" === typeof h ? f = new h("unhandledrejection", {cancelable:!0}) : (f = z.document.createEvent("CustomEvent"), f.initCustomEvent("unhandledrejection", !1, !0, f)); f.promise = this; - f.reason = this.m; + f.reason = this.C; return k(f); }; - b.prototype.O = function() { + b.prototype.ta = function() { if (null != this.h) { for (var f = 0; f < this.h.length; ++f) { - g.l(this.h[f]); + g.A(this.h[f]); } this.h = null; } }; var g = new c(); - b.prototype.U = function(f) { - var h = this.o(); - f.J(h.resolve, h.reject); + b.prototype.ya = function(f) { + var h = this.D(); + f.fa(h.resolve, h.reject); }; - b.prototype.V = function(f, h) { - var k = this.o(); + b.prototype.za = function(f, h) { + var k = this.D(); try { f.call(h, k.resolve, k.reject); } catch (l) { @@ -282,41 +417,41 @@ y("Promise", function(a) { } }; b.prototype.then = function(f, h) { - function k(n, q) { - return "function" == typeof n ? function(r) { + function k(p, q) { + return "function" == typeof p ? function(r) { try { - l(n(r)); - } catch (u) { - m(u); + l(p(r)); + } catch (x) { + m(x); } } : q; } - var l, m, p = new b(function(n, q) { - l = n; + var l, m, n = new b(function(p, q) { + l = p; m = q; }); - this.J(k(f, l), k(h, m)); - return p; + this.fa(k(f, l), k(h, m)); + return n; }; b.prototype.catch = function(f) { return this.then(void 0, f); }; - b.prototype.J = function(f, h) { + b.prototype.fa = function(f, h) { function k() { - switch(l.l) { + switch(l.A) { case 1: - f(l.m); + f(l.C); break; case 2: - h(l.m); + h(l.C); break; default: - throw Error("Unexpected state: " + l.l); + throw Error("Unexpected state: " + l.A); } } var l = this; - null == this.h ? g.l(k) : this.h.push(k); - this.M = !0; + null == this.h ? g.A(k) : this.h.push(k); + this.R = !0; }; b.resolve = d; b.reject = function(f) { @@ -326,35 +461,323 @@ y("Promise", function(a) { }; b.race = function(f) { return new b(function(h, k) { - for (var l = ea(f), m = l.next(); !m.done; m = l.next()) { - d(m.value).J(h, k); + for (var l = y(f), m = l.next(); !m.done; m = l.next()) { + d(m.value).fa(h, k); } }); }; b.all = function(f) { - var h = ea(f), k = h.next(); + var h = y(f), k = h.next(); return k.done ? d([]) : new b(function(l, m) { - function p(r) { - return function(u) { - n[r] = u; + function n(r) { + return function(x) { + p[r] = x; q--; - 0 == q && l(n); + 0 == q && l(p); }; } - var n = [], q = 0; + var p = [], q = 0; do { - n.push(void 0), q++, d(k.value).J(p(n.length - 1), m), k = h.next(); + p.push(void 0), q++, d(k.value).fa(n(p.length - 1), m), k = h.next(); } while (!k.done); }); }; return b; }); -y("Object.is", function(a) { +function wa(a, b) { + a instanceof String && (a += ""); + var c = 0, d = !1, e = {next:function() { + if (!d && c < a.length) { + var g = c++; + return {value:b(g, a[g]), done:!1}; + } + d = !0; + return {done:!0, value:void 0}; + }}; + e[Symbol.iterator] = function() { + return e; + }; + return e; +} +B("Array.prototype.values", function(a) { + return a ? a : function() { + return wa(this, function(b, c) { + return c; + }); + }; +}); +function G(a, b) { + return Object.prototype.hasOwnProperty.call(a, b); +} +B("WeakMap", function(a) { + function b(k) { + this.h = (h += Math.random() + 1).toString(); + if (k) { + k = y(k); + for (var l; !(l = k.next()).done;) { + l = l.value, this.set(l[0], l[1]); + } + } + } + function c() { + } + function d(k) { + var l = typeof k; + return "object" === l && null !== k || "function" === l; + } + function e(k) { + if (!G(k, f)) { + var l = new c(); + ca(k, f, {value:l}); + } + } + function g(k) { + var l = Object[k]; + l && (Object[k] = function(m) { + if (m instanceof c) { + return m; + } + Object.isExtensible(m) && e(m); + return l(m); + }); + } + if (function() { + if (!a || !Object.seal) { + return !1; + } + try { + var k = Object.seal({}), l = Object.seal({}), m = new a([[k, 2], [l, 3]]); + if (2 != m.get(k) || 3 != m.get(l)) { + return !1; + } + m.delete(k); + m.set(l, 4); + return !m.has(k) && 4 == m.get(l); + } catch (n) { + return !1; + } + }()) { + return a; + } + var f = "$jscomp_hidden_" + Math.random(); + g("freeze"); + g("preventExtensions"); + g("seal"); + var h = 0; + b.prototype.set = function(k, l) { + if (!d(k)) { + throw Error("Invalid WeakMap key"); + } + e(k); + if (!G(k, f)) { + throw Error("WeakMap key fail: " + k); + } + k[f][this.h] = l; + return this; + }; + b.prototype.get = function(k) { + return d(k) && G(k, f) ? k[f][this.h] : void 0; + }; + b.prototype.has = function(k) { + return d(k) && G(k, f) && G(k[f], this.h); + }; + b.prototype.delete = function(k) { + return d(k) && G(k, f) && G(k[f], this.h) ? delete k[f][this.h] : !1; + }; + return b; +}); +B("Map", function(a) { + function b() { + var h = {}; + return h.O = h.next = h.head = h; + } + function c(h, k) { + var l = h[1]; + return va(function() { + if (l) { + for (; l.head != h[1];) { + l = l.O; + } + for (; l.next != l.head;) { + return l = l.next, {done:!1, value:k(l)}; + } + l = null; + } + return {done:!0, value:void 0}; + }); + } + function d(h, k) { + var l = k && typeof k; + "object" == l || "function" == l ? g.has(k) ? l = g.get(k) : (l = "" + ++f, g.set(k, l)) : l = "p_" + k; + var m = h[0][l]; + if (m && G(h[0], l)) { + for (h = 0; h < m.length; h++) { + var n = m[h]; + if (k !== k && n.key !== n.key || k === n.key) { + return {id:l, list:m, index:h, F:n}; + } + } + } + return {id:l, list:m, index:-1, F:void 0}; + } + function e(h) { + this[0] = {}; + this[1] = b(); + this.size = 0; + if (h) { + h = y(h); + for (var k; !(k = h.next()).done;) { + k = k.value, this.set(k[0], k[1]); + } + } + } + if (function() { + if (!a || "function" != typeof a || !a.prototype.entries || "function" != typeof Object.seal) { + return !1; + } + try { + var h = Object.seal({x:4}), k = new a(y([[h, "s"]])); + if ("s" != k.get(h) || 1 != k.size || k.get({x:4}) || k.set({x:4}, "t") != k || 2 != k.size) { + return !1; + } + var l = k.entries(), m = l.next(); + if (m.done || m.value[0] != h || "s" != m.value[1]) { + return !1; + } + m = l.next(); + return m.done || 4 != m.value[0].x || "t" != m.value[1] || !l.next().done ? !1 : !0; + } catch (n) { + return !1; + } + }()) { + return a; + } + var g = new WeakMap(); + e.prototype.set = function(h, k) { + h = 0 === h ? 0 : h; + var l = d(this, h); + l.list || (l.list = this[0][l.id] = []); + l.F ? l.F.value = k : (l.F = {next:this[1], O:this[1].O, head:this[1], key:h, value:k}, l.list.push(l.F), this[1].O.next = l.F, this[1].O = l.F, this.size++); + return this; + }; + e.prototype.delete = function(h) { + h = d(this, h); + return h.F && h.list ? (h.list.splice(h.index, 1), h.list.length || delete this[0][h.id], h.F.O.next = h.F.next, h.F.next.O = h.F.O, h.F.head = null, this.size--, !0) : !1; + }; + e.prototype.clear = function() { + this[0] = {}; + this[1] = this[1].O = b(); + this.size = 0; + }; + e.prototype.has = function(h) { + return !!d(this, h).F; + }; + e.prototype.get = function(h) { + return (h = d(this, h).F) && h.value; + }; + e.prototype.entries = function() { + return c(this, function(h) { + return [h.key, h.value]; + }); + }; + e.prototype.keys = function() { + return c(this, function(h) { + return h.key; + }); + }; + e.prototype.values = function() { + return c(this, function(h) { + return h.value; + }); + }; + e.prototype.forEach = function(h, k) { + for (var l = this.entries(), m; !(m = l.next()).done;) { + m = m.value, h.call(k, m[1], m[0], this); + } + }; + e.prototype[Symbol.iterator] = e.prototype.entries; + var f = 0; + return e; +}); +B("Array.prototype.keys", function(a) { + return a ? a : function() { + return wa(this, function(b) { + return b; + }); + }; +}); +B("Set", function(a) { + function b(c) { + this.h = new Map(); + if (c) { + c = y(c); + for (var d; !(d = c.next()).done;) { + this.add(d.value); + } + } + this.size = this.h.size; + } + if (function() { + if (!a || "function" != typeof a || !a.prototype.entries || "function" != typeof Object.seal) { + return !1; + } + try { + var c = Object.seal({x:4}), d = new a(y([c])); + if (!d.has(c) || 1 != d.size || d.add(c) != d || 1 != d.size || d.add({x:4}) != d || 2 != d.size) { + return !1; + } + var e = d.entries(), g = e.next(); + if (g.done || g.value[0] != c || g.value[1] != c) { + return !1; + } + g = e.next(); + return g.done || g.value[0] == c || 4 != g.value[0].x || g.value[1] != g.value[0] ? !1 : e.next().done; + } catch (f) { + return !1; + } + }()) { + return a; + } + b.prototype.add = function(c) { + c = 0 === c ? 0 : c; + this.h.set(c, c); + this.size = this.h.size; + return this; + }; + b.prototype.delete = function(c) { + c = this.h.delete(c); + this.size = this.h.size; + return c; + }; + b.prototype.clear = function() { + this.h.clear(); + this.size = 0; + }; + b.prototype.has = function(c) { + return this.h.has(c); + }; + b.prototype.entries = function() { + return this.h.entries(); + }; + b.prototype.values = function() { + return this.h.values(); + }; + b.prototype.keys = b.prototype.values; + b.prototype[Symbol.iterator] = b.prototype.values; + b.prototype.forEach = function(c, d) { + var e = this; + this.h.forEach(function(g) { + return c.call(d, g, g, e); + }); + }; + return b; +}); +B("Object.is", function(a) { return a ? a : function(b, c) { return b === c ? 0 !== b || 1 / b === 1 / c : b !== b && c !== c; }; }); -y("Array.prototype.includes", function(a) { +B("Array.prototype.includes", function(a) { return a ? a : function(b, c) { var d = this; d instanceof String && (d = String(d)); @@ -369,7 +792,7 @@ y("Array.prototype.includes", function(a) { return !1; }; }); -y("String.prototype.includes", function(a) { +B("String.prototype.includes", function(a) { return a ? a : function(b, c) { if (null == this) { throw new TypeError("The 'this' value for String.prototype.includes must not be null or undefined"); @@ -380,645 +803,98 @@ y("String.prototype.includes", function(a) { return -1 !== this.indexOf(b, c || 0); }; }); -var fa = "function" == typeof Object.assign ? Object.assign : function(a, b) { +B("Array.prototype.entries", function(a) { + return a ? a : function() { + return wa(this, function(b, c) { + return [b, c]; + }); + }; +}); +var xa = "function" == typeof Object.assign ? Object.assign : function(a, b) { for (var c = 1; c < arguments.length; c++) { var d = arguments[c]; if (d) { for (var e in d) { - Object.prototype.hasOwnProperty.call(d, e) && (a[e] = d[e]); + G(d, e) && (a[e] = d[e]); } } } return a; }; -y("Object.assign", function(a) { - return a || fa; +B("Object.assign", function(a) { + return a || xa; }); -function C(a) { - return "undefined" !== typeof a ? a : !0; -} -function ha(a) { - for (var b = Array(a), c = 0; c < a; c++) { - b[c] = D(); +B("Array.prototype.flat", function(a) { + return a ? a : function(b) { + b = void 0 === b ? 1 : b; + var c = []; + Array.prototype.forEach.call(this, function(d) { + Array.isArray(d) && 0 < b ? (d = Array.prototype.flat.call(d, b - 1), c.push.apply(c, d)) : c.push(d); + }); + return c; + }; +}); +function H(a, b, c) { + var d = typeof c, e = typeof a; + if ("undefined" !== d) { + if ("undefined" !== e) { + if (c) { + if ("function" === e && d === e) { + return function(g) { + return a(c(g)); + }; + } + b = a.constructor; + if (b === c.constructor) { + if (b === Array) { + return c.concat(a); + } + if (b === Map) { + b = new Map(c); + d = y(a); + for (e = d.next(); !e.done; e = d.next()) { + e = e.value, b.set(e[0], e[1]); + } + return b; + } + if (b === Set) { + b = new Set(c); + d = y(a.values()); + for (e = d.next(); !e.done; e = d.next()) { + b.add(e.value); + } + return b; + } + } + } + return a; + } + return c; } - return b; + return "undefined" === e ? b : a; } -function D() { +function I() { return Object.create(null); } -function ia(a, b) { +function ya(a, b) { return b.length - a.length; } -function E(a) { +function J(a) { return "string" === typeof a; } -function F(a) { +function K(a) { return "object" === typeof a; } -function G(a) { - return "function" === typeof a; -} -;function ja(a, b) { - var c = ka; - if (a && (b && (a = I(a, b)), this.K && (a = I(a, this.K)), this.L && 1 < a.length && (a = I(a, this.L)), c || "" === c)) { - a = a.split(c); - if (this.filter) { - b = this.filter; - c = a.length; - for (var d = [], e = 0, g = 0; e < c; e++) { - var f = a[e]; - f && !b[f] && (d[g++] = f); - } - a = d; - } - return a; - } - return a; -} -var ka = /[\p{Z}\p{S}\p{P}\p{C}]+/u, la = /[\u0300-\u036f]/g; -function ma(a, b) { - for (var c = Object.keys(a), d = c.length, e = [], g = "", f = 0, h = 0, k, l; h < d; h++) { - k = c[h], (l = a[k]) ? (e[f++] = J(b ? "(?!\\b)" + k + "(\\b|_)" : k), e[f++] = l) : g += (g ? "|" : "") + k; - } - g && (e[f++] = J(b ? "(?!\\b)(" + g + ")(\\b|_)" : "(" + g + ")"), e[f] = ""); - return e; -} -function I(a, b) { - for (var c = 0, d = b.length; c < d && (a = a.replace(b[c], b[c + 1]), a); c += 2) { - } - return a; -} -function J(a) { - return new RegExp(a, "g"); -} -function na(a) { - for (var b = "", c = "", d = 0, e = a.length, g = void 0; d < e; d++) { - (g = a[d]) !== c && (b += c = g); +function za(a) { + var b = []; + a = y(a.keys()); + for (var c = a.next(); !c.done; c = a.next()) { + b.push(c.value); } return b; } -;var pa = {encode:oa, G:!1, H:""}; -function oa(a) { - return ja.call(this, ("" + a).toLowerCase(), !1); -} -;var qa = {}, K = {}; -function ra(a) { - L(a, "add"); - L(a, "append"); - L(a, "search"); - L(a, "update"); - L(a, "remove"); -} -function L(a, b) { - a[b + "Async"] = function() { - var c = this, d = arguments, e = d[d.length - 1]; - if (G(e)) { - var g = e; - delete d[d.length - 1]; - } - e = new Promise(function(f) { - setTimeout(function() { - c.async = !0; - var h = c[b].apply(c, d); - c.async = !1; - f(h); - }); - }); - return g ? (e.then(g), this) : e; - }; -} -;function sa(a, b, c, d) { - var e = a.length, g = [], f, h = 0; - d && (d = []); - for (var k = e - 1; 0 <= k; k--) { - for (var l = a[k], m = l.length, p = D(), n = !B, q = 0; q < m; q++) { - var r = l[q], u = r.length; - if (u) { - for (var A = 0, w, z; A < u; A++) { - if (z = r[A], B) { - if (B[z]) { - if (!k) { - if (c) { - c--; - } else { - if (g[h++] = z, h === b) { - return g; - } - } - } - if (k || d) { - p[z] = 1; - } - n = !0; - } - d && (w = (f[z] || 0) + 1, f[z] = w, w < e && (w = d[w - 2] || (d[w - 2] = []), w[w.length] = z)); - } else { - p[z] = 1; - } - } - } - } - if (d) { - B || (f = p); - } else if (!n) { - return []; - } - var B = p; - } - if (d) { - for (a = d.length - 1; 0 <= a; a--) { - for (e = d[a], f = e.length, k = 0; k < f; k++) { - if (l = e[k], !B[l]) { - if (c) { - c--; - } else { - if (g[h++] = l, h === b) { - return g; - } - } - B[l] = 1; - } - } - } - } - return g; -} -function ta(a, b) { - for (var c = D(), d = D(), e = [], g = 0; g < a.length; g++) { - c[a[g]] = 1; - } - for (a = 0; a < b.length; a++) { - g = b[a]; - for (var f = 0, h; f < g.length; f++) { - h = g[f], c[h] && !d[h] && (d[h] = 1, e[e.length] = h); - } - } - return e; -} -;function M(a) { - this.l = !0 !== a && a; - this.cache = D(); - this.h = []; -} -function ua(a, b, c) { - F(a) && (a = a.query); - var d = this.cache.get(a); - d || (d = this.search(a, b, c), this.cache.set(a, d)); - return d; -} -M.prototype.set = function(a, b) { - if (!this.cache[a]) { - var c = this.h.length; - c === this.l ? delete this.cache[this.h[c - 1]] : c++; - for (--c; 0 < c; c--) { - this.h[c] = this.h[c - 1]; - } - this.h[0] = a; - } - this.cache[a] = b; -}; -M.prototype.get = function(a) { - var b = this.cache[a]; - if (this.l && b && (a = this.h.indexOf(a))) { - var c = this.h[a - 1]; - this.h[a - 1] = this.h[a]; - this.h[a] = c; - } - return b; -}; -var va = {memory:{charset:"latin:extra", F:3, C:4, s:!1}, performance:{F:3, C:3, B:!1, context:{depth:2, F:1}}, match:{charset:"latin:extra", H:"reverse"}, score:{charset:"latin:advanced", F:20, C:3, context:{depth:3, F:9}}, "default":{}}; -function wa(a, b, c, d, e, g, f, h) { - setTimeout(function() { - var k = a(c ? c + "." + d : d, JSON.stringify(f)); - k && k.then ? k.then(function() { - b.export(a, b, c, e, g + 1, h); - }) : b.export(a, b, c, e, g + 1, h); - }); -} -;function N(a, b) { - if (!(this instanceof N)) { - return new N(a); - } - var c; - if (a) { - if (E(a)) { - va[a] || console.warn("Preset not found: " + a), a = va[a]; - } else { - if (c = a.preset) { - c[c] || console.warn("Preset not found: " + c), a = Object.assign({}, c[c], a); - } - } - c = a.charset; - var d = a.lang; - E(c) && (-1 === c.indexOf(":") && (c += ":default"), c = K[c]); - E(d) && (d = qa[d]); - } else { - a = {}; - } - var e, g, f = a.context || {}; - this.encode = a.encode || c && c.encode || oa; - this.register = b || D(); - this.F = e = a.resolution || 9; - this.H = b = c && c.H || a.tokenize || "strict"; - this.depth = "strict" === b && f.depth; - this.l = C(f.bidirectional); - this.B = g = C(a.optimize); - this.s = C(a.fastupdate); - this.C = a.minlength || 1; - this.o = a.boost; - this.map = g ? ha(e) : D(); - this.m = e = f.resolution || 1; - this.h = g ? ha(e) : D(); - this.G = c && c.G || a.rtl; - this.K = (b = a.matcher || d && d.K) && ma(b, !1); - this.L = (b = a.stemmer || d && d.L) && ma(b, !0); - if (c = b = a.filter || d && d.filter) { - c = b; - d = D(); - f = 0; - for (e = c.length; f < e; f++) { - d[c[f]] = 1; - } - c = d; - } - this.filter = c; - this.cache = (b = a.cache) && new M(b); -} -t = N.prototype; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.add = function(a, b, c, d) { - if (b && (a || 0 === a)) { - if (!d && !c && this.register[a]) { - return this.update(a, b); - } - b = this.encode(b); - if (d = b.length) { - for (var e = D(), g = D(), f = this.depth, h = this.F, k = 0; k < d; k++) { - var l = b[this.G ? d - 1 - k : k], m = l.length; - if (l && m >= this.C && (f || !g[l])) { - var p = O(h, d, k), n = ""; - switch(this.H) { - case "full": - if (2 < m) { - for (p = 0; p < m; p++) { - for (var q = m; q > p; q--) { - if (q - p >= this.C) { - var r = O(h, d, k, m, p); - n = l.substring(p, q); - P(this, g, n, r, a, c); - } - } - } - break; - } - case "reverse": - if (1 < m) { - for (q = m - 1; 0 < q; q--) { - n = l[q] + n, n.length >= this.C && P(this, g, n, O(h, d, k, m, q), a, c); - } - n = ""; - } - case "forward": - if (1 < m) { - for (q = 0; q < m; q++) { - n += l[q], n.length >= this.C && P(this, g, n, p, a, c); - } - break; - } - default: - if (this.o && (p = Math.min(p / this.o(b, l, k) | 0, h - 1)), P(this, g, l, p, a, c), f && 1 < d && k < d - 1) { - for (m = D(), n = this.m, p = l, q = Math.min(f + 1, d - k), r = m[p] = 1; r < q; r++) { - if ((l = b[this.G ? d - 1 - k - r : k + r]) && l.length >= this.C && !m[l]) { - m[l] = 1; - var u = this.l && l > p; - P(this, e, u ? p : l, O(n + (d / 2 > n ? 0 : 1), d, k, q - 1, r - 1), a, c, u ? l : p); - } - } - } - } - } - } - this.s || (this.register[a] = 1); - } - } - return this; -}; -function O(a, b, c, d, e) { - return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; -} -function P(a, b, c, d, e, g, f) { - var h = f ? a.h : a.map; - if (!b[c] || f && !b[c][f]) { - a.B && (h = h[d]), f ? (b = b[c] || (b[c] = D()), b[f] = 1, h = h[f] || (h[f] = D())) : b[c] = 1, h = h[c] || (h[c] = []), a.B || (h = h[d] || (h[d] = [])), g && h.includes(e) || (h[h.length] = e, a.s && (a = a.register[e] || (a.register[e] = []), a[a.length] = h)); - } -} -t.search = function(a, b, c) { - c || (!b && F(a) ? (c = a, a = c.query) : F(b) && (c = b)); - var d = [], e = 0; - if (c) { - a = c.query || a; - b = c.limit; - e = c.offset || 0; - var g = c.context; - var f = c.suggest; - } - if (a) { - a = this.encode("" + a); - var h = a.length; - if (1 < h) { - c = D(); - for (var k = [], l = 0, m = 0, p; l < h; l++) { - if ((p = a[l]) && p.length >= this.C && !c[p]) { - if (this.B || f || this.map[p]) { - k[m++] = p, c[p] = 1; - } else { - return d; - } - } - } - a = k; - h = a.length; - } - } - if (!h) { - return d; - } - b || (b = 100); - g = this.depth && 1 < h && !1 !== g; - c = 0; - if (g) { - var n = a[0]; - c = 1; - } else { - 1 < h && a.sort(ia); - } - for (; c < h; c++) { - l = a[c]; - g ? (k = xa(this, d, f, b, e, 2 === h, l, n), f && !1 === k && d.length || (n = l)) : k = xa(this, d, f, b, e, 1 === h, l); - if (k) { - return k; - } - if (f && c === h - 1) { - k = d.length; - if (!k) { - if (g) { - g = 0; - c = -1; - continue; - } - return d; - } - if (1 === k) { - return ya(d[0], b, e); - } - } - } - return sa(d, b, e, f); -}; -function xa(a, b, c, d, e, g, f, h) { - var k = [], l = h ? a.h : a.map; - a.B || (l = za(l, f, h, a.l)); - if (l) { - for (var m = 0, p = Math.min(l.length, h ? a.m : a.F), n = 0, q = 0, r, u; n < p; n++) { - if (r = l[n]) { - if (a.B && (r = za(r, f, h, a.l)), e && r && g && (u = r.length, u <= e ? (e -= u, r = null) : (r = r.slice(e), e = 0)), r && (k[m++] = r, g && (q += r.length, q >= d))) { - break; - } - } - } - if (m) { - if (g) { - return ya(k, d, 0); - } - b[b.length] = k; - return; - } - } - return !c && k; -} -function ya(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function za(a, b, c, d) { - c ? (d = d && b > c, a = (a = a[d ? b : c]) && a[d ? c : b]) : a = a[b]; - return a; -} -t.contain = function(a) { - return !!this.register[a]; -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a, b) { - var c = this.register[a]; - if (c) { - if (this.s) { - for (var d = 0, e; d < c.length; d++) { - e = c[d], e.splice(e.indexOf(a), 1); - } - } else { - Q(this.map, a, this.F, this.B), this.depth && Q(this.h, a, this.m, this.B); - } - b || delete this.register[a]; - if (this.cache) { - for (b = this.cache, c = 0; c < b.h.length; c++) { - e = b.h[c], d = b.cache[e], d.includes(a) && (b.h.splice(c--, 1), delete b.cache[e]); - } - } - } - return this; -}; -function Q(a, b, c, d, e) { - var g = 0; - if (a.constructor === Array) { - if (e) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), g++) : g++; - } else { - e = Math.min(a.length, c); - for (var f = 0, h; f < e; f++) { - if (h = a[f]) { - g = Q(h, b, c, d, e), d || g || delete a[f]; - } - } - } - } else { - for (f in a) { - (g = Q(a[f], b, c, d, e)) || delete a[f]; - } - } - return g; -} -t.searchCache = ua; -t.export = function(a, b, c, d, e, g) { - var f = !0; - "undefined" === typeof g && (f = new Promise(function(m) { - g = m; - })); - switch(e || (e = 0)) { - case 0: - var h = "reg"; - if (this.s) { - var k = D(); - for (var l in this.register) { - k[l] = 1; - } - } else { - k = this.register; - } - break; - case 1: - h = "cfg"; - k = {doc:0, opt:this.B ? 1 : 0}; - break; - case 2: - h = "map"; - k = this.map; - break; - case 3: - h = "ctx"; - k = this.h; - break; - default: - "undefined" === typeof c && g && g(); - return; - } - wa(a, b || this, c, h, d, e, k, g); - return f; -}; -t.import = function(a, b) { - if (b) { - switch(E(b) && (b = JSON.parse(b)), a) { - case "cfg": - this.B = !!b.opt; - break; - case "reg": - this.s = !1; - this.register = b; - break; - case "map": - this.map = b; - break; - case "ctx": - this.h = b; - } - } -}; -ra(N.prototype); -function Aa(a) { - a = a.data; - var b = self._index, c = a.args, d = a.task; - switch(d) { - case "init": - d = a.options || {}; - a = a.factory; - b = d.encode; - d.cache = !1; - b && 0 === b.indexOf("function") && (d.encode = Function("return " + b)()); - a ? (Function("return " + a)()(self), self._index = new self.FlexSearch.Index(d), delete self.FlexSearch) : self._index = new N(d); - break; - default: - a = a.id, b = b[d].apply(b, c), postMessage("search" === d ? {id:a, msg:b} : {id:a}); - } -} -;var Ba = 0; -function R(a) { - if (!(this instanceof R)) { - return new R(a); - } - var b; - a ? G(b = a.encode) && (a.encode = b.toString()) : a = {}; - (b = (self || window)._factory) && (b = b.toString()); - var c = "undefined" === typeof window && self.exports, d = this; - this.A = Ca(b, c, a.worker); - this.h = D(); - if (this.A) { - if (c) { - this.A.on("message", function(e) { - d.h[e.id](e.msg); - delete d.h[e.id]; - }); - } else { - this.A.onmessage = function(e) { - e = e.data; - d.h[e.id](e.msg); - delete d.h[e.id]; - }; - } - this.A.postMessage({task:"init", factory:b, options:a}); - } -} -S("add"); -S("append"); -S("search"); -S("update"); -S("remove"); -function S(a) { - R.prototype[a] = R.prototype[a + "Async"] = function() { - var b = this, c = [].slice.call(arguments), d = c[c.length - 1]; - if (G(d)) { - var e = d; - c.splice(c.length - 1, 1); - } - d = new Promise(function(g) { - setTimeout(function() { - b.h[++Ba] = g; - b.A.postMessage({task:a, id:Ba, args:c}); - }); - }); - return e ? (d.then(e), this) : d; - }; -} -function Ca(a, b, c) { - try { - var d = b ? new (require("worker_threads")["Worker"])(__dirname + "/node/node.js") : a ? new Worker(URL.createObjectURL(new Blob(["onmessage=" + Aa.toString()], {type:"text/javascript"}))) : new Worker(E(c) ? c : "worker/worker.js", {type:"module"}); - } catch (e) { - } - return d; -} -;function T(a) { - if (!(this instanceof T)) { - return new T(a); - } - var b = a.document || a.doc || a, c; - this.I = []; - this.h = []; - this.m = []; - this.register = D(); - this.key = (c = b.key || b.id) && U(c, this.m) || "id"; - this.s = C(a.fastupdate); - this.o = (c = b.store) && !0 !== c && []; - this.store = c && D(); - this.D = (c = b.tag) && U(c, this.m); - this.l = c && D(); - this.cache = (c = a.cache) && new M(c); - a.cache = !1; - this.A = a.worker; - this.async = !1; - c = D(); - var d = b.index || b.field || b; - E(d) && (d = [d]); - for (var e = 0, g, f = void 0; e < d.length; e++) { - g = d[e], E(g) || (f = g, g = g.field), f = F(f) ? Object.assign({}, a, f) : a, this.A && (c[g] = new R(f), c[g].A || (this.A = !1)), this.A || (c[g] = new N(f, this.register)), this.I[e] = U(g, this.m), this.h[e] = g; - } - if (this.o) { - for (a = b.store, E(a) && (a = [a]), b = 0; b < a.length; b++) { - this.o[b] = U(a[b], this.m); - } - } - this.index = c; -} -function U(a, b) { - for (var c = a.split(":"), d = 0, e = 0; e < c.length; e++) { - a = c[e], 0 <= a.indexOf("[]") && (a = a.substring(0, a.length - 2)) && (b[d] = !0), a && (c[d++] = a); - } - d < c.length && (c.length = d); - return 1 < d ? c : c[0]; -} -function V(a, b) { - if (E(b)) { +function Aa(a, b) { + if (J(b)) { a = a[b]; } else { for (var c = 0; a && c < b.length; c++) { @@ -1027,21 +903,1544 @@ function V(a, b) { } return a; } -function W(a, b, c, d, e) { - a = a[e]; - if (d === c.length - 1) { - b[e] = a; - } else if (a) { - if (a.constructor === Array) { - for (b = b[e] = Array(a.length), e = 0; e < a.length; e++) { - W(a, b, c, d, e); +function Ba(a) { + for (var b = 0, c = 0, d = void 0; c < a.length; c++) { + (d = a[c]) && b < d.length && (b = d.length); + } + return b; +} +;var Ca = /[^\p{L}\p{N}]+/u, Da = /(\d{3})/g, Ea = /(\D)(\d{3})/g, Fa = /(\d{3})(\D)/g, Ga = "".normalize && /[\u0300-\u036f]/g, Ha = !Ga && new Map([["\u00aa", "a"], ["\u00b2", "2"], ["\u00b3", "3"], ["\u00b9", "1"], ["\u00ba", "o"], ["\u00bc", "1\u20444"], ["\u00bd", "1\u20442"], ["\u00be", "3\u20444"], ["\u00e0", "a"], ["\u00e1", "a"], ["\u00e2", "a"], ["\u00e3", "a"], ["\u00e4", "a"], ["\u00e5", "a"], ["\u00e7", "c"], ["\u00e8", "e"], ["\u00e9", "e"], ["\u00ea", "e"], ["\u00eb", "e"], ["\u00ec", +"i"], ["\u00ed", "i"], ["\u00ee", "i"], ["\u00ef", "i"], ["\u00f1", "n"], ["\u00f2", "o"], ["\u00f3", "o"], ["\u00f4", "o"], ["\u00f5", "o"], ["\u00f6", "o"], ["\u00f9", "u"], ["\u00fa", "u"], ["\u00fb", "u"], ["\u00fc", "u"], ["\u00fd", "y"], ["\u00ff", "y"], ["\u0101", "a"], ["\u0103", "a"], ["\u0105", "a"], ["\u0107", "c"], ["\u0109", "c"], ["\u010b", "c"], ["\u010d", "c"], ["\u010f", "d"], ["\u0113", "e"], ["\u0115", "e"], ["\u0117", "e"], ["\u0119", "e"], ["\u011b", "e"], ["\u011d", "g"], ["\u011f", +"g"], ["\u0121", "g"], ["\u0123", "g"], ["\u0125", "h"], ["\u0129", "i"], ["\u012b", "i"], ["\u012d", "i"], ["\u012f", "i"], ["\u0133", "ij"], ["\u0135", "j"], ["\u0137", "k"], ["\u013a", "l"], ["\u013c", "l"], ["\u013e", "l"], ["\u0140", "l"], ["\u0144", "n"], ["\u0146", "n"], ["\u0148", "n"], ["\u0149", "n"], ["\u014d", "o"], ["\u014f", "o"], ["\u0151", "o"], ["\u0155", "r"], ["\u0157", "r"], ["\u0159", "r"], ["\u015b", "s"], ["\u015d", "s"], ["\u015f", "s"], ["\u0161", "s"], ["\u0163", "t"], ["\u0165", +"t"], ["\u0169", "u"], ["\u016b", "u"], ["\u016d", "u"], ["\u016f", "u"], ["\u0171", "u"], ["\u0173", "u"], ["\u0175", "w"], ["\u0177", "y"], ["\u017a", "z"], ["\u017c", "z"], ["\u017e", "z"], ["\u017f", "s"], ["\u01a1", "o"], ["\u01b0", "u"], ["\u01c6", "dz"], ["\u01c9", "lj"], ["\u01cc", "nj"], ["\u01ce", "a"], ["\u01d0", "i"], ["\u01d2", "o"], ["\u01d4", "u"], ["\u01d6", "u"], ["\u01d8", "u"], ["\u01da", "u"], ["\u01dc", "u"], ["\u01df", "a"], ["\u01e1", "a"], ["\u01e3", "ae"], ["\u00e6", "ae"], +["\u01fd", "ae"], ["\u01e7", "g"], ["\u01e9", "k"], ["\u01eb", "o"], ["\u01ed", "o"], ["\u01ef", "\u0292"], ["\u01f0", "j"], ["\u01f3", "dz"], ["\u01f5", "g"], ["\u01f9", "n"], ["\u01fb", "a"], ["\u01ff", "\u00f8"], ["\u0201", "a"], ["\u0203", "a"], ["\u0205", "e"], ["\u0207", "e"], ["\u0209", "i"], ["\u020b", "i"], ["\u020d", "o"], ["\u020f", "o"], ["\u0211", "r"], ["\u0213", "r"], ["\u0215", "u"], ["\u0217", "u"], ["\u0219", "s"], ["\u021b", "t"], ["\u021f", "h"], ["\u0227", "a"], ["\u0229", "e"], +["\u022b", "o"], ["\u022d", "o"], ["\u022f", "o"], ["\u0231", "o"], ["\u0233", "y"], ["\u02b0", "h"], ["\u02b1", "h"], ["\u0266", "h"], ["\u02b2", "j"], ["\u02b3", "r"], ["\u02b4", "\u0279"], ["\u02b5", "\u027b"], ["\u02b6", "\u0281"], ["\u02b7", "w"], ["\u02b8", "y"], ["\u02e0", "\u0263"], ["\u02e1", "l"], ["\u02e2", "s"], ["\u02e3", "x"], ["\u02e4", "\u0295"], ["\u0390", "\u03b9"], ["\u03ac", "\u03b1"], ["\u03ad", "\u03b5"], ["\u03ae", "\u03b7"], ["\u03af", "\u03b9"], ["\u03b0", "\u03c5"], ["\u03ca", +"\u03b9"], ["\u03cb", "\u03c5"], ["\u03cc", "\u03bf"], ["\u03cd", "\u03c5"], ["\u03ce", "\u03c9"], ["\u03d0", "\u03b2"], ["\u03d1", "\u03b8"], ["\u03d2", "\u03a5"], ["\u03d3", "\u03a5"], ["\u03d4", "\u03a5"], ["\u03d5", "\u03c6"], ["\u03d6", "\u03c0"], ["\u03f0", "\u03ba"], ["\u03f1", "\u03c1"], ["\u03f2", "\u03c2"], ["\u03f5", "\u03b5"], ["\u0439", "\u0438"], ["\u0450", "\u0435"], ["\u0451", "\u0435"], ["\u0453", "\u0433"], ["\u0457", "\u0456"], ["\u045c", "\u043a"], ["\u045d", "\u0438"], ["\u045e", +"\u0443"], ["\u0477", "\u0475"], ["\u04c2", "\u0436"], ["\u04d1", "\u0430"], ["\u04d3", "\u0430"], ["\u04d7", "\u0435"], ["\u04db", "\u04d9"], ["\u04dd", "\u0436"], ["\u04df", "\u0437"], ["\u04e3", "\u0438"], ["\u04e5", "\u0438"], ["\u04e7", "\u043e"], ["\u04eb", "\u04e9"], ["\u04ed", "\u044d"], ["\u04ef", "\u0443"], ["\u04f1", "\u0443"], ["\u04f3", "\u0443"], ["\u04f5", "\u0447"]]); +function M(a) { + a = void 0 === a ? {} : a; + if (!(this instanceof M)) { + return new (Function.prototype.bind.apply(M, [null].concat(ba(arguments))))(); + } + for (var b = 0; b < arguments.length; b++) { + this.assign(arguments[b]); + } +} +M.prototype.assign = function(a) { + this.normalize = H(a.normalize, !0, this.normalize); + var b = a.ra, c = b || a.Ha || a.split; + if ("object" === typeof c) { + var d = !b, e = ""; + a.ra || (e += "\\p{Z}"); + c.Ea && (e += "\\p{L}"); + c.Ia && (e += "\\p{N}", d = !!b); + c.Ka && (e += "\\p{S}"); + c.Ja && (e += "\\p{P}"); + c.control && (e += "\\p{C}"); + if (c = c.char) { + e += "object" === typeof c ? c.join("") : c; + } + this.split = new RegExp("[" + (b ? "^" : "") + e + "]+", "u"); + this.numeric = d; + } else { + this.split = H(c, Ca, this.split), this.numeric = H(this.numeric, !0); + } + this.la = H(a.la, null, this.la); + this.ca = H(a.ca, null, this.ca); + this.rtl = a.rtl || !1; + this.H = H(a.H, !0, this.H); + this.filter = H((c = a.filter) && new Set(c), null, this.filter); + this.N = H((c = a.N) && new Map(c), null, this.N); + this.I = H((c = a.I) && new Map(c), null, this.I); + this.U = H((c = a.U) && new Map(c), null, this.U); + this.P = H(a.P, null, this.P); + this.ka = H(a.ka, 1, this.ka); + this.sa = H(a.sa, 0, this.sa); + if (this.cache = c = H(a.cache, !0, this.cache)) { + this.ea = null, this.R = "number" === typeof c ? c : 2e5, this.W = new Map(), this.ba = new Map(), this.A = this.h = 128; + } + this.C = ""; + this.G = null; + this.D = ""; + this.K = null; + if (this.N) { + for (a = y(this.N.keys()), b = a.next(); !b.done; b = a.next()) { + this.C += (this.C ? "|" : "") + b.value; + } + } + if (this.U) { + for (a = y(this.U.keys()), b = a.next(); !b.done; b = a.next()) { + this.D += (this.D ? "|" : "") + b.value; + } + } + return this; +}; +M.prototype.encode = function(a) { + var b = this; + if (this.cache && a.length <= this.h) { + if (this.ea) { + if (this.W.has(a)) { + return this.W.get(a); } } else { - b = b[e] || (b[e] = D()), e = c[++d], W(a, b, c, d, e); + this.ea = setTimeout(Ia, 0, this); + } + } + this.normalize && ("function" === typeof this.normalize ? a = this.normalize(a) : Ga ? a = a.normalize("NFKD").replace(Ga, "").toLowerCase() : (a = a.toLowerCase(), this.I = this.I ? new Map([].concat(ba(Ha), ba(this.I))) : new Map(Ha))); + this.la && (a = this.la(a)); + this.numeric && 3 < a.length && (a = a.replace(Ea, "$1 $2").replace(Fa, "$1 $2").replace(Da, "$1 ")); + for (var c = !(this.H || this.I || this.filter || this.N || this.U || this.P), d = [], e = this.split || "" === this.split ? a.split(this.split) : a, g = 0, f = void 0, h = void 0; g < e.length; g++) { + if ((f = h = e[g]) && !(f.length < this.ka)) { + if (c) { + d.push(f); + } else { + if (!this.filter || !this.filter.has(f)) { + if (this.cache && f.length <= this.A) { + if (this.ea) { + var k = this.ba.get(f); + if (k || "" === k) { + k && d.push(k); + continue; + } + } else { + this.ea = setTimeout(Ia, 0, this); + } + } + k = void 0; + this.U && 2 < f.length && (this.K || (this.K = new RegExp("(?!^)(" + this.D + ")$")), f = f.replace(this.K, function(q) { + return b.U.get(q); + }), k = 1); + this.N && 1 < f.length && (this.G || (this.G = new RegExp("(" + this.C + ")", "g")), f = f.replace(this.G, function(q) { + return b.N.get(q); + }), k = 1); + f && k && (f.length < this.ka || this.filter && this.filter.has(f)) && (f = ""); + if (f && (this.I || this.H && 1 < f.length)) { + k = ""; + for (var l = 0, m = "", n = void 0, p = void 0; l < f.length; l++) { + n = f.charAt(l), n === m && this.H || ((p = this.I && this.I.get(n)) || "" === p ? p === m && this.H || !(m = p) || (k += p) : k += m = n); + } + f = k; + } + if (f && this.P) { + for (k = 0; f && k < this.P.length; k += 2) { + f = f.replace(this.P[k], this.P[k + 1]); + } + } + this.cache && h.length <= this.A && (this.ba.set(h, f), this.ba.size > this.R && (this.ba.clear(), this.A = this.A / 1.1 | 0)); + f && d.push(f); + } + } + } + } + this.ca && (d = this.ca(d) || d); + this.cache && a.length <= this.h && (this.W.set(a, d), this.W.size > this.R && (this.W.clear(), this.h = this.h / 1.1 | 0)); + return d; +}; +function Ia(a) { + a.ea = null; + a.W.clear(); + a.ba.clear(); +} +;function Ja(a, b, c) { + a = ("object" === typeof a ? "" + a.query : a).toLowerCase(); + var d = this.cache.get(a); + if (!d) { + d = this.search(a, b, c); + if (d instanceof Promise) { + var e = this; + d.then(function(g) { + e.cache.set(a, g); + }); + } + this.cache.set(a, d); + } + return d; +} +function N(a) { + this.limit = a && !0 !== a ? a : 1000; + this.cache = new Map(); + this.h = ""; +} +N.prototype.set = function(a, b) { + this.cache.has(a) || (this.cache.set(this.h = a, b), this.limit && this.cache.size > this.limit && this.cache.delete(this.cache.keys().next().value)); +}; +N.prototype.get = function(a) { + var b = this.cache.get(a); + b && this.limit && this.h !== a && (this.cache.delete(a), this.cache.set(this.h = a, b)); + return b; +}; +N.prototype.remove = function(a) { + for (var b = y(this.cache), c = b.next(); !c.done; c = b.next()) { + c = c.value; + var d = c[0]; + c[1].includes(a) && this.cache.delete(d); + } +}; +N.prototype.clear = function() { + this.cache.clear(); + this.h = ""; +}; +function Ka(a, b, c, d) { + for (var e = [], g = 0, f; g < a.index.length; g++) { + if (f = a.index[g], b >= f.length) { + b -= f.length; + } else { + b = f[d ? "splice" : "slice"](b, c); + if (f = b.length) { + if (e = e.length ? e.concat(b) : b, c -= f, d && (a.length -= f), !c) { + break; + } + } + b = 0; + } + } + return e; +} +function Q(a) { + if (!(this instanceof Q)) { + return new Q(a); + } + this.index = a ? [a] : []; + this.length = a ? a.length : 0; + var b = this; + return new Proxy([], {get:function(c, d) { + if ("length" === d) { + return b.length; + } + if ("push" === d) { + return function(e) { + b.index[b.index.length - 1].push(e); + b.length++; + }; + } + if ("pop" === d) { + return function() { + if (b.length) { + return b.length--, b.index[b.index.length - 1].pop(); + } + }; + } + if ("indexOf" === d) { + return function(e) { + for (var g = 0, f = 0, h, k; f < b.index.length; f++) { + h = b.index[f]; + k = h.indexOf(e); + if (0 <= k) { + return g + k; + } + g += h.length; + } + return -1; + }; + } + if ("includes" === d) { + return function(e) { + for (var g = 0; g < b.index.length; g++) { + if (b.index[g].includes(e)) { + return !0; + } + } + return !1; + }; + } + if ("slice" === d) { + return function(e, g) { + return Ka(b, e || 0, g || b.length, !1); + }; + } + if ("splice" === d) { + return function(e, g) { + return Ka(b, e || 0, g || b.length, !0); + }; + } + if ("constructor" === d) { + return Array; + } + if ("symbol" !== typeof d) { + return (c = b.index[d / Math.pow(2, 31) | 0]) && c[d]; + } + }, set:function(c, d, e) { + c = d / Math.pow(2, 31) | 0; + (b.index[c] || (b.index[c] = []))[d] = e; + b.length++; + return !0; + }}); +} +Q.prototype.clear = function() { + this.index.length = 0; +}; +Q.prototype.push = function() { +}; +function R(a) { + a = void 0 === a ? 8 : a; + if (!(this instanceof R)) { + return new R(a); + } + this.index = I(); + this.C = []; + this.size = 0; + 32 < a ? (this.h = La, this.A = BigInt(a)) : (this.h = Ma, this.A = a); +} +R.prototype.get = function(a) { + var b = this.h(a); + return (b = this.index[b]) && b.get(a); +}; +R.prototype.set = function(a, b) { + var c = this.h(a), d = this.index[c]; + d ? (c = d.size, d.set(a, b), (c -= d.size) && this.size++) : (this.index[c] = d = new Map([[a, b]]), this.C.push(d)); +}; +function S(a) { + a = void 0 === a ? 8 : a; + if (!(this instanceof S)) { + return new S(a); + } + this.index = I(); + this.h = []; + 32 < a ? (this.C = La, this.A = BigInt(a)) : (this.C = Ma, this.A = a); +} +S.prototype.add = function(a) { + var b = this.C(a), c = this.index[b]; + c ? (b = c.size, c.add(a), (b -= c.size) && this.size++) : (this.index[b] = c = new Set([a]), this.h.push(c)); +}; +w = R.prototype; +w.has = S.prototype.has = function(a) { + var b = this.C(a); + return (b = this.index[b]) && b.has(a); +}; +w.delete = S.prototype.delete = function(a) { + var b = this.C(a); + (b = this.index[b]) && b.delete(a) && this.size--; +}; +w.clear = S.prototype.clear = function() { + this.index = I(); + this.h = []; + this.size = 0; +}; +w.values = S.prototype.values = function Na() { + var b, c = this, d, e, g; + return sa(Na, function(f) { + switch(f.h) { + case 1: + b = 0; + case 2: + if (!(b < c.h.length)) { + f.h = 0; + break; + } + d = y(c.h[b].values()); + e = d.next(); + case 5: + if (e.done) { + b++; + f.h = 2; + break; + } + g = e.value; + return F(f, g, 6); + case 6: + e = d.next(), f.h = 5; + } + }); +}; +w.keys = S.prototype.keys = function Oa() { + var b, c = this, d, e, g; + return sa(Oa, function(f) { + switch(f.h) { + case 1: + b = 0; + case 2: + if (!(b < c.h.length)) { + f.h = 0; + break; + } + d = y(c.h[b].keys()); + e = d.next(); + case 5: + if (e.done) { + b++; + f.h = 2; + break; + } + g = e.value; + return F(f, g, 6); + case 6: + e = d.next(), f.h = 5; + } + }); +}; +w.entries = S.prototype.entries = function Pa() { + var b, c = this, d, e, g; + return sa(Pa, function(f) { + switch(f.h) { + case 1: + b = 0; + case 2: + if (!(b < c.h.length)) { + f.h = 0; + break; + } + d = y(c.h[b].entries()); + e = d.next(); + case 5: + if (e.done) { + b++; + f.h = 2; + break; + } + g = e.value; + return F(f, g, 6); + case 6: + e = d.next(), f.h = 5; + } + }); +}; +function Ma(a) { + var b = Math.pow(2, this.A) - 1; + if ("number" == typeof a) { + return a & b; + } + for (var c = 0, d = this.A + 1, e = 0; e < a.length; e++) { + c = (c * d ^ a.charCodeAt(e)) & b; + } + return 32 === this.A ? c + Math.pow(2, 31) : c; +} +function La() { + throw Error("The keystore is limited to 32 for EcmaScript5"); +} +;function Qa(a, b, c, d, e, g, f, h) { + (d = a(c ? c + "." + d : d, JSON.stringify(f))) && d.then ? d.then(function() { + b.export(a, b, c, e, g + 1, h); + }) : b.export(a, b, c, e, g + 1, h); +} +;var Ra = I(), T = I(); +var Sa = {normalize:function(a) { + return a.toLowerCase(); +}, H:!1}; +var Ta = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +function Ua(a) { + Va.call(a, "add"); + Va.call(a, "append"); + Va.call(a, "search"); + Va.call(a, "update"); + Va.call(a, "remove"); +} +function Va(a) { + this[a + "Async"] = function() { + var b = arguments, c = b[b.length - 1]; + if ("function" === typeof c) { + var d = c; + delete b[b.length - 1]; + } + this.async = !0; + b = this[a].apply(this, b); + this.async = !1; + b.then ? b.then(d) : d(b); + return b; + }; +} +;I(); +U.prototype.add = function(a, b, c, d) { + if (b && (a || 0 === a)) { + if (!d && !c && this.B.has(a)) { + return this.update(a, b); + } + b = this.encoder.encode(b); + if (d = b.length) { + for (var e = I(), g = I(), f = this.depth, h = this.resolution, k = 0; k < d; k++) { + var l = b[this.rtl ? d - 1 - k : k], m = l.length; + if (m && (f || !g[l])) { + var n = this.score ? this.score(b, l, k, null, 0) : Wa(h, d, k), p = ""; + switch(this.tokenize) { + case "full": + if (2 < m) { + for (n = 0; n < m; n++) { + for (var q = m; q > n; q--) { + p = l.substring(n, q); + var r = this.score ? this.score(b, l, k, p, n) : Wa(h, d, k, m, n); + Xa(this, g, p, r, a, c); + } + } + break; + } + case "reverse": + if (1 < m) { + for (q = m - 1; 0 < q; q--) { + p = l[q] + p, r = this.score ? this.score(b, l, k, p, q) : Wa(h, d, k, m, q), Xa(this, g, p, r, a, c); + } + p = ""; + } + case "forward": + if (1 < m) { + for (q = 0; q < m; q++) { + p += l[q], Xa(this, g, p, n, a, c); + } + break; + } + default: + if (Xa(this, g, l, n, a, c), f && 1 < d && k < d - 1) { + for (m = I(), p = this.na, n = l, q = Math.min(f + 1, d - k), r = m[n] = 1; r < q; r++) { + if ((l = b[this.rtl ? d - 1 - k - r : k + r]) && !m[l]) { + m[l] = 1; + var x = this.score ? this.score(b, n, k, l, r) : Wa(p + (d / 2 > p ? 0 : 1), d, k, q - 1, r - 1), u = this.bidirectional && l > n; + Xa(this, e, u ? n : l, x, a, c, u ? l : n); + } + } + } + } + } + } + this.fastupdate || this.B.add(a); + } else { + b = ""; + } + } + this.db && (b || this.X.push({del:a}), this.qa && Ya(this)); + return this; +}; +function Xa(a, b, c, d, e, g, f) { + var h = f ? a.M : a.map, k; + if (!b[c] || !f || !(k = b[c])[f]) { + if (f ? (b = k || (b[c] = I()), b[f] = 1, (k = h.get(f)) ? h = k : h.set(f, h = new Map())) : b[c] = 1, (k = h.get(c)) ? h = k : h.set(c, h = k = []), h = h[d] || (h[d] = []), !g || !h.includes(e)) { + if (h.length === Math.pow(2, 31) - 1) { + b = new Q(h); + if (a.fastupdate) { + for (c = y(a.B.values()), g = c.next(); !g.done; g = c.next()) { + g = g.value, g.includes(h) && (g[g.indexOf(h)] = b); + } + } + k[d] = h = b; + } + h.push(e); + a.fastupdate && ((d = a.B.get(e)) ? d.push(h) : a.B.set(e, [h])); } } } +function Wa(a, b, c, d, e) { + return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; +} +;function V(a, b, c, d) { + if (1 === a.length) { + return a = a[0], a = c || a.length > b ? b ? a.slice(c, c + b) : a.slice(c) : a, d ? Za(a) : a; + } + for (var e = [], g = 0, f = void 0, h = void 0; g < a.length; g++) { + if ((f = a[g]) && (h = f.length)) { + if (c) { + if (c >= h) { + c -= h; + continue; + } + c < h && (f = b ? f.slice(c, c + b) : f.slice(c), h = f.length, c = 0); + } + if (e.length) { + h > b && (f = f.slice(0, b), h = f.length), e.push(f); + } else { + if (h >= b) { + return h > b && (f = f.slice(0, b)), d ? Za(f) : f; + } + e = [f]; + } + b -= h; + if (!b) { + break; + } + } + } + if (!e.length) { + return e; + } + e = 1 < e.length ? [].concat.apply([], e) : e[0]; + return d ? Za(e) : e; +} +function Za(a) { + for (var b = 0; b < a.length; b++) { + a[b] = {score:b, id:a[b]}; + } + return a; +} +;W.prototype.or = function() { + var a = this, b = arguments, c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.or.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.or.apply(this, c); + } + var d = []; + c = []; + for (var e = 0, g = 0, f, h, k = 0, l = void 0; k < b.length; k++) { + if (l = b[k]) { + var m = void 0; + if (l instanceof W) { + m = l.result; + } else if (l.constructor === Array) { + m = l; + } else if (l.index) { + l.resolve = !1, m = l.index.search(l).result; + } else if (l.and) { + m = this.and(l.and); + } else if (l.xor) { + m = this.xor(l.xor); + } else if (l.J) { + m = this.J(l.J); + } else { + e = l.limit || 0; + g = l.offset || 0; + f = l.enrich; + h = l.resolve; + continue; + } + d[k] = m; + m instanceof Promise && c.push(m); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result.length && (d = [a.result].concat(d)); + a.result = $a(d, e, g, f, h, a.V); + return h ? a.result : a; + }); + } + this.result.length && (d = [this.result].concat(d)); + this.result = $a(d, e, g, f, h, a.V); + return h ? this.result : this; +}; +function $a(a, b, c, d, e, g) { + if (!a.length) { + return a; + } + "object" === typeof b && (c = b.offset || 0, d = b.enrich || !1, b = b.limit || 0); + if (2 > a.length) { + return e ? V(a[0], b, c, d) : a[0]; + } + d = []; + for (var f = 0, h = I(), k = Ba(a), l = 0, m; l < k; l++) { + for (var n = 0; n < a.length; n++) { + if (m = a[n]) { + if (m = m[l]) { + for (var p = 0, q; p < m.length; p++) { + if (q = m[p], !h[q]) { + if (h[q] = 1, c) { + c--; + } else { + if (e) { + d.push(q); + } else { + var r = l + (n ? g : 0); + d[r] || (d[r] = []); + d[r].push(q); + } + if (b && ++f === b) { + return d; + } + } + } + } + } + } + } + } + return d; +} +;W.prototype.and = function() { + if (this.result.length) { + var a = this, b = arguments, c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.and.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.and.apply(this, c); + } + var d = []; + c = []; + for (var e = 0, g = 0, f, h = 0, k = void 0; h < b.length; h++) { + if (k = b[h]) { + var l = void 0; + if (k instanceof W) { + l = k.result; + } else if (k.constructor === Array) { + l = k; + } else if (k.index) { + k.resolve = !1, l = k.index.search(k).result; + } else if (k.or) { + l = this.or(k.or); + } else if (k.xor) { + l = this.xor(k.xor); + } else if (k.J) { + l = this.J(k.J); + } else { + e = k.limit || 0; + g = k.offset || 0; + f = k.resolve; + continue; + } + d[h] = l; + l instanceof Promise && c.push(l); + } + } + if (c.length) { + return Promise.all(c).then(function() { + d = [a.result].concat(d); + a.result = ab(d, e, g, f, a.V); + return f ? a.result : a; + }); + } + d = [this.result].concat(d); + this.result = ab(d, e, g, f, a.V); + return f ? this.result : this; + } + return this; +}; +function ab(a, b, c, d, e) { + if (2 > a.length) { + return []; + } + var g = [], f = 0, h = I(), k = Ba(a); + if (!k) { + return g; + } + for (var l = 0, m; l < a.length; l++) { + m = a[l]; + if (!m || !m.length) { + return []; + } + for (var n = I(), p = 0, q = l === a.length - 1, r = 0, x; r < k; r++) { + if (x = m[r]) { + for (var u = 0, t, v; u < x.length; u++) { + if (t = x[u], !l) { + n[t] = r + 1 + (l ? e : 0), p = 1; + } else if (q) { + if (v = h[t]) { + if (p = 1, c) { + c--; + } else { + if (d ? g.push(t) : (v--, r < v && (v = r), g[v] || (g[v] = []), g[v].push(t)), b && ++f === b) { + return g; + } + } + } + } else if (v = h[t]) { + r + 1 < v && (v = r + 1), n[t] = v, p = 1; + } + } + } + } + if (!p) { + return []; + } + h = n; + } + return g; +} +;W.prototype.xor = function() { + var a = this, b = arguments, c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.xor.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.xor.apply(this, c); + } + var d = []; + c = []; + for (var e = 0, g = 0, f, h, k = 0, l = void 0; k < b.length; k++) { + if (l = b[k]) { + var m = void 0; + if (l instanceof W) { + m = l.result; + } else if (l.constructor === Array) { + m = l; + } else if (l.index) { + l.resolve = !1, m = l.index.search(l).result; + } else if (l.or) { + m = this.or(l.or); + } else if (l.and) { + m = this.and(l.and); + } else if (l.J) { + m = this.J(l.J); + } else { + e = l.limit || 0; + g = l.offset || 0; + f = l.enrich; + h = l.resolve; + continue; + } + d[k] = m; + m instanceof Promise && c.push(m); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result.length && (d = [a.result].concat(d)); + a.result = bb(d, e, g, f, !h, a.V); + return h ? a.result : a; + }); + } + this.result.length && (d = [this.result].concat(d)); + this.result = bb(d, e, g, f, !h, a.V); + return h ? this.result : this; +}; +function bb(a, b, c, d, e, g) { + if (!a.length) { + return a; + } + if (2 > a.length) { + return e ? V(a[0], b, c, d) : a[0]; + } + b = []; + c = I(); + d = 0; + for (var f; d < a.length; d++) { + if (f = a[d]) { + for (var h = 0, k; h < f.length; h++) { + if (k = f[h]) { + for (var l = 0, m; l < k.length; l++) { + m = k[l], c[m] ? c[m]++ : c[m] = 1; + } + } + } + } + } + for (d = 0; d < a.length; d++) { + if (f = a[d]) { + for (h = 0; h < f.length; h++) { + if (k = f[h]) { + for (l = 0; l < k.length; l++) { + if (m = k[l], 1 === c[m]) { + if (e) { + b.push(m); + } else { + var n = h + (d ? g : 0); + b[n] || (b[n] = []); + b[n].push(m); + } + } + } + } + } + } + } + return b; +} +;W.prototype.J = function() { + var a = this, b = arguments, c = b[0]; + if (c instanceof Promise) { + return c.then(function() { + return a.J.apply(a, b); + }); + } + if (c[0] && c[0].index) { + return this.J.apply(this, c); + } + var d = []; + c = []; + for (var e, g = 0, f = void 0; g < b.length; g++) { + if (f = b[g]) { + var h = void 0; + if (f instanceof W) { + h = f.result; + } else if (f.constructor === Array) { + h = f; + } else if (f.index) { + f.resolve = !1, h = f.index.search(f).result; + } else if (f.or) { + h = this.or(f.or); + } else if (f.and) { + h = this.and(f.and); + } else if (f.xor) { + h = this.xor(f.xor); + } else { + e = f.resolve; + continue; + } + d[g] = h; + h instanceof Promise && c.push(h); + } + } + if (c.length) { + return Promise.all(c).then(function() { + a.result = cb.call(a, d, e); + return e ? a.result : a; + }); + } + this.result = cb.call(this, d, e); + return e ? this.result : this; +}; +function cb(a, b) { + if (!a.length) { + return this.result; + } + var c = []; + a = new Set(a.flat().flat()); + for (var d = 0, e; d < this.result.length; d++) { + if (e = this.result[d]) { + for (var g = 0, f; g < e.length; g++) { + f = e[g], a.has(f) || (b ? c.push(f) : (c[d] || (c[d] = []), c[d].push(f))); + } + } + } + return c; +} +;function W(a) { + if (a && a.index) { + return a.resolve = !1, this.index = a.index, a.index.search(a); + } + if (!(this instanceof W)) { + return new W(a); + } + if (a instanceof W) { + return a; + } + this.index = null; + this.result = a || []; + this.V = 0; +} +W.prototype.limit = function(a) { + if (this.result.length) { + for (var b = [], c = 0, d = 0, e; d < this.result.length; d++) { + if (e = this.result[d], e.length + c < a) { + b[d] = e, c += e.length; + } else { + b[d] = e.slice(0, a - c); + this.result = b; + break; + } + } + } + return this; +}; +W.prototype.offset = function(a) { + if (this.result.length) { + for (var b = [], c = 0, d = 0, e; d < this.result.length; d++) { + e = this.result[d], e.length + c < a ? c += e.length : (b[d] = e.slice(a - c), c = a); + } + this.result = b; + } + return this; +}; +W.prototype.resolve = function(a, b, c) { + db = 1; + var d = this.result; + this.result = this.index = null; + return d.length ? ("object" === typeof a && (c = a.enrich, b = a.offset, a = a.limit), V(d, a || 100, b, c)) : d; +}; +function eb(a, b, c, d) { + var e = a.length, g = [], f = 0, h; + d && (d = []); + for (var k = e - 1, l; 0 <= k; k--) { + var m = a[k]; + e = I(); + l = !x; + for (var n = 0, p; n < m.length; n++) { + if ((p = m[n]) && p.length) { + for (var q = 0, r; q < p.length; q++) { + if (r = p[q], x) { + if (x[r]) { + if (!k) { + if (c) { + c--; + } else { + if (g[f++] = r, f === b) { + return g; + } + } + } + if (k || d) { + e[r] = 1; + } + l = !0; + } + d && !h[r] && (h[r] = 1, (d[n] || (d[n] = [])).push(r)); + } else { + e[r] = 1; + } + } + } + } + if (d) { + x || (h = e); + } else if (!l) { + return []; + } + var x = e; + } + if (d) { + for (a = d.length - 1; 0 <= a; a--) { + for (h = d[a], e = h.length, m = 0; m < e; m++) { + if (k = h[m], !x[k]) { + if (c) { + c--; + } else { + if (g[f++] = k, f === b) { + return g; + } + } + x[k] = 1; + } + } + } + } + return g; +} +function fb(a, b) { + for (var c = I(), d = I(), e = [], g = 0; g < a.length; g++) { + c[a[g]] = 1; + } + for (a = 0; a < b.length; a++) { + g = b[a]; + for (var f = 0, h; f < g.length; f++) { + h = g[f], c[h] && !d[h] && (d[h] = 1, e.push(h)); + } + } + return e; +} +;var db = 1; +U.prototype.search = function(a, b, c) { + c || (!b && K(a) ? (c = a, a = "") : K(b) && (c = b, b = 0)); + var d = [], e = 0, g; + if (c) { + a = c.query || a; + b = c.limit || b; + e = c.offset || 0; + var f = c.context; + var h = c.suggest; + (g = db && !1 !== c.resolve) || (db = 0); + var k = g && c.enrich; + var l = this.db && c.tag; + } else { + g = this.resolve || db; + } + a = this.encoder.encode(a); + var m = a.length; + b || !g || (b = 100); + if (1 === m) { + return gb.call(this, a[0], "", b, e, g, k, l); + } + f = this.depth && !1 !== f; + if (2 === m && f && !h) { + return gb.call(this, a[0], a[1], b, e, g, k, l); + } + var n = c = 0; + if (1 < m) { + for (var p = I(), q = [], r = 0, x = void 0; r < m; r++) { + if ((x = a[r]) && !p[x]) { + if (h || this.db || X(this, x)) { + q.push(x), p[x] = 1; + } else { + return g ? d : new W(d); + } + x = x.length; + c = Math.max(c, x); + n = n ? Math.min(n, x) : x; + } + } + a = q; + m = a.length; + } + if (!m) { + return g ? d : new W(d); + } + var u = 0; + if (1 === m) { + return gb.call(this, a[0], "", b, e, g, k, l); + } + if (2 === m && f && !h) { + return gb.call(this, a[0], a[1], b, e, g, k, l); + } + if (1 < m) { + if (f) { + var t = a[0]; + u = 1; + } else { + 9 < c && 3 < c / n && a.sort(ya); + } + } + if (this.db) { + if (this.db.search && (f = this.db.search(this, a, b, e, h, g, k, l), !1 !== f)) { + return f; + } + var v = this; + return function() { + var A, D, C; + return ua(function(E) { + switch(E.h) { + case 1: + D = A = void 0; + case 2: + if (!(u < m)) { + E.h = 4; + break; + } + D = a[u]; + return t ? F(E, X(v, D, t), 8) : F(E, X(v, D), 7); + case 7: + A = E.G; + A = hb(A, d, h, v.resolution, b, e, 1 === m); + E.h = 6; + break; + case 8: + A = E.G, A = hb(A, d, h, v.na, b, e, 2 === m), h && !1 === A && d.length || (t = D); + case 6: + if (A) { + return E.return(A); + } + if (h && u === m - 1) { + C = d.length; + if (!C) { + if (t) { + t = ""; + u = -1; + E.h = 3; + break; + } + return E.return(d); + } + if (1 === C) { + return E.return(g ? V(d[0], b, e) : new W(d[0])); + } + } + case 3: + u++; + E.h = 2; + break; + case 4: + return E.return(g ? eb(d, b, e, h) : new W(d[0])); + } + }); + }(); + } + for (k = f = void 0; u < m; u++) { + k = a[u]; + t ? (f = X(this, k, t), f = hb(f, d, h, this.na, b, e, 2 === m), h && !1 === f && d.length || (t = k)) : (f = X(this, k), f = hb(f, d, h, this.resolution, b, e, 1 === m)); + if (f) { + return f; + } + if (h && u === m - 1) { + f = d.length; + if (!f) { + if (t) { + t = ""; + u = -1; + continue; + } + return d; + } + if (1 === f) { + return g ? V(d[0], b, e) : new W(d[0]); + } + } + } + return g ? eb(d, b, e, h) : new W(d[0]); +}; +function gb(a, b, c, d, e, g, f) { + a = X(this, a, b, c, d, e, g, f); + return this.db ? a.then(function(h) { + return e ? h : h && h.length ? e ? V(h, c, d) : new W(h) : e ? [] : new W([]); + }) : a && a.length ? e ? V(a, c, d) : new W(a) : e ? [] : new W([]); +} +function hb(a, b, c, d, e, g, f) { + var h = []; + if (a) { + d = Math.min(a.length, d); + for (var k = 0, l = 0, m; k < d; k++) { + if (m = a[k]) { + if (g && m && f && (m.length <= g ? (g -= m.length, m = null) : (m = m.slice(g), g = 0)), m && (h[k] = m, f && (l += m.length, l >= e))) { + break; + } + } + } + if (h.length) { + if (f) { + return V(h, e, 0); + } + b.push(h); + return; + } + } + return !c && h; +} function X(a, b, c, d, e, g, f, h) { + var k; + c && (k = a.bidirectional && b > c); + if (a.db) { + return c ? a.db.get(k ? c : b, k ? b : c, d, e, g, f, h) : a.db.get(b, "", d, e, g, f, h); + } + a = c ? (a = a.M.get(k ? b : c)) && a.get(k ? c : b) : a.map.get(b); + return a; +} +;U.prototype.remove = function(a, b) { + var c = this.B.size && (this.fastupdate ? this.B.get(a) : this.B.has(a)); + if (c) { + if (this.fastupdate) { + for (var d = 0, e; d < c.length; d++) { + if (e = c[d]) { + if (2 > e.length) { + e.pop(); + } else { + var g = e.indexOf(a); + g === c.length - 1 ? e.pop() : e.splice(g, 1); + } + } + } + } else { + ib(this.map, a), this.depth && ib(this.M, a); + } + b || this.B.delete(a); + } + this.db && (this.X.push({del:a}), this.qa && Ya(this)); + this.cache && this.cache.remove(a); + return this; +}; +function ib(a, b) { + var c = 0; + if (a.constructor === Array) { + for (var d = 0, e = void 0, g; d < a.length; d++) { + if ((e = a[d]) && e.length) { + if (g = e.indexOf(b), 0 <= g) { + 1 < e.length ? (e.splice(g, 1), c++) : delete a[d]; + break; + } else { + c++; + } + } + } + } else { + for (d = y(a), e = d.next(); !e.done; e = d.next()) { + g = e.value, e = g[0], (g = ib(g[1], b)) ? c += g : a.delete(e); + } + } + return c; +} +;function U(a, b) { + if (!(this instanceof U)) { + return new U(a); + } + if (a) { + var c = J(a) ? a : a.preset; + c && (Ta[c] || console.warn("Preset not found: " + c), a = Object.assign({}, Ta[c], a)); + } else { + a = {}; + } + c = a.context || {}; + var d = a.encode || a.encoder || Sa; + this.encoder = d.encode ? d : "object" === typeof d ? new M(d) : {encode:d}; + var e; + this.resolution = a.resolution || 9; + this.tokenize = e = a.tokenize || "strict"; + this.depth = "strict" === e && c.depth || 0; + this.bidirectional = !1 !== c.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + (e = a.keystore || 0) && (this.keystore = e); + this.map = e ? new R(e) : new Map(); + this.M = e ? new R(e) : new Map(); + this.B = b || (this.fastupdate ? e ? new R(e) : new Map() : e ? new S(e) : new Set()); + this.na = c.resolution || 1; + this.rtl = d.rtl || a.rtl || !1; + this.cache = (e = a.cache || null) && new N(e); + this.resolve = !1 !== a.resolve; + if (e = a.db) { + this.db = e.mount(this); + } + this.qa = !1 !== a.commit; + this.X = []; + this.h = null; +} +w = U.prototype; +w.mount = function(a) { + this.h && (clearTimeout(this.h), this.h = null); + return a.mount(this); +}; +w.commit = function(a, b) { + this.h && (clearTimeout(this.h), this.h = null); + return this.db.commit(this, a, b); +}; +function Ya(a) { + a.h || (a.h = setTimeout(function() { + a.h = null; + a.db.commit(a, void 0, void 0); + }, 0)); +} +w.clear = function() { + this.map.clear(); + this.M.clear(); + this.B.clear(); + this.cache && this.cache.clear(); + this.db && (this.h && clearTimeout(this.h), this.h = null, this.X = [{clear:!0}]); + return this; +}; +w.append = function(a, b) { + return this.add(a, b, !0); +}; +w.contain = function(a) { + return this.db ? this.db.has(a) : this.B.has(a); +}; +w.update = function(a, b) { + if (this.async) { + var c = this, d = this.remove(a); + return d.then ? d.then(function() { + return c.add(a, b); + }) : this.add(a, b); + } + return this.remove(a).add(a, b); +}; +function jb(a) { + var b = 0; + if (a.constructor === Array) { + for (var c = 0, d = void 0; c < a.length; c++) { + (d = a[c]) && (b += d.length); + } + } else { + for (c = y(a), d = c.next(); !d.done; d = c.next()) { + var e = d.value; + d = e[0]; + (e = jb(e[1])) ? b += e : a.delete(d); + } + } + return b; +} +w.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + jb(this.map); + this.depth && jb(this.M); + return this; +}; +w.searchCache = Ja; +w.export = function(a, b, c, d, e, g) { + var f = !0; + "undefined" === typeof g && (f = new Promise(function(n) { + g = n; + })); + switch(e || (e = 0)) { + case 0: + var h = "reg"; + if (this.fastupdate) { + var k = I(); + for (var l = y(this.B.keys()), m = l.next(); !m.done; m = l.next()) { + k[m.value] = 1; + } + } else { + k = this.B; + } + break; + case 1: + h = "cfg"; + k = {doc:0, opt:this.A ? 1 : 0}; + break; + case 2: + h = "map"; + k = this.map; + break; + case 3: + h = "ctx"; + k = this.M; + break; + default: + "undefined" === typeof c && g && g(); + return; + } + Qa(a, b || this, c, h, d, e, k, g); + return f; +}; +w.import = function(a, b) { + if (b) { + switch(J(b) && (b = JSON.parse(b)), a) { + case "cfg": + this.A = !!b.opt; + break; + case "reg": + this.fastupdate = !1; + this.B = b; + break; + case "map": + this.map = b; + break; + case "ctx": + this.M = b; + } + } +}; +Ua(U.prototype); +function kb(a) { + var b, c, d, e, g, f, h, k; + return ua(function(l) { + a = a.data; + b = self._index; + c = a.args; + d = a.task; + switch(d) { + case "init": + e = a.options || {}; + (g = e.config) && (e = g); + (f = a.factory) ? (Function("return " + f)()(self), self._index = new self.FlexSearch.Index(e), delete self.FlexSearch) : self._index = new U(e); + postMessage({id:a.id}); + break; + default: + h = a.id, k = b[d].apply(b, c), postMessage("search" === d ? {id:h, msg:k} : {id:h}); + } + l.h = 0; + }); +} +;var lb = 0; +function Y(a) { + function b(g) { + g = g.data || g; + var f = g.id, h = f && e.h[f]; + h && (h(g.msg), delete e.h[f]); + } + if (!(this instanceof Y)) { + return new Y(a); + } + a || (a = {}); + var c = (self || window)._factory; + c && (c = c.toString()); + var d = "undefined" === typeof window && self.exports, e = this; + this.worker = mb(c, d, a.worker); + this.h = I(); + if (this.worker) { + d ? this.worker.on("message", b) : this.worker.onmessage = b; + if (a.config) { + return new Promise(function(g) { + e.h[++lb] = function() { + g(e); + }; + e.worker.postMessage({id:lb, task:"init", factory:c, options:a}); + }); + } + this.worker.postMessage({task:"init", factory:c, options:a}); + } +} +nb("add"); +nb("append"); +nb("search"); +nb("update"); +nb("remove"); +function nb(a) { + Y.prototype[a] = Y.prototype[a + "Async"] = function() { + var b = this, c = [].slice.call(arguments), d = c[c.length - 1]; + if ("function" === typeof d) { + var e = d; + c.splice(c.length - 1, 1); + } + d = new Promise(function(g) { + b.h[++lb] = g; + b.worker.postMessage({task:a, id:lb, args:c}); + }); + return e ? (d.then(e), this) : d; + }; +} +function mb(a, b, c) { + return b ? new (require("worker_threads")["Worker"])(__dirname + "/node/node.js") : a ? new window.Worker(URL.createObjectURL(new Blob(["onmessage=" + kb.toString()], {type:"text/javascript"}))) : new window.Worker(J(c) ? c : "worker/worker.js", {type:"module"}); +} +;Z.prototype.add = function(a, b, c) { + K(a) && (b = a, a = Aa(b, this.key)); + if (b && (a || 0 === a)) { + if (!c && this.B.has(a)) { + return this.update(a, b); + } + for (var d = 0, e; d < this.field.length; d++) { + e = this.T[d]; + var g = this.index.get(this.field[d]); + if ("function" === typeof e) { + (e = e(b)) && g.add(a, e, !1, !0); + } else { + var f = e.aa; + if (!f || f(b)) { + e instanceof String ? e = ["" + e] : J(e) && (e = [e]), ob(b, e, this.da, 0, g, a, e[0], c); + } + } + } + if (this.tag) { + for (d = 0; d < this.S.length; d++) { + f = this.S[d]; + var h = this.ma[d]; + g = this.tag.get(h); + e = I(); + if ("function" === typeof f) { + if (f = f(b), !f) { + continue; + } + } else { + var k = f.aa; + if (k && !k(b)) { + continue; + } + f instanceof String && (f = "" + f); + f = Aa(b, f); + } + if (g && f) { + for (J(f) && (f = [f]), h = 0, k = void 0; h < f.length; h++) { + var l = f[h]; + if (!e[l]) { + e[l] = 1; + var m; + (m = g.get(l)) ? k = m : g.set(l, k = []); + if (!c || !k.includes(a)) { + if (k.length === Math.pow(2, 31) - 1) { + m = new Q(k); + if (this.fastupdate) { + for (var n = y(this.B.values()), p = n.next(); !p.done; p = n.next()) { + p = p.value, p.includes(k) && (p[p.indexOf(k)] = m); + } + } + g.set(l, k = m); + } + k.push(a); + this.fastupdate && ((l = this.B.get(a)) ? l.push(k) : this.B.set(a, [k])); + } + } + } + } else { + g || console.warn("Tag '" + h + "' was not found"); + } + } + } + if (this.store && (!c || !this.store.has(a))) { + if (this.L) { + var q = I(); + for (c = 0; c < this.L.length; c++) { + if (d = this.L[c], g = d.aa, !g || g(b)) { + g = void 0; + if ("function" === typeof d) { + g = d(b); + if (!g) { + continue; + } + d = [d.Aa]; + } else if (J(d) || d instanceof String) { + q[d] = b[d]; + continue; + } + pb(b, q, d, 0, d[0], g); + } + } + } + this.store.set(a, q || b); + } + } + return this; +}; +function pb(a, b, c, d, e, g) { + a = a[e]; + if (d === c.length - 1) { + b[e] = g || a; + } else if (a) { + if (a.constructor === Array) { + for (b = b[e] = Array(a.length), e = 0; e < a.length; e++) { + pb(a, b, c, d, e); + } + } else { + b = b[e] || (b[e] = I()), e = c[++d], pb(a, b, c, d, e); + } + } +} +function ob(a, b, c, d, e, g, f, h) { if (a = a[f]) { if (d === b.length - 1) { if (a.constructor === Array) { @@ -1057,223 +2456,462 @@ function X(a, b, c, d, e, g, f, h) { } else { if (a.constructor === Array) { for (f = 0; f < a.length; f++) { - X(a, b, c, d, e, g, f, h); + ob(a, b, c, d, e, g, f, h); } } else { - f = b[++d], X(a, b, c, d, e, g, f, h); + f = b[++d], ob(a, b, c, d, e, g, f, h); } } + } else { + e.db && e.remove(g); } } -t = T.prototype; -t.add = function(a, b, c) { - F(a) && (b = a, a = V(b, this.key)); - if (b && (a || 0 === a)) { - if (!c && this.register[a]) { - return this.update(a, b); - } - for (var d = 0, e, g; d < this.h.length; d++) { - g = this.h[d], e = this.I[d], E(e) && (e = [e]), X(b, e, this.m, 0, this.index[g], a, e[0], c); - } - if (this.D) { - d = V(b, this.D); - e = D(); - E(d) && (d = [d]); - g = 0; - for (var f; g < d.length; g++) { - if (f = d[g], !e[f] && (e[f] = 1, f = this.l[f] || (this.l[f] = []), !c || !f.includes(a))) { - if (f[f.length] = a, this.s) { - var h = this.register[a] || (this.register[a] = []); - h[h.length] = f; - } - } - } - } - if (this.store && (!c || !this.store[a])) { - if (this.o) { - var k = D(); - for (c = 0; c < this.o.length; c++) { - d = this.o[c], E(d) ? k[d] = b[d] : W(b, k, d, 0, d[0]); - } - } - this.store[a] = k || b; - } - } - return this; -}; -t.append = function(a, b) { - return this.add(a, b, !0); -}; -t.update = function(a, b) { - return this.remove(a).add(a, b); -}; -t.remove = function(a) { - F(a) && (a = V(a, this.key)); - if (this.register[a]) { - for (var b = 0; b < this.h.length && (this.index[this.h[b]].remove(a, !this.A), !this.s); b++) { - } - if (this.D && !this.s) { - for (var c in this.l) { - b = this.l[c]; - var d = b.indexOf(a); - -1 !== d && (1 < b.length ? b.splice(d, 1) : delete this.l[c]); - } - } - this.store && delete this.store[a]; - delete this.register[a]; - } - return this; -}; -t.search = function(a, b, c, d) { - c || (!b && F(a) ? (c = a, a = "") : F(b) && (c = b, b = 0)); - var e = [], g = [], f, h = 0; +;Z.prototype.search = function(a, b, c, d) { + c || (!b && K(a) ? (c = a, a = "") : K(b) && (c = b, b = 0)); + var e = [], g = [], f = 0; if (c) { if (c.constructor === Array) { - var k = c; + var h = c; c = null; } else { a = c.query || a; - k = (f = c.pluck) || c.index || c.field; - var l = c.tag; - var m = this.store && c.enrich; - var p = "and" === c.bool; - b = c.limit || b || 100; - var n = c.offset || 0; - if (l && (E(l) && (l = [l]), !a)) { - g = 0; - for (f = void 0; g < l.length; g++) { - if (f = Da.call(this, l[g], b, n, m)) { - e[e.length] = f, h++; + var k = c.pluck; + var l = c.merge; + h = k || c.field || c.index; + var m = this.tag && c.tag; + var n = this.store && c.enrich; + var p = c.suggest; + b = c.limit || b; + var q = c.offset || 0; + b || (b = 100); + if (m && (!this.db || !d)) { + m.constructor !== Array && (m = [m]); + for (var r = [], x = 0, u = void 0; x < m.length; x++) { + u = m[x]; + if (J(u)) { + throw Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + if (u.field && u.tag) { + var t = u.tag; + if (t.constructor === Array) { + for (var v = 0; v < t.length; v++) { + r.push(u.field, t[v]); + } + } else { + r.push(u.field, t); + } + } else { + t = Object.keys(u); + v = 0; + for (var A = void 0, D = void 0; v < t.length; v++) { + if (A = t[v], D = u[A], D.constructor === Array) { + for (var C = 0; C < D.length; C++) { + r.push(A, D[C]); + } + } else { + r.push(A, D); + } + } } } - return h ? e : []; + if (!r.length) { + throw Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + m = r; + if (!a) { + g = []; + if (r.length) { + for (k = 0; k < r.length; k += 2) { + p = void 0; + if (this.db) { + p = this.index.get(r[k]); + if (!p) { + console.warn("Tag '" + r[k] + ":" + r[k + 1] + "' will be skipped because there is no field '" + r[k] + "'."); + continue; + } + g.push(p = p.db.tag(r[k + 1], b, q, n)); + } else { + p = qb.call(this, r[k], r[k + 1], b, q, n); + } + e.push({field:r[k], tag:r[k + 1], result:p}); + } + } + return g.length ? Promise.all(g).then(function(O) { + for (var P = 0; P < O.length; P++) { + e[P].result = O[P]; + } + return e; + }) : e; + } } - E(k) && (k = [k]); + J(h) && (h = [h]); } } - k || (k = this.h); - p = p && (1 < k.length || l && 1 < l.length); - for (var q = !d && (this.A || this.async) && [], r = 0, u = void 0, A = void 0, w = void 0; r < k.length; r++) { - if (w = void 0, A = k[r], E(A) || (w = A, A = w.field, a = w.query || a, b = w.limit || b, m = w.enrich || m), q) { - q[r] = this.index[A].searchAsync(a, b, w || c); - } else { - d ? u = d[r] : u = this.index[A].search(a, b, w || c); - w = u && u.length; - if (l && w) { - var z = [], B = 0; - p && (z[0] = [u]); - var Y = 0, H = void 0; - for (H = void 0; Y < l.length; Y++) { - if (H = l[Y], w = (H = this.l[H]) && H.length) { - B++, z[z.length] = p ? [H] : H; + h || (h = this.field); + r = !d && (this.worker || this.async) && []; + x = 0; + for (v = u = t = void 0; x < h.length; x++) { + if (u = h[x], !this.db || !this.tag || this.T[x]) { + t = void 0; + J(u) || (t = u, u = t.field, a = t.query || a, b = t.limit || b, p = t.suggest || p); + if (d) { + t = d[x]; + } else { + v = t || c; + t = this.index.get(u); + if (m) { + if (this.db) { + v.tag = m; + var E = t.db.Fa; + v.field = h; + } + E || (v.enrich = !1); + } + if (r) { + r[x] = t.searchAsync(a, b, v); + v && n && (v.enrich = n); + continue; + } else { + t = t.search(a, b, v), v && n && (v.enrich = n); + } + } + v = t && t.length; + if (m && v) { + A = []; + D = 0; + if (this.db && d) { + if (!E) { + for (C = h.length; C < d.length; C++) { + var L = d[C]; + if (L && L.length) { + D++, A.push(L); + } else if (!p) { + return e; + } + } + } + } else { + C = 0; + for (var Fb = L = void 0; C < m.length; C += 2) { + L = this.tag.get(m[C]); + if (!L) { + if (console.warn("Tag '" + m[C] + ":" + m[C + 1] + "' will be skipped because there is no field '" + m[C] + "'."), p) { + continue; + } else { + return e; + } + } + if (Fb = (L = L && L.get(m[C + 1])) && L.length) { + D++, A.push(L); + } else if (!p) { + return e; + } } } - B && (u = p ? sa(z, b || 100, n || 0) : ta(u, z), w = u.length); + if (D) { + t = fb(t, A); + v = t.length; + if (!v && !p) { + return e; + } + D--; + } } - if (w) { - g[h] = A, e[h++] = u; - } else if (p) { - return []; + if (v) { + g[f] = u, e.push(t), f++; + } else if (1 === h.length) { + return e; } } } - if (q) { - var Ka = this; - return new Promise(function(La) { - Promise.all(q).then(function(Ma) { - La(Ka.search(a, b, c, Ma)); - }); + if (r) { + if (this.db && m && m.length && !E) { + for (n = 0; n < m.length; n += 2) { + g = this.index.get(m[n]); + if (!g) { + if (console.warn("Tag '" + m[n] + ":" + m[n + 1] + "' was not found because there is no field '" + m[n] + "'."), p) { + continue; + } else { + return e; + } + } + r.push(g.db.tag(m[n + 1], b, q, !1)); + } + } + var Gb = this; + return Promise.all(r).then(function(O) { + return O.length ? Gb.search(a, b, c, O) : O; }); } - if (!h) { - return []; + if (!f) { + return e; } - if (f && (!m || !this.store)) { + if (k && (!n || !this.store)) { return e[0]; } - l = 0; - for (n = void 0; l < g.length; l++) { - n = e[l]; - n.length && m && (n = Ea.call(this, n)); - if (f) { - return n; + r = []; + q = 0; + for (p = void 0; q < g.length; q++) { + p = e[q]; + n && p.length && !p[0].doc && (this.db ? r.push(p = this.index.get(this.field[0]).db.enrich(p)) : p.length && (p = rb.call(this, p))); + if (k) { + return p; } - e[l] = {field:g[l], result:n}; + e[q] = {field:g[q], result:p}; } - return e; -}; -function Da(a, b, c, d) { - var e = this.l[a], g = e && e.length - c; - if (g && 0 < g) { - if (g > b || c) { - e = e.slice(c, c + b); + return n && this.db && r.length ? Promise.all(r).then(function(O) { + for (var P = 0; P < O.length; P++) { + e[P].result = O[P]; } - d && (e = Ea.call(this, e)); - return {tag:a, result:e}; + return l ? sb(e, b) : e; + }) : l ? sb(e, b) : e; +}; +function sb(a, b) { + for (var c = [], d = I(), e = 0, g, f; e < a.length; e++) { + g = a[e]; + f = g.result; + for (var h = 0, k, l, m; h < f.length; h++) { + if (l = f[h], k = l.id, m = d[k]) { + m.push(g.field); + } else { + if (c.length === b) { + return c; + } + l.field = d[k] = [g.field]; + c.push(l); + } + } + } + return c; +} +function qb(a, b, c, d, e) { + var g = this.tag.get(a); + if (!g) { + return console.warn("Tag '" + a + "' was not found"), []; + } + if ((a = (g = g && g.get(b)) && g.length - d) && 0 < a) { + if (a > c || d) { + g = g.slice(d, d + c); + } + e && (g = rb.call(this, g)); + return g; } } -function Ea(a) { +function rb(a) { for (var b = Array(a.length), c = 0, d; c < a.length; c++) { - d = a[c], b[c] = {id:d, doc:this.store[d]}; + d = a[c], b[c] = {id:d, doc:this.store.get(d)}; } return b; } -t.contain = function(a) { - return !!this.register[a]; +;function Z(a) { + if (!(this instanceof Z)) { + return new Z(a); + } + var b = a.document || a.doc || a, c, d; + this.T = []; + this.field = []; + this.da = []; + this.key = (c = b.key || b.id) && tb(c, this.da) || "id"; + (d = a.keystore || 0) && (this.keystore = d); + this.B = (this.fastupdate = !!a.fastupdate) ? d ? new R(d) : new Map() : d ? new S(d) : new Set(); + this.L = (c = b.store || null) && !0 !== c && []; + this.store = c && (d ? new R(d) : new Map()); + this.cache = (c = a.cache || null) && new N(c); + a.cache = !1; + this.worker = a.worker; + this.async = !1; + c = new Map(); + d = b.index || b.field || b; + J(d) && (d = [d]); + for (var e = 0, g, f = void 0; e < d.length; e++) { + g = d[e]; + J(g) || (f = g, g = g.field); + f = K(f) ? Object.assign({}, a, f) : a; + if (this.worker) { + var h = new Y(f); + c.set(g, h); + h.worker || (this.worker = !1); + } + this.worker || c.set(g, new U(f, this.B)); + f.Y ? this.T[e] = f.Y : (this.T[e] = tb(g, this.da), f.filter && ("string" === typeof this.T[e] && (this.T[e] = new String(this.T[e])), this.T[e].aa = f.filter)); + this.field[e] = g; + } + if (this.L) { + for (d = b.store, J(d) && (d = [d]), e = 0; e < d.length; e++) { + g = d[e], f = g.field || g, g.Y ? (this.L[e] = g.Y, g.Y.Aa = f) : (this.L[e] = tb(f, this.da), g.filter && ("string" === typeof this.L[e] && (this.L[e] = new String(this.L[e])), this.L[e].aa = g.filter)); + } + } + this.index = c; + this.tag = null; + if (c = b.tag) { + if ("string" === typeof c && (c = [c]), c.length) { + for (this.tag = new Map(), this.S = [], this.ma = [], b = 0; b < c.length; b++) { + d = c[b]; + e = d.field || d; + if (!e) { + throw Error("The tag field from the document descriptor is undefined."); + } + d.Y ? this.S[b] = d.Y : (this.S[b] = tb(e, this.da), d.filter && ("string" === typeof this.S[b] && (this.S[b] = new String(this.S[b])), this.S[b].aa = d.filter)); + this.ma[b] = e; + this.tag.set(e, new Map()); + } + } + } + a.db && this.mount(a.db); +} +w = Z.prototype; +w.mount = function(a) { + var b = this.field; + if (this.tag) { + for (var c = 0, d; c < this.ma.length; c++) { + d = this.ma[c]; + var e = this.index.get(d); + e || (this.index.set(d, e = new U({}, this.B)), b === this.field && (b = b.slice(0)), b.push(d)); + e.tag = this.tag.get(d); + } + } + c = []; + d = {db:a.db, type:a.type, fastupdate:a.fastupdate}; + e = 0; + for (var g; e < b.length; e++) { + d.field = g = b[e]; + g = this.index.get(g); + var f = new a.constructor(a.id, d); + f.id = a.id; + c[e] = f.mount(g); + g.document = !0; + e ? g.Ba = !0 : g.store = this.store; + } + this.db = this.async = !0; + return Promise.all(c); }; -t.get = function(a) { - return this.store[a]; +w.commit = function(a, b) { + var c = this, d, e, g, f; + return ua(function(h) { + if (1 == h.h) { + d = []; + e = y(c.index.values()); + for (g = e.next(); !g.done; g = e.next()) { + f = g.value, d.push(f.db.commit(f, a, b)); + } + return F(h, Promise.all(d), 2); + } + c.B.clear(); + h.h = 0; + }); }; -t.set = function(a, b) { - this.store[a] = b; +function tb(a, b) { + for (var c = a.split(":"), d = 0, e = 0; e < c.length; e++) { + a = c[e], "]" === a[a.length - 1] && (a = a.substring(0, a.length - 2)) && (b[d] = !0), a && (c[d++] = a); + } + d < c.length && (c.length = d); + return 1 < d ? c : c[0]; +} +w.append = function(a, b) { + return this.add(a, b, !0); +}; +w.update = function(a, b) { + return this.remove(a).add(a, b); +}; +w.remove = function(a) { + K(a) && (a = Aa(a, this.key)); + for (var b = y(this.index.values()), c = b.next(); !c.done; c = b.next()) { + c.value.remove(a, !0); + } + if (this.B.has(a)) { + if (this.tag && !this.fastupdate) { + for (b = y(this.tag.values()), c = b.next(); !c.done; c = b.next()) { + c = c.value; + for (var d = y(c), e = d.next(); !e.done; e = d.next()) { + var g = e.value; + e = g[0]; + g = g[1]; + var f = g.indexOf(a); + -1 < f && (1 < g.length ? g.splice(f, 1) : c.delete(e)); + } + } + } + this.store && this.store.delete(a); + this.B.delete(a); + } + this.cache && this.cache.remove(a); return this; }; -t.searchCache = ua; -t.export = function(a, b, c, d, e, g) { +w.clear = function() { + for (var a = y(this.index.values()), b = a.next(); !b.done; b = a.next()) { + b.value.clear(); + } + if (this.tag) { + for (a = y(this.tag.values()), b = a.next(); !b.done; b = a.next()) { + b.value.clear(); + } + } + this.store && this.store.clear(); + return this; +}; +w.contain = function(a) { + return this.db ? this.index.get(this.field[0]).db.has(a) : this.B.has(a); +}; +w.cleanup = function() { + for (var a = y(this.index.values()), b = a.next(); !b.done; b = a.next()) { + b.value.cleanup(); + } + return this; +}; +w.get = function(a) { + return this.db ? this.index.get(this.field[0]).db.enrich(a).then(function(b) { + return b[0] && b[0].doc; + }) : this.store.get(a); +}; +w.set = function(a, b) { + this.store.set(a, b); + return this; +}; +w.searchCache = Ja; +w.export = function(a, b, c, d, e, g) { var f; - "undefined" === typeof g && (f = new Promise(function(p) { - g = p; + "undefined" === typeof g && (f = new Promise(function(k) { + g = k; })); e || (e = 0); d || (d = 0); - if (d < this.h.length) { - var h = this.h[d], k = this.index[h]; + if (d < this.field.length) { + c = this.field[d]; + var h = this.index[c]; b = this; - setTimeout(function() { - k.export(a, b, e ? h : "", d, e++, g) || (d++, e = 1, b.export(a, b, h, d, e, g)); - }); + h.export(a, b, e ? c : "", d, e++, g) || (d++, b.export(a, b, c, d, 1, g)); } else { switch(e) { case 1: - var l = "tag"; - var m = this.l; + b = "tag"; + h = this.h; c = null; break; case 2: - l = "store"; - m = this.store; + b = "store"; + h = this.store; c = null; break; default: g(); return; } - wa(a, this, c, l, d, e, m, g); + Qa(a, this, c, b, d, e, h, g); } return f; }; -t.import = function(a, b) { +w.import = function(a, b) { if (b) { - switch(E(b) && (b = JSON.parse(b)), a) { + switch(J(b) && (b = JSON.parse(b)), a) { case "tag": - this.l = b; + this.h = b; break; case "reg": - this.s = !1; - this.register = b; + this.fastupdate = !1; + this.B = b; a = 0; - for (var c; a < this.h.length; a++) { - c = this.index[this.h[a]], c.register = b, c.s = !1; + for (var c; a < this.field.length; a++) { + c = this.index[this.field[a]], c.B = b, c.fastupdate = !1; } break; case "store": @@ -1284,51 +2922,384 @@ t.import = function(a, b) { } } }; -ra(T.prototype); -var Ga = {encode:Fa, G:!1, H:""}, Ha = [J("[\u00e0\u00e1\u00e2\u00e3\u00e4\u00e5]"), "a", J("[\u00e8\u00e9\u00ea\u00eb]"), "e", J("[\u00ec\u00ed\u00ee\u00ef]"), "i", J("[\u00f2\u00f3\u00f4\u00f5\u00f6\u0151]"), "o", J("[\u00f9\u00fa\u00fb\u00fc\u0171]"), "u", J("[\u00fd\u0177\u00ff]"), "y", J("\u00f1"), "n", J("[\u00e7c]"), "k", J("\u00df"), "s", J(" & "), " and "]; -function Fa(a) { - var b = a = "" + a; - b.normalize && (b = b.normalize("NFD").replace(la, "")); - return ja.call(this, b.toLowerCase(), !a.normalize && Ha); +Ua(Z.prototype); +var ub = "undefined" !== typeof window && (window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB), vb = ["map", "ctx", "tag", "reg", "cfg"]; +function wb(a, b) { + b = void 0 === b ? {} : b; + if (!(this instanceof wb)) { + return new wb(a, b); + } + "object" === typeof a && (b = a = a.name); + a || console.info("Default storage space was used, because a name was not passed."); + this.id = "flexsearch" + (a ? ":" + a.toLowerCase().replace(/[^a-z0-9_\-]/g, "") : ""); + this.field = b.field ? b.field.toLowerCase().replace(/[^a-z0-9_\-]/g, "") : ""; + this.Fa = !1; + this.db = null; + this.h = {}; } -;var Ja = {encode:Ia, G:!1, H:"strict"}, Na = /[^a-z0-9]+/, Oa = {b:"p", v:"f", w:"f", z:"s", x:"s", "\u00df":"s", d:"t", n:"m", c:"k", g:"k", j:"k", q:"k", i:"e", y:"e", u:"o"}; -function Ia(a) { - a = Fa.call(this, a).join(" "); - var b = []; - if (a) { - for (var c = a.split(Na), d = c.length, e = 0, g, f = 0; e < d; e++) { - if ((a = c[e]) && (!this.filter || !this.filter[a])) { - g = a[0]; - for (var h = Oa[g] || g, k = h, l = 1; l < a.length; l++) { - g = a[l], (g = Oa[g] || g) && g !== k && (h += g, k = g); - } - b[f++] = h; +w = wb.prototype; +w.mount = function(a) { + if (a instanceof Z) { + return a.mount(this); + } + a.db = this; + return xb(this); +}; +function xb(a) { + navigator.storage && navigator.storage.persist(); + return a.db || new Promise(function(b, c) { + var d = ub.open(a.id + (a.field ? ":" + a.field : ""), 1); + d.onupgradeneeded = function() { + var e = a.db = this.result; + vb.forEach(function(g) { + e.objectStoreNames.contains(g) || e.createObjectStore(g); + }); + }; + d.onblocked = function(e) { + console.error("blocked", e); + c(); + }; + d.onerror = function(e) { + console.error(this.error, e); + c(); + }; + d.onsuccess = function() { + a.db = this.result; + a.db.onversionchange = function() { + a.close(); + }; + b(a); + }; + }); +} +w.close = function() { + this.db.close(); + this.db = null; +}; +w.clear = function() { + for (var a = this.db.transaction(vb, "readwrite"), b = 0; b < vb.length; b++) { + a.objectStore(vb[b]).clear(); + } + return yb(a); +}; +w.get = function(a, b, c, d, e, g) { + c = void 0 === c ? 0 : c; + d = void 0 === d ? 0 : d; + e = void 0 === e ? !0 : e; + g = void 0 === g ? !1 : g; + a = this.db.transaction(b ? "ctx" : "map", "readonly").objectStore(b ? "ctx" : "map").get(b ? b + ":" + a : a); + var f = this; + return yb(a).then(function(h) { + var k = []; + if (!h || !h.length) { + return k; + } + if (e) { + if (!c && !d && 1 === h.length) { + return h[0]; } + for (var l = 0, m = void 0; l < h.length; l++) { + if ((m = h[l]) && m.length) { + if (d >= m.length) { + d -= m.length; + } else { + for (var n = c ? d + Math.min(m.length - d, c) : m.length, p = d; p < n; p++) { + k.push(m[p]); + } + d = 0; + if (k.length === c) { + break; + } + } + } + } + return g ? f.enrich(k) : k; + } + return h; + }); +}; +w.tag = function(a, b, c, d) { + b = void 0 === b ? 0 : b; + c = void 0 === c ? 0 : c; + d = void 0 === d ? !1 : d; + a = this.db.transaction("tag", "readonly").objectStore("tag").get(a); + var e = this; + return yb(a).then(function(g) { + if (!g || !g.length || c >= g.length) { + return []; + } + if (!b && !c) { + return g; + } + g = g.slice(c, c + b); + return d ? e.enrich(g) : g; + }); +}; +w.enrich = function(a) { + "object" !== typeof a && (a = [a]); + for (var b = this.db.transaction("reg", "readonly").objectStore("reg"), c = [], d = 0; d < a.length; d++) { + c[d] = yb(b.get(a[d])); + } + return Promise.all(c).then(function(e) { + for (var g = 0; g < e.length; g++) { + e[g] = {id:a[g], doc:e[g] ? JSON.parse(e[g]) : null}; + } + return e; + }); +}; +w.has = function(a) { + a = this.db.transaction("reg", "readonly").objectStore("reg").getKey(a); + return yb(a); +}; +w.search = null; +w.info = function() { +}; +w.transaction = function(a, b, c) { + var d = this, e = this.h[a + ":" + b]; + if (e) { + return c.call(this, e); + } + var g = this.db.transaction(a, b); + this.h[a + ":" + b] = e = g.objectStore(a); + return new Promise(function(f, h) { + g.onerror = function(k) { + d.h[a + ":" + b] = null; + g.abort(); + g = e = null; + h(k); + }; + g.oncomplete = function(k) { + g = e = d.h[a + ":" + b] = null; + f(k || !0); + }; + return c.call(d, e); + }); +}; +w.commit = function(a, b, c) { + var d = this, e, g, f; + return ua(function(h) { + switch(h.h) { + case 1: + if (b) { + return F(h, d.clear(), 12); + } + e = a.X; + a.X = []; + g = 0; + f = void 0; + case 4: + if (!(g < e.length)) { + h.h = 6; + break; + } + f = e[g]; + if (!f.clear) { + e[g] = f.Ga; + h.h = 5; + break; + } + return F(h, d.clear(), 8); + case 8: + b = !0; + h.h = 6; + break; + case 5: + g++; + h.h = 4; + break; + case 6: + if (b) { + h.h = 3; + break; + } + c || (e = e.concat(za(a.B))); + if (!e.length) { + h.h = 10; + break; + } + return F(h, d.remove(e), 11); + case 11: + case 10: + h.h = 3; + break; + case 12: + a.X = []; + case 3: + return a.B.size ? F(h, d.transaction("map", "readwrite", function(k) { + for (var l = y(a.map), m = l.next(), n = {}; !m.done; n = {Z:void 0, ia:void 0}, m = l.next()) { + m = m.value, n.ia = m[0], n.Z = m[1], n.Z.length && (b ? k.put(n.Z, n.ia) : k.get(n.ia).onsuccess = function(p) { + return function() { + var q = this.result, r; + if (q && q.length) { + for (var x = Math.max(q.length, p.Z.length), u = 0, t; u < x; u++) { + if ((t = p.Z[u]) && t.length) { + if ((r = q[u]) && r.length) { + for (var v = 0; v < t.length; v++) { + r.push(t[v]); + } + } else { + q[u] = t; + } + r = 1; + } + } + } else { + q = p.Z, r = 1; + } + r && k.put(q, p.ia); + }; + }(n)); + } + }), 13) : h.return(); + case 13: + return F(h, d.transaction("ctx", "readwrite", function(k) { + for (var l = y(a.M), m = l.next(), n = {}; !m.done; n = {ga:void 0}, m = l.next()) { + m = m.value; + n.ga = m[0]; + m = y(m[1]); + for (var p = m.next(), q = {}; !p.done; q = {$:void 0, ja:void 0}, p = m.next()) { + p = p.value, q.ja = p[0], q.$ = p[1], q.$.length && (b ? k.put(q.$, n.ga + ":" + q.ja) : k.get(n.ga + ":" + q.ja).onsuccess = function(r, x) { + return function() { + var u = this.result, t; + if (u && u.length) { + for (var v = Math.max(u.length, r.$.length), A = 0, D; A < v; A++) { + if ((D = r.$[A]) && D.length) { + if ((t = u[A]) && t.length) { + for (var C = 0; C < D.length; C++) { + t.push(D[C]); + } + } else { + u[A] = D; + } + t = 1; + } + } + } else { + u = r.$, t = 1; + } + t && k.put(u, x.ga + ":" + r.ja); + }; + }(q, n)); + } + } + }), 14); + case 14: + if (a.store) { + return F(h, d.transaction("reg", "readwrite", function(k) { + for (var l = y(a.store), m = l.next(); !m.done; m = l.next()) { + var n = m.value; + m = n[0]; + n = n[1]; + k.put("object" === typeof n ? JSON.stringify(n) : 1, m); + } + }), 16); + } + if (a.Ba) { + h.h = 16; + break; + } + return F(h, d.transaction("reg", "readwrite", function(k) { + for (var l = y(a.B.keys()), m = l.next(); !m.done; m = l.next()) { + k.put(1, m.value); + } + }), 16); + case 16: + if (!a.tag) { + h.h = 20; + break; + } + return F(h, d.transaction("tag", "readwrite", function(k) { + for (var l = y(a.tag), m = l.next(), n = {}; !m.done; n = {ha:void 0, oa:void 0}, m = l.next()) { + m = m.value, n.oa = m[0], n.ha = m[1], n.ha.length && (k.get(n.oa).onsuccess = function(p) { + return function() { + var q = this.result; + q = q && q.length ? q.concat(p.ha) : p.ha; + k.put(q, p.oa); + }; + }(n)); + } + }), 20); + case 20: + a.map.clear(), a.M.clear(), a.tag && a.tag.clear(), a.store && a.store.clear(), a.document || a.B.clear(), h.h = 0; + } + }); +}; +function zb(a, b, c) { + for (var d = a.value, e, g, f = 0, h = 0, k; h < d.length; h++) { + if (k = c ? d : d[h]) { + for (var l = 0, m, n; l < b.length; l++) { + if (n = b[l], m = k.indexOf(g ? parseInt(n, 10) : n), 0 > m && !g && "string" === typeof n && !isNaN(n) && (m = k.indexOf(parseInt(n, 10))) && (g = 1), 0 <= m) { + if (e = 1, 1 < k.length) { + k.splice(m, 1); + } else { + d[h] = []; + break; + } + } + } + f += k.length; + } + if (c) { + break; } } - return b; + f ? e && a.update(d) : a.delete(); + a.continue(); } -;var Qa = {encode:Pa, G:!1, H:""}, Ra = [J("ae"), "a", J("oe"), "o", J("sh"), "s", J("th"), "t", J("ph"), "f", J("pf"), "f", J("(?![aeo])h(?![aeo])"), "", J("(?!^[aeo])h(?!^[aeo])"), ""]; -function Pa(a, b) { - a && (a = Ia.call(this, a).join(" "), 2 < a.length && (a = I(a, Ra)), b || (1 < a.length && (a = na(a)), a && (a = a.split(" ")))); - return a || []; +w.remove = function(a) { + "object" !== typeof a && (a = [a]); + return Promise.all([this.transaction("map", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + var c = this.result; + c && zb(c, a); + }; + }), this.transaction("ctx", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + var c = this.result; + c && zb(c, a); + }; + }), this.transaction("tag", "readwrite", function(b) { + b.openCursor().onsuccess = function() { + var c = this.result; + c && zb(c, a, !0); + }; + }), this.transaction("reg", "readwrite", function(b) { + for (var c = 0; c < a.length; c++) { + b.delete(a[c]); + } + })]); +}; +function yb(a) { + return new Promise(function(b, c) { + a.onsuccess = function() { + b(this.result); + }; + a.oncomplete = function() { + b(this.result); + }; + a.onerror = c; + a = null; + }); } -;var Ta = {encode:Sa, G:!1, H:""}, Ua = J("(?!\\b)[aeo]"); -function Sa(a) { - a && (a = Pa.call(this, a, !0), 1 < a.length && (a = a.replace(Ua, "")), 1 < a.length && (a = na(a)), a && (a = a.split(" "))); - return a || []; -} -;K["latin:default"] = pa; -K["latin:simple"] = Ga; -K["latin:balance"] = Ja; -K["latin:advanced"] = Qa; -K["latin:extra"] = Ta; -var Z = {Index:N, Document:T, Worker:R, registerCharset:function(a, b) { - K[a] = b; -}, registerLanguage:function(a, b) { - qa[a] = b; -}}, Va; -(Va = self.define) && Va.amd ? Va([], function() { - return Z; -}) : self.exports ? self.exports = Z : self.FlexSearch = Z; +;var Ab = new Map([["b", "p"], ["v", "f"], ["w", "f"], ["z", "s"], ["x", "s"], ["d", "t"], ["n", "m"], ["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], ["i", "e"], ["y", "e"], ["u", "o"]]), Bb = {normalize:!0, H:!0, I:Ab}; +var Cb = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]), Db = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"], Eb = {normalize:!0, H:!0, I:Ab, P:Db, N:Cb}; +var Hb = {normalize:!0, H:!0, I:Ab, P:Db.concat([/(?!^)[aeoy]/g, ""]), N:Cb}; +var Ib = {a:"", e:"", i:"", o:"", u:"", y:"", b:1, f:1, p:1, v:1, c:2, g:2, j:2, k:2, q:2, s:2, x:2, z:2, "\u00df":2, d:3, t:3, l:4, m:5, n:5, r:6}; +T["latin:exact"] = {normalize:!1, H:!1}; +T["latin:default"] = Sa; +T["latin:simple"] = {normalize:!0, H:!0}; +T["latin:balance"] = Bb; +T["latin:advanced"] = Eb; +T["latin:extra"] = Hb; +T["latin:soundex"] = {normalize:!0, H:!1, ra:{Ea:!0}, ca:function(a) { + for (var b = 0; b < a.length; b++) { + for (var c = a[b], d = c.charAt(0), e = Ib[d], g = 1, f; g < c.length && (f = c.charAt(g), "h" === f || "w" === f || !(f = Ib[f]) || f === e || (d += f, e = f, 4 !== d.length)); g++) { + } + a[b] = d; + } +}}; +var Jb = {Index:U, Encoder:M, Charset:T, Language:Ra, Document:Z, Worker:Y, Resolver:W, IndexedDB:wb}, Kb = self, Lb; +(Lb = Kb.define) && Lb.amd ? Lb([], function() { + return Jb; +}) : "object" === typeof Kb.exports ? Kb.exports = Jb : Kb.FlexSearch = Jb; }(this)); diff --git a/dist/flexsearch.es5.min.js b/dist/flexsearch.es5.min.js index 0f1857e..59c6267 100644 --- a/dist/flexsearch.es5.min.js +++ b/dist/flexsearch.es5.min.js @@ -1,48 +1,123 @@ /**! - * FlexSearch.js v0.7.41 (Es5) + * FlexSearch.js v0.8.0 (ES5) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -(function(self){'use strict';var t;function aa(a){var b=0;return function(){return b>>0)+"_",e=0;return b}); -y("Symbol.iterator",function(a){if(a)return a;a=Symbol("Symbol.iterator");for(var b="Array Int8Array Uint8Array Uint8ClampedArray Int16Array Uint16Array Int32Array Uint32Array Float32Array Float64Array".split(" "),c=0;cc&&(c=Math.max(c+e,0));c=this.C&&(f||!g[l])){var p=O(h,d,k),n="";switch(this.H){case "full":if(2p;q--)if(q-p>=this.C){var r=O(h,d,k,m,p);n=l.substring(p,q);P(this,g,n,r,a,c)}break}case "reverse":if(1=this.C&&P(this,g,n,O(h,d,k, -m,q),a,c);n=""}case "forward":if(1=this.C&&P(this,g,n,p,a,c);break}default:if(this.o&&(p=Math.min(p/this.o(b,l,k)|0,h-1)),P(this,g,l,p,a,c),f&&1=this.C&&!m[l]){m[l]=1;var u=this.l&&l>p;P(this,e,u?p:l,O(n+(d/2>n?0:1),d,k,q-1,r-1),a,c,u?l:p)}}}}this.s||(this.register[a]=1)}}return this}; -function O(a,b,c,d,e){return c&&1=this.C&&!c[p])if(this.B||f||this.map[p])k[m++]=p,c[p]=1;else return d;a=k;h=a.length}}if(!h)return d;b||(b=100);g=this.depth&&1=d)))break;if(m){if(g)return ya(k,d,0);b[b.length]=k;return}}return!c&&k}function ya(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function za(a,b,c,d){c?(d=d&&b>c,a=(a=a[d?b:c])&&a[d?c:b]):a=a[b];return a}t.contain=function(a){return!!this.register[a]};t.update=function(a,b){return this.remove(a).add(a,b)}; -t.remove=function(a,b){var c=this.register[a];if(c){if(this.s)for(var d=0,e;db||c)e=e.slice(c,c+b);d&&(e=Ea.call(this,e));return{tag:a,result:e}}}function Ea(a){for(var b=Array(a.length),c=0,d;c>>0)+"_",e=0;return b}); +B("Symbol.iterator",function(a){if(a)return a;a=Symbol("Symbol.iterator");for(var b="Array Int8Array Uint8Array Uint8ClampedArray Int16Array Uint16Array Int32Array Uint32Array Float32Array Float64Array".split(" "),c=0;cc&&(c=Math.max(c+e,0));cthis.R&&(this.ba.clear(),this.A=this.A/1.1|0));f&&d.push(f)}this.ca&&(d=this.ca(d)||d);this.cache&&a.length<=this.h&&(this.W.set(a,d),this.W.size>this.R&&(this.W.clear(),this.h=this.h/1.1|0));return d};function Ia(a){a.ea=null;a.W.clear();a.ba.clear()};function Ja(a,b,c){a=("object"===typeof a?""+a.query:a).toLowerCase();var d=this.cache.get(a);if(!d){d=this.search(a,b,c);if(d instanceof Promise){var e=this;d.then(function(h){e.cache.set(a,h)})}this.cache.set(a,d)}return d}function N(a){this.limit=a&&!0!==a?a:1E3;this.cache=new Map;this.h=""}N.prototype.set=function(a,b){this.cache.has(a)||(this.cache.set(this.h=a,b),this.limit&&this.cache.size>this.limit&&this.cache.delete(this.cache.keys().next().value))}; +N.prototype.get=function(a){var b=this.cache.get(a);b&&this.limit&&this.h!==a&&(this.cache.delete(a),this.cache.set(this.h=a,b));return b};N.prototype.remove=function(a){for(var b=y(this.cache),c=b.next();!c.done;c=b.next()){c=c.value;var d=c[0];c[1].includes(a)&&this.cache.delete(d)}};N.prototype.clear=function(){this.cache.clear();this.h=""};function Ka(a,b,c,d){for(var e=[],h=0,f;h=f.length)b-=f.length;else{b=f[d?"splice":"slice"](b,c);if(f=b.length)if(e=e.length?e.concat(b):b,c-=f,d&&(a.length-=f),!c)break;b=0}return e} +function Q(a){if(!(this instanceof Q))return new Q(a);this.index=a?[a]:[];this.length=a?a.length:0;var b=this;return new Proxy([],{get:function(c,d){if("length"===d)return b.length;if("push"===d)return function(e){b.index[b.index.length-1].push(e);b.length++};if("pop"===d)return function(){if(b.length)return b.length--,b.index[b.index.length-1].pop()};if("indexOf"===d)return function(e){for(var h=0,f=0,g,k;fn;q--){p=l.substring(n,q);var r=this.score?this.score(b,l,k,p,n):Wa(g,d,k,m,n);Xa(this,h,p,r,a,c)}break}case "reverse":if(1< +m){for(q=m-1;0p?0:1),d,k,q-1,r-1),u=this.bidirectional&&l>n;Xa(this,e,u?n:l,x,a,c,u?l:n)}}}}this.fastupdate||this.B.add(a)}else b=""}this.db&& +(b||this.X.push({del:a}),this.qa&&Ya(this));return this}; +function Xa(a,b,c,d,e,h,f){var g=f?a.M:a.map,k;if(!b[c]||!f||!(k=b[c])[f])if(f?(b=k||(b[c]=I()),b[f]=1,(k=g.get(f))?g=k:g.set(f,g=new Map)):b[c]=1,(k=g.get(c))?g=k:g.set(c,g=k=[]),g=g[d]||(g[d]=[]),!h||!g.includes(e)){if(g.length===Math.pow(2,31)-1){b=new Q(g);if(a.fastupdate)for(c=y(a.B.values()),h=c.next();!h.done;h=c.next())h=h.value,h.includes(g)&&(h[h.indexOf(g)]=b);k[d]=g=b}g.push(e);a.fastupdate&&((d=a.B.get(e))?d.push(g):a.B.set(e,[g]))}} +function Wa(a,b,c,d,e){return c&&1b?b?a.slice(c,c+b):a.slice(c):a,d?Za(a):a;for(var e=[],h=0,f=void 0,g=void 0;h=g){c-=g;continue}cb&&(f=f.slice(0,b),g=f.length),e.push(f);else{if(g>=b)return g>b&&(f=f.slice(0,b)),d?Za(f):f;e=[f]}b-=g;if(!b)break}if(!e.length)return e;e=1a.length)return e?V(a[0],b,c,d):a[0];d=[];for(var f=0,g=I(),k=Ba(a),l=0,m;la.length)return[];var h=[],f=0,g=I(),k=Ba(a);if(!k)return h;for(var l=0,m;la.length)return e?V(a[0],b,c,d):a[0];b=[];c=I();d=0;for(var f;d=e)))break;if(g.length){if(f)return V(g,e,0);b.push(g);return}}return!c&&g}function X(a,b,c,d,e,h,f,g){var k;c&&(k=a.bidirectional&&b>c);if(a.db)return c?a.db.get(k?c:b,k?b:c,d,e,h,f,g):a.db.get(b,"",d,e,h,f,g);a=c?(a=a.M.get(k?b:c))&&a.get(k?c:b):a.map.get(b);return a};U.prototype.remove=function(a,b){var c=this.B.size&&(this.fastupdate?this.B.get(a):this.B.has(a));if(c){if(this.fastupdate)for(var d=0,e;de.length)e.pop();else{var h=e.indexOf(a);h===c.length-1?e.pop():e.splice(h,1)}}else ib(this.map,a),this.depth&&ib(this.M,a);b||this.B.delete(a)}this.db&&(this.X.push({del:a}),this.qa&&Ya(this));this.cache&&this.cache.remove(a);return this}; +function ib(a,b){var c=0;if(a.constructor===Array)for(var d=0,e=void 0,h;dc||d)a=a.slice(d,d+c);e&&(a=rb.call(this,a));return a}}function rb(a){for(var b=Array(a.length),c=0,d;c=m.length)d-=m.length;else{for(var n=c?d+Math.min(m.length-d,c):m.length,p=d;p=h.length)return[];if(!b&&!c)return h;h=h.slice(c,c+b);return d?e.enrich(h):h})}; +w.enrich=function(a){"object"!==typeof a&&(a=[a]);for(var b=this.db.transaction("reg","readonly").objectStore("reg"),c=[],d=0;dm&&!h&&"string"===typeof n&&!isNaN(n)&&(m=k.indexOf(parseInt(n,10)))&&(h=1),0<=m)if(e=1,1 this.s.get(r)), l = 1); + this.m && 1 < f.length && (this.I || (this.I = new RegExp("(" + this.i + ")", "g")), f = f.replace(this.I, r => this.m.get(r)), l = 1); + f && l && (f.length < this.G || this.filter && this.filter.has(f)) && (f = ""); + if (f && (this.v || this.l && 1 < f.length)) { + e = ""; + for (let r = 0, k = "", p, q; r < f.length; r++) { + p = f.charAt(r), p === k && this.l || ((q = this.v && this.v.get(p)) || "" === q ? q === k && this.l || !(k = q) || (e += q) : e += k = p); + } + f = e; + } + if (f && this.o) { + for (e = 0; f && e < this.o.length; e += 2) { + f = f.replace(this.o[e], this.o[e + 1]); + } + } + this.cache && h.length <= this.h && (this.A.set(h, f), this.A.size > this.L && (this.A.clear(), this.h = this.h / 1.1 | 0)); + f && b.push(f); + } + this.D && (b = this.D(b) || b); + this.cache && a.length <= this.g && (this.u.set(a, b), this.u.size > this.L && (this.u.clear(), this.g = this.g / 1.1 | 0)); + return b; +}; +function E(a) { + a.C = null; + a.u.clear(); + a.A.clear(); +} +;function F(a) { + this.limit = a && !0 !== a ? a : 1000; + this.cache = new Map(); + this.g = ""; +} +F.prototype.set = function(a, c) { + this.cache.has(a) || (this.cache.set(this.g = a, c), this.limit && this.cache.size > this.limit && this.cache.delete(this.cache.keys().next().value)); +}; +F.prototype.get = function(a) { + const c = this.cache.get(a); + c && this.limit && this.g !== a && (this.cache.delete(a), this.cache.set(this.g = a, c)); + return c; +}; +F.prototype.remove = function(a) { + for (const c of this.cache) { + const b = c[0]; + c[1].includes(a) && this.cache.delete(b); + } +}; +F.prototype.clear = function() { + this.cache.clear(); + this.g = ""; +}; +function G(a = 8) { + if (!(this instanceof G)) { + return new G(a); + } + this.index = v(); + this.i = []; + this.size = 0; + 32 < a ? (this.g = H, this.h = BigInt(a)) : (this.g = I, this.h = a); +} +G.prototype.get = function(a) { + const c = this.index[this.g(a)]; + return c && c.get(a); +}; +G.prototype.set = function(a, c) { + var b = this.g(a); + let d = this.index[b]; + d ? (b = d.size, d.set(a, c), (b -= d.size) && this.size++) : (this.index[b] = d = new Map([[a, c]]), this.i.push(d)); +}; +function J(a = 8) { + if (!(this instanceof J)) { + return new J(a); + } + this.index = v(); + this.g = []; + 32 < a ? (this.i = H, this.h = BigInt(a)) : (this.i = I, this.h = a); +} +J.prototype.add = function(a) { + var c = this.i(a); + let b = this.index[c]; + b ? (c = b.size, b.add(a), (c -= b.size) && this.size++) : (this.index[c] = b = new Set([a]), this.g.push(b)); +}; +t = G.prototype; +t.has = J.prototype.has = function(a) { + const c = this.index[this.i(a)]; + return c && c.has(a); +}; +t.delete = J.prototype.delete = function(a) { + const c = this.index[this.i(a)]; + c && c.delete(a) && this.size--; +}; +t.clear = J.prototype.clear = function() { + this.index = v(); + this.g = []; + this.size = 0; +}; +t.values = J.prototype.values = function*() { + for (let a = 0; a < this.g.length; a++) { + for (let c of this.g[a].values()) { + yield c; + } + } +}; +t.keys = J.prototype.keys = function*() { + for (let a = 0; a < this.g.length; a++) { + for (let c of this.g[a].keys()) { + yield c; + } + } +}; +t.entries = J.prototype.entries = function*() { + for (let a = 0; a < this.g.length; a++) { + for (let c of this.g[a].entries()) { + yield c; + } + } +}; +function I(a) { + let c = 2 ** this.h - 1; + if ("number" == typeof a) { + return a & c; + } + let b = 0, d = this.h + 1; + for (let e = 0; e < a.length; e++) { + b = (b * d ^ a.charCodeAt(e)) & c; + } + return 32 === this.h ? b + 2 ** 31 : b; +} +function H(a) { + let c = BigInt(2) ** this.h - BigInt(1); + var b = typeof a; + if ("bigint" === b) { + return a & c; + } + if ("number" === b) { + return BigInt(a) & c; + } + b = BigInt(0); + let d = this.h + BigInt(1); + for (let e = 0; e < a.length; e++) { + b = (b * d ^ BigInt(a.charCodeAt(e))) & c; } return b; } -function y() { - return Object.create(null); -} -function z(a, b) { - return b.length - a.length; -} -;const A = /[\p{Z}\p{S}\p{P}\p{C}]+/u; -function B(a, b) { - const c = Object.keys(a), d = c.length, e = []; - let h = "", f = 0; - for (let g = 0, l, n; g < d; g++) { - l = c[g], (n = a[l]) ? (e[f++] = new RegExp(b ? "(?!\\b)" + l + "(\\b|_)" : l, "g"), e[f++] = n) : h += (h ? "|" : "") + l; - } - h && (e[f++] = new RegExp(b ? "(?!\\b)(" + h + ")(\\b|_)" : "(" + h + ")", "g"), e[f] = ""); - return e; -} -function C(a, b) { - for (let c = 0, d = b.length; c < d && (a = a.replace(b[c], b[c + 1]), a); c += 2) { - } - return a; -} -;function E(a) { - if (a = ("" + a).toLowerCase()) { - if (this.o && (a = C(a, this.o)), this.A && 1 < a.length && (a = C(a, this.A)), A || "" === A) { - const b = a.split(A); - if (this.filter) { - a = this.filter; - const c = b.length, d = []; - for (let e = 0, h = 0; e < c; e++) { - const f = b[e]; - f && !a[f] && (d[h++] = f); - } - a = d; - } else { - a = b; - } +;const K = v(), L = v(); +var aa = {normalize:function(a) { + return a.toLowerCase(); +}, l:!1}; +const M = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +v(); +N.prototype.add = function(a, c, b, d) { + if (c && (a || 0 === a)) { + if (!d && !b && this.j.has(a)) { + return this.update(a, c); } - } - return a; -} -;const F = {}, G = {}; -function H(a, b, c, d) { - const e = a.length; - let h = [], f, g, l = 0; - d && (d = []); - for (let n = e - 1; 0 <= n; n--) { - const m = a[n], r = m.length, p = y(); - let q = !f; - for (let k = 0; k < r; k++) { - const u = m[k], L = u.length; - if (L) { - for (let D = 0, x, w; D < L; D++) { - if (w = u[D], f) { - if (f[w]) { - if (!n) { - if (c) { - c--; - } else { - if (h[l++] = w, l === b) { - return h; - } - } - } - if (n || d) { - p[w] = 1; - } - q = !0; - } - if (d && (x = (g[w] || 0) + 1, g[w] = x, x < e)) { - const M = d[x - 2] || (d[x - 2] = []); - M[M.length] = w; - } - } else { - p[w] = 1; - } - } - } - } - if (d) { - f || (g = p); - } else if (!q) { - return []; - } - f = p; - } - if (d) { - for (let n = d.length - 1, m, r; 0 <= n; n--) { - m = d[n]; - r = m.length; - for (let p = 0, q; p < r; p++) { - if (q = m[p], !f[q]) { - if (c) { - c--; - } else { - if (h[l++] = q, l === b) { - return h; - } - } - f[q] = 1; - } - } - } - } - return h; -} -;function I(a, b) { - if (!(this instanceof I)) { - return new I(a); - } - let c; - if (a) { - var d = a.charset; - c = a.lang; - "string" === typeof d && (-1 === d.indexOf(":") && (d += ":default"), d = G[d]); - "string" === typeof c && (c = F[c]); - } else { - a = {}; - } - let e, h, f = a.context || {}; - this.encode = a.encode || d && d.encode || E; - this.register = b || y(); - this.s = e = a.resolution || 9; - this.B = b = d && d.B || a.tokenize || "strict"; - this.i = "strict" === b && f.depth; - this.j = t(f.bidirectional); - this.g = h = t(a.optimize); - this.m = t(a.fastupdate); - this.h = a.minlength || 1; - this.C = a.boost; - this.map = h ? v(e) : y(); - this.v = e = f.resolution || 1; - this.l = h ? v(e) : y(); - this.u = d && d.u || a.rtl; - this.o = (b = a.matcher || c && c.o) && B(b, !1); - this.A = (b = a.stemmer || c && c.A) && B(b, !0); - if (a = b = a.filter || c && c.filter) { - a = b; - d = y(); - for (let g = 0, l = a.length; g < l; g++) { - d[a[g]] = 1; - } - a = d; - } - this.filter = a; -} -I.prototype.append = function(a, b) { - return this.add(a, b, !0); -}; -I.prototype.add = function(a, b, c, d) { - if (b && (a || 0 === a)) { - if (!d && !c && this.register[a]) { - return this.update(a, b); - } - b = this.encode(b); - if (d = b.length) { - const n = y(), m = y(), r = this.i, p = this.s; - for (let q = 0; q < d; q++) { - let k = b[this.u ? d - 1 - q : q]; - var e = k.length; - if (k && e >= this.h && (r || !m[k])) { - var h = J(p, d, q), f = ""; - switch(this.B) { + c = this.encoder.encode(c); + if (d = c.length) { + const r = v(), k = v(), p = this.depth, q = this.resolution; + for (let m = 0; m < d; m++) { + let n = c[this.rtl ? d - 1 - m : m]; + var e = n.length; + if (e && (p || !k[n])) { + var g = this.score ? this.score(c, n, m, null, 0) : O(q, d, m), f = ""; + switch(this.tokenize) { case "full": if (2 < e) { - for (h = 0; h < e; h++) { - for (var g = e; g > h; g--) { - if (g - h >= this.h) { - var l = J(p, d, q, e, h); - f = k.substring(h, g); - K(this, m, f, l, a, c); - } + for (g = 0; g < e; g++) { + for (var h = e; h > g; h--) { + f = n.substring(g, h); + var l = this.score ? this.score(c, n, m, f, g) : O(q, d, m, e, g); + P(this, k, f, l, a, b); } } break; } case "reverse": if (1 < e) { - for (g = e - 1; 0 < g; g--) { - f = k[g] + f, f.length >= this.h && K(this, m, f, J(p, d, q, e, g), a, c); + for (h = e - 1; 0 < h; h--) { + f = n[h] + f, l = this.score ? this.score(c, n, m, f, h) : O(q, d, m, e, h), P(this, k, f, l, a, b); } f = ""; } case "forward": if (1 < e) { - for (g = 0; g < e; g++) { - f += k[g], f.length >= this.h && K(this, m, f, h, a, c); + for (h = 0; h < e; h++) { + f += n[h], P(this, k, f, g, a, b); } break; } default: - if (this.C && (h = Math.min(h / this.C(b, k, q) | 0, p - 1)), K(this, m, k, h, a, c), r && 1 < d && q < d - 1) { - for (e = y(), f = this.v, h = k, g = Math.min(r + 1, d - q), e[h] = 1, l = 1; l < g; l++) { - if ((k = b[this.u ? d - 1 - q - l : q + l]) && k.length >= this.h && !e[k]) { - e[k] = 1; - const u = this.j && k > h; - K(this, n, u ? h : k, J(f + (d / 2 > f ? 0 : 1), d, q, g - 1, l - 1), a, c, u ? k : h); + if (P(this, k, n, g, a, b), p && 1 < d && m < d - 1) { + for (e = v(), f = this.M, g = n, h = Math.min(p + 1, d - m), e[g] = 1, l = 1; l < h; l++) { + if ((n = c[this.rtl ? d - 1 - m - l : m + l]) && !e[n]) { + e[n] = 1; + const x = this.score ? this.score(c, g, m, n, l) : O(f + (d / 2 > f ? 0 : 1), d, m, h - 1, l - 1), R = this.bidirectional && n > g; + P(this, r, R ? g : n, x, a, b, R ? n : g); } } } } } } - this.m || (this.register[a] = 1); + this.fastupdate || this.j.add(a); } } return this; }; -function J(a, b, c, d, e) { - return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; +function P(a, c, b, d, e, g, f) { + let h = f ? a.B : a.map, l; + c[b] && f && (l = c[b])[f] || (f ? (c = l || (c[b] = v()), c[f] = 1, (l = h.get(f)) ? h = l : h.set(f, h = new Map())) : c[b] = 1, (l = h.get(b)) ? h = l : h.set(b, h = []), h = h[d] || (h[d] = []), g && h.includes(e) || (h.push(e), a.fastupdate && ((c = a.j.get(e)) ? c.push(h) : a.j.set(e, [h])))); } -function K(a, b, c, d, e, h, f) { - let g = f ? a.l : a.map; - if (!b[c] || f && !b[c][f]) { - a.g && (g = g[d]), f ? (b = b[c] || (b[c] = y()), b[f] = 1, g = g[f] || (g[f] = y())) : b[c] = 1, g = g[c] || (g[c] = []), a.g || (g = g[d] || (g[d] = [])), h && g.includes(e) || (g[g.length] = e, a.m && (a = a.register[e] || (a.register[e] = []), a[a.length] = g)); - } +function O(a, c, b, d, e) { + return b && 1 < a ? c + (d || 0) <= a ? b + (e || 0) : (a - 1) / (c + (d || 0)) * (b + (e || 0)) + 1 | 0 : 0; } -I.prototype.search = function(a, b, c) { - c || (b || "object" !== typeof a ? "object" === typeof b && (c = b) : (c = a, a = c.query)); - let d = [], e; - let h, f = 0; - if (c) { - a = c.query || a; - b = c.limit; - f = c.offset || 0; - var g = c.context; - h = !1; +;function Q(a, c, b) { + if (1 === a.length) { + return a = a[0], a = b || a.length > c ? c ? a.slice(b, b + c) : a.slice(b) : a; } - if (a && (a = this.encode("" + a), e = a.length, 1 < e)) { - c = y(); - var l = []; - for (let m = 0, r = 0, p; m < e; m++) { - if ((p = a[m]) && p.length >= this.h && !c[p]) { - if (this.g || h || this.map[p]) { - l[r++] = p, c[p] = 1; - } else { - return d; + let d = []; + for (let e = 0, g, f; e < a.length; e++) { + if ((g = a[e]) && (f = g.length)) { + if (b) { + if (b >= f) { + b -= f; + continue; + } + b < f && (g = c ? g.slice(b, b + c) : g.slice(b), f = g.length, b = 0); + } + if (d.length) { + f > c && (g = g.slice(0, c), f = g.length), d.push(g); + } else { + if (f >= c) { + return f > c && (g = g.slice(0, c)), g; + } + d = [g]; + } + c -= f; + if (!c) { + break; + } + } + } + return d.length ? d = 1 < d.length ? [].concat.apply([], d) : d[0] : d; +} +;function ba(a, c, b, d) { + var e = a.length; + let g = [], f = 0, h, l, r; + d && (d = []); + for (let k = e - 1, p; 0 <= k; k--) { + r = a[k]; + e = v(); + p = !h; + for (let q = 0, m; q < r.length; q++) { + if ((m = r[q]) && m.length) { + for (let n = 0, x; n < m.length; n++) { + if (x = m[n], h) { + if (h[x]) { + if (!k) { + if (b) { + b--; + } else { + if (g[f++] = x, f === c) { + return g; + } + } + } + if (k || d) { + e[x] = 1; + } + p = !0; + } + d && !l[x] && (l[x] = 1, (d[q] || (d[q] = [])).push(x)); + } else { + e[x] = 1; + } } } } - a = l; - e = a.length; + if (d) { + h || (l = e); + } else if (!p) { + return []; + } + h = e; } - if (!e) { + if (d) { + for (let k = d.length - 1, p, q; 0 <= k; k--) { + p = d[k]; + q = p.length; + for (let m = 0, n; m < q; m++) { + if (n = p[m], !h[n]) { + if (b) { + b--; + } else { + if (g[f++] = n, f === c) { + return g; + } + } + h[n] = 1; + } + } + } + } + return g; +} +;N.prototype.search = function(a, c, b) { + b || (c || "object" !== typeof a ? "object" === typeof c && (b = c, c = 0) : (b = a, a = "")); + let d = []; + let e, g = 0; + if (b) { + a = b.query || a; + c = b.limit || c; + g = b.offset || 0; + var f = b.context; + e = b.suggest; + } + a = this.encoder.encode(a); + b = a.length; + c || (c = 100); + if (1 === b) { + return S.call(this, a[0], "", c, g); + } + f = this.depth && !1 !== f; + if (2 === b && f && !e) { + return S.call(this, a[0], a[1], c, g); + } + let h = 0, l = 0; + if (1 < b) { + var r = v(); + const p = []; + for (let q = 0, m; q < b; q++) { + if ((m = a[q]) && !r[m]) { + if (e || T(this, m)) { + p.push(m), r[m] = 1; + } else { + return d; + } + const n = m.length; + h = Math.max(h, n); + l = l ? Math.min(l, n) : n; + } + } + a = p; + b = a.length; + } + if (!b) { return d; } - b || (b = 100); - g = this.i && 1 < e && !1 !== g; - c = 0; - let n; - g ? (n = a[0], c = 1) : 1 < e && a.sort(z); - for (let m, r; c < e; c++) { - r = a[c]; - g ? (m = N(this, d, h, b, f, 2 === e, r, n), h && !1 === m && d.length || (n = r)) : m = N(this, d, h, b, f, 1 === e, r); - if (m) { - return m; + r = 0; + let k; + if (1 === b) { + return S.call(this, a[0], "", c, g); + } + if (2 === b && f && !e) { + return S.call(this, a[0], a[1], c, g); + } + 1 < b && (f ? (k = a[0], r = 1) : 9 < h && 3 < h / l && a.sort(w)); + for (let p, q; r < b; r++) { + q = a[r]; + k ? (p = T(this, q, k), p = U(p, d, e, this.M, c, g, 2 === b), e && !1 === p && d.length || (k = q)) : (p = T(this, q), p = U(p, d, e, this.resolution, c, g, 1 === b)); + if (p) { + return p; } - if (h && c === e - 1) { - l = d.length; - if (!l) { - if (g) { - g = 0; - c = -1; + if (e && r === b - 1) { + f = d.length; + if (!f) { + if (k) { + k = ""; + r = -1; continue; } return d; } - if (1 === l) { - return O(d[0], b, f); + if (1 === f) { + return Q(d[0], c, g); } } } - return H(d, b, f, h); + return ba(d, c, g, e); }; -function N(a, b, c, d, e, h, f, g) { - let l = [], n = g ? a.l : a.map; - a.g || (n = P(n, f, g, a.j)); - if (n) { - let m = 0; - const r = Math.min(n.length, g ? a.v : a.s); - for (let p = 0, q = 0, k, u; p < r; p++) { - if (k = n[p]) { - if (a.g && (k = P(k, f, g, a.j)), e && k && h && (u = k.length, u <= e ? (e -= u, k = null) : (k = k.slice(e), e = 0)), k && (l[m++] = k, h && (q += k.length, q >= d))) { +function S(a, c, b, d) { + return (a = T(this, a, c)) && a.length ? Q(a, b, d) : []; +} +function U(a, c, b, d, e, g, f) { + let h = []; + if (a) { + d = Math.min(a.length, d); + for (let l = 0, r = 0, k; l < d; l++) { + if (k = a[l]) { + if (g && k && f && (k.length <= g ? (g -= k.length, k = null) : (k = k.slice(g), g = 0)), k && (h[l] = k, f && (r += k.length, r >= e))) { break; } } } - if (m) { - if (h) { - return O(l, d, 0); + if (h.length) { + if (f) { + return Q(h, e, 0); } - b[b.length] = l; + c.push(h); return; } } - return !c && l; + return !b && h; } -function O(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function P(a, b, c, d) { - c ? (d = d && b > c, a = (a = a[d ? b : c]) && a[d ? c : b]) : a = a[b]; +function T(a, c, b) { + let d; + b && (d = a.bidirectional && c > b); + a = b ? (a = a.B.get(d ? c : b)) && a.get(d ? b : c) : a.map.get(c); return a; } -I.prototype.contain = function(a) { - return !!this.register[a]; -}; -I.prototype.update = function(a, b) { - return this.remove(a).add(a, b); -}; -I.prototype.remove = function(a, b) { - const c = this.register[a]; - if (c) { - if (this.m) { - for (let d = 0, e; d < c.length; d++) { - e = c[d], e.splice(e.indexOf(a), 1); +;N.prototype.remove = function(a, c) { + const b = this.j.size && (this.fastupdate ? this.j.get(a) : this.j.has(a)); + if (b) { + if (this.fastupdate) { + for (let d = 0, e; d < b.length; d++) { + if (e = b[d]) { + if (2 > e.length) { + e.pop(); + } else { + const g = e.indexOf(a); + g === b.length - 1 ? e.pop() : e.splice(g, 1); + } + } } } else { - Q(this.map, a, this.s, this.g), this.i && Q(this.l, a, this.v, this.g); + V(this.map, a), this.depth && V(this.B, a); } - b || delete this.register[a]; + c || this.j.delete(a); } + this.cache && this.cache.remove(a); return this; }; -function Q(a, b, c, d, e) { - let h = 0; +function V(a, c) { + let b = 0; if (a.constructor === Array) { - if (e) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), h++) : h++; - } else { - e = Math.min(a.length, c); - for (let f = 0, g; f < e; f++) { - if (g = a[f]) { - h = Q(g, b, c, d, e), d || h || delete a[f]; + for (let d = 0, e, g; d < a.length; d++) { + if ((e = a[d]) && e.length) { + if (g = e.indexOf(c), 0 <= g) { + 1 < e.length ? (e.splice(g, 1), b++) : delete a[d]; + break; + } else { + b++; } } } } else { - for (let f in a) { - (h = Q(a[f], b, c, d, e)) || delete a[f]; + for (let d of a) { + const e = d[0], g = V(d[1], c); + g ? b += g : a.delete(e); } } - return h; + return b; } -;const R = {Index:I, Document:null, Worker:null, registerCharset:function(a, b) { - G[a] = b; -}, registerLanguage:function(a, b) { - F[a] = b; -}}; -let S; -(S = self.define) && S.amd ? S([], function() { - return R; -}) : self.exports ? self.exports = R : self.FlexSearch = R; +;function N(a, c) { + if (!(this instanceof N)) { + return new N(a); + } + if (a) { + var b = "string" === typeof a ? a : a.preset; + b && (M[b] || console.warn("Preset not found: " + b), a = Object.assign({}, M[b], a)); + } else { + a = {}; + } + b = a.context || {}; + const d = a.encode || a.encoder || aa; + this.encoder = d.encode ? d : "object" === typeof d ? new D(d) : {encode:d}; + let e; + this.resolution = a.resolution || 9; + this.tokenize = e = a.tokenize || "strict"; + this.depth = "strict" === e && b.depth || 0; + this.bidirectional = !1 !== b.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + this.map = (e = !1, new Map()); + this.B = e ? new G(e) : new Map(); + this.j = c || (this.fastupdate ? e ? new G(e) : new Map() : e ? new J(e) : new Set()); + this.M = b.resolution || 1; + this.rtl = d.rtl || a.rtl || !1; + this.cache = (e = a.cache || null) && new F(e); +} +t = N.prototype; +t.clear = function() { + this.map.clear(); + this.B.clear(); + this.j.clear(); + this.cache && this.cache.clear(); + return this; +}; +t.append = function(a, c) { + return this.add(a, c, !0); +}; +t.contain = function(a) { + return this.j.has(a); +}; +t.update = function(a, c) { + if (this.async) { + const b = this, d = this.remove(a); + return d.then ? d.then(() => b.add(a, c)) : this.add(a, c); + } + return this.remove(a).add(a, c); +}; +function W(a) { + let c = 0; + if (a.constructor === Array) { + for (let b = 0, d; b < a.length; b++) { + (d = a[b]) && (c += d.length); + } + } else { + for (const b of a) { + const d = b[0], e = W(b[1]); + e ? c += e : a.delete(d); + } + } + return c; +} +t.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + W(this.map); + this.depth && W(this.B); + return this; +}; +t.searchCache = function(a, c, b) { + a = ("object" === typeof a ? "" + a.query : a).toLowerCase(); + let d = this.cache.get(a); + if (!d) { + d = this.search(a, c, b); + if (d instanceof Promise) { + const e = this; + d.then(function(g) { + e.cache.set(a, g); + }); + } + this.cache.set(a, d); + } + return d; +}; +const X = {Index:N, Encoder:D, Charset:L, Language:K, Document:null, Worker:null, Resolver:null, IndexedDB:null}, Y = self; +let Z; +(Z = Y.define) && Z.amd ? Z([], function() { + return X; +}) : "object" === typeof Y.exports ? Y.exports = X : Y.FlexSearch = X; }(this)); diff --git a/dist/flexsearch.light.min.js b/dist/flexsearch.light.min.js index 26454d2..f212104 100644 --- a/dist/flexsearch.light.min.js +++ b/dist/flexsearch.light.min.js @@ -1,18 +1,28 @@ /**! - * FlexSearch.js v0.7.41 (Light) + * FlexSearch.js v0.8.0 (Light) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -(function(self){'use strict';function t(a){return"undefined"!==typeof a?a:!0}function v(a){const b=Array(a);for(let c=0;c=this.h&&(r||!m[k])){var h=J(p,d,q),f="";switch(this.B){case "full":if(2h;g--)if(g-h>=this.h){var l=J(p,d,q,e,h);f=k.substring(h,g);K(this,m,f,l,a,c)}break}case "reverse":if(1=this.h&&K(this, -m,f,J(p,d,q,e,g),a,c);f=""}case "forward":if(1=this.h&&K(this,m,f,h,a,c);break}default:if(this.C&&(h=Math.min(h/this.C(b,k,q)|0,p-1)),K(this,m,k,h,a,c),r&&1=this.h&&!e[k]){e[k]=1;const u=this.j&&k>h;K(this,n,u?h:k,J(f+(d/2>f?0:1),d,q,g-1,l-1),a,c,u?k:h)}}}}this.m||(this.register[a]=1)}}return this}; -function J(a,b,c,d,e){return c&&1=this.h&&!c[p])if(this.g||h||this.map[p])l[r++]=p,c[p]=1;else return d;a=l;e=a.length}if(!e)return d;b||(b=100);g=this.i&&1=d)))break;if(m){if(h)return O(l,d,0);b[b.length]=l;return}}return!c&&l}function O(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function P(a,b,c,d){c?(d=d&&b>c,a=(a=a[d?b:c])&&a[d?c:b]):a=a[b];return a}I.prototype.contain=function(a){return!!this.register[a]};I.prototype.update=function(a,b){return this.remove(a).add(a,b)};I.prototype.remove=function(a,b){const c=this.register[a];if(c){if(this.m)for(let d=0,e;dthis.s.get(r)),l=1);this.m&&1this.m.get(r)), +l=1);f&&l&&(f.lengththis.L&&(this.A.clear(),this.h=this.h/1.1|0));f&&b.push(f)}this.D&&(b=this.D(b)||b);this.cache&&a.length<=this.g&& +(this.u.set(a,b),this.u.size>this.L&&(this.u.clear(),this.g=this.g/1.1|0));return b};function E(a){a.C=null;a.u.clear();a.A.clear()};function F(a){this.limit=a&&!0!==a?a:1E3;this.cache=new Map;this.g=""}F.prototype.set=function(a,c){this.cache.has(a)||(this.cache.set(this.g=a,c),this.limit&&this.cache.size>this.limit&&this.cache.delete(this.cache.keys().next().value))};F.prototype.get=function(a){const c=this.cache.get(a);c&&this.limit&&this.g!==a&&(this.cache.delete(a),this.cache.set(this.g=a,c));return c};F.prototype.remove=function(a){for(const c of this.cache){const b=c[0];c[1].includes(a)&&this.cache.delete(b)}}; +F.prototype.clear=function(){this.cache.clear();this.g=""};function G(a=8){if(!(this instanceof G))return new G(a);this.index=v();this.i=[];this.size=0;32g;h--){f=n.substring(g,h);var l=this.score?this.score(c,n,m,f,g):O(q,d,m,e,g);P(this,k,f,l,a,b)}break}case "reverse":if(1< +e){for(h=e-1;0f?0:1),d,m,h-1,l-1),Q=this.bidirectional&&n>g;P(this,r,Q?g:n,x,a,b,Q?n:g)}}}}this.fastupdate||this.j.add(a)}}return this}; +function P(a,c,b,d,e,g,f){let h=f?a.B:a.map,l;c[b]&&f&&(l=c[b])[f]||(f?(c=l||(c[b]=v()),c[f]=1,(l=h.get(f))?h=l:h.set(f,h=new Map)):c[b]=1,(l=h.get(b))?h=l:h.set(b,h=[]),h=h[d]||(h[d]=[]),g&&h.includes(e)||(h.push(e),a.fastupdate&&((c=a.j.get(e))?c.push(h):a.j.set(e,[h]))))}function O(a,c,b,d,e){return b&&1c?c?a.slice(b,b+c):a.slice(b):a;let d=[];for(let e=0,g,f;e=f){b-=f;continue}bc&&(g=g.slice(0,c),f=g.length),d.push(g);else{if(f>=c)return f>c&&(g=g.slice(0,c)),g;d=[g]}c-=f;if(!c)break}return d.length?d=1=e)))break;if(h.length){if(f)return R(h,e,0);c.push(h);return}}return!b&&h}function T(a,c,b){let d;b&&(d=a.bidirectional&&c>b);a=b?(a=a.B.get(d?c:b))&&a.get(d?b:c):a.map.get(c);return a};N.prototype.remove=function(a,c){const b=this.j.size&&(this.fastupdate?this.j.get(a):this.j.has(a));if(b){if(this.fastupdate)for(let d=0,e;de.length)e.pop();else{const g=e.indexOf(a);g===b.length-1?e.pop():e.splice(g,1)}}else V(this.map,a),this.depth&&V(this.B,a);c||this.j.delete(a)}this.cache&&this.cache.remove(a);return this}; +function V(a,c){let b=0;if(a.constructor===Array)for(let d=0,e,g;db.add(a,c)):this.add(a,c)}return this.remove(a).add(a,c)};function W(a){let c=0;if(a.constructor===Array)for(let b=0,d;b this.s.get(r)), l = 1); + this.m && 1 < f.length && (this.I || (this.I = new RegExp("(" + this.i + ")", "g")), f = f.replace(this.I, r => this.m.get(r)), l = 1); + f && l && (f.length < this.G || this.filter && this.filter.has(f)) && (f = ""); + if (f && (this.v || this.l && 1 < f.length)) { + e = ""; + for (let r = 0, k = "", p, q; r < f.length; r++) { + p = f.charAt(r), p === k && this.l || ((q = this.v && this.v.get(p)) || "" === q ? q === k && this.l || !(k = q) || (e += q) : e += k = p); + } + f = e; + } + if (f && this.o) { + for (e = 0; f && e < this.o.length; e += 2) { + f = f.replace(this.o[e], this.o[e + 1]); + } + } + this.cache && h.length <= this.h && (this.A.set(h, f), this.A.size > this.L && (this.A.clear(), this.h = this.h / 1.1 | 0)); + f && b.push(f); + } + this.D && (b = this.D(b) || b); + this.cache && a.length <= this.g && (this.u.set(a, b), this.u.size > this.L && (this.u.clear(), this.g = this.g / 1.1 | 0)); + return b; +}; +function E(a) { + a.C = null; + a.u.clear(); + a.A.clear(); +} +;function F(a) { + this.limit = a && !0 !== a ? a : 1000; + this.cache = new Map(); + this.g = ""; +} +F.prototype.set = function(a, c) { + this.cache.has(a) || (this.cache.set(this.g = a, c), this.limit && this.cache.size > this.limit && this.cache.delete(this.cache.keys().next().value)); +}; +F.prototype.get = function(a) { + const c = this.cache.get(a); + c && this.limit && this.g !== a && (this.cache.delete(a), this.cache.set(this.g = a, c)); + return c; +}; +F.prototype.remove = function(a) { + for (const c of this.cache) { + const b = c[0]; + c[1].includes(a) && this.cache.delete(b); + } +}; +F.prototype.clear = function() { + this.cache.clear(); + this.g = ""; +}; +function G(a = 8) { + if (!(this instanceof G)) { + return new G(a); + } + this.index = v(); + this.i = []; + this.size = 0; + 32 < a ? (this.g = H, this.h = BigInt(a)) : (this.g = I, this.h = a); +} +G.prototype.get = function(a) { + const c = this.index[this.g(a)]; + return c && c.get(a); +}; +G.prototype.set = function(a, c) { + var b = this.g(a); + let d = this.index[b]; + d ? (b = d.size, d.set(a, c), (b -= d.size) && this.size++) : (this.index[b] = d = new Map([[a, c]]), this.i.push(d)); +}; +function J(a = 8) { + if (!(this instanceof J)) { + return new J(a); + } + this.index = v(); + this.g = []; + 32 < a ? (this.i = H, this.h = BigInt(a)) : (this.i = I, this.h = a); +} +J.prototype.add = function(a) { + var c = this.i(a); + let b = this.index[c]; + b ? (c = b.size, b.add(a), (c -= b.size) && this.size++) : (this.index[c] = b = new Set([a]), this.g.push(b)); +}; +t = G.prototype; +t.has = J.prototype.has = function(a) { + const c = this.index[this.i(a)]; + return c && c.has(a); +}; +t.delete = J.prototype.delete = function(a) { + const c = this.index[this.i(a)]; + c && c.delete(a) && this.size--; +}; +t.clear = J.prototype.clear = function() { + this.index = v(); + this.g = []; + this.size = 0; +}; +t.values = J.prototype.values = function*() { + for (let a = 0; a < this.g.length; a++) { + for (let c of this.g[a].values()) { + yield c; + } + } +}; +t.keys = J.prototype.keys = function*() { + for (let a = 0; a < this.g.length; a++) { + for (let c of this.g[a].keys()) { + yield c; + } + } +}; +t.entries = J.prototype.entries = function*() { + for (let a = 0; a < this.g.length; a++) { + for (let c of this.g[a].entries()) { + yield c; + } + } +}; +function I(a) { + let c = 2 ** this.h - 1; + if ("number" == typeof a) { + return a & c; + } + let b = 0, d = this.h + 1; + for (let e = 0; e < a.length; e++) { + b = (b * d ^ a.charCodeAt(e)) & c; + } + return 32 === this.h ? b + 2 ** 31 : b; +} +function H(a) { + let c = BigInt(2) ** this.h - BigInt(1); + var b = typeof a; + if ("bigint" === b) { + return a & c; + } + if ("number" === b) { + return BigInt(a) & c; + } + b = BigInt(0); + let d = this.h + BigInt(1); + for (let e = 0; e < a.length; e++) { + b = (b * d ^ BigInt(a.charCodeAt(e))) & c; } return b; } -function y() { - return Object.create(null); -} -function z(a, b) { - return b.length - a.length; -} -;const A = /[\p{Z}\p{S}\p{P}\p{C}]+/u; -function B(a, b) { - const c = Object.keys(a), d = c.length, e = []; - let h = "", f = 0; - for (let g = 0, l, n; g < d; g++) { - l = c[g], (n = a[l]) ? (e[f++] = new RegExp(b ? "(?!\\b)" + l + "(\\b|_)" : l, "g"), e[f++] = n) : h += (h ? "|" : "") + l; - } - h && (e[f++] = new RegExp(b ? "(?!\\b)(" + h + ")(\\b|_)" : "(" + h + ")", "g"), e[f] = ""); - return e; -} -function C(a, b) { - for (let c = 0, d = b.length; c < d && (a = a.replace(b[c], b[c + 1]), a); c += 2) { - } - return a; -} -;function E(a) { - if (a = ("" + a).toLowerCase()) { - if (this.o && (a = C(a, this.o)), this.A && 1 < a.length && (a = C(a, this.A)), A || "" === A) { - const b = a.split(A); - if (this.filter) { - a = this.filter; - const c = b.length, d = []; - for (let e = 0, h = 0; e < c; e++) { - const f = b[e]; - f && !a[f] && (d[h++] = f); - } - a = d; - } else { - a = b; - } +;const K = v(), L = v(); +var M = {normalize:function(a) { + return a.toLowerCase(); +}, l:!1}; +const N = {memory:{resolution:1}, performance:{resolution:6, fastupdate:!0, context:{depth:1, resolution:3}}, match:{tokenize:"forward"}, score:{resolution:9, context:{depth:2, resolution:9}}}; +v(); +P.prototype.add = function(a, c, b, d) { + if (c && (a || 0 === a)) { + if (!d && !b && this.j.has(a)) { + return this.update(a, c); } - } - return a; -} -;const F = {}, G = {}; -function H(a, b, c, d) { - const e = a.length; - let h = [], f, g, l = 0; - d && (d = []); - for (let n = e - 1; 0 <= n; n--) { - const m = a[n], r = m.length, p = y(); - let q = !f; - for (let k = 0; k < r; k++) { - const u = m[k], J = u.length; - if (J) { - for (let D = 0, x, w; D < J; D++) { - if (w = u[D], f) { - if (f[w]) { - if (!n) { - if (c) { - c--; - } else { - if (h[l++] = w, l === b) { - return h; - } - } - } - if (n || d) { - p[w] = 1; - } - q = !0; - } - if (d && (x = (g[w] || 0) + 1, g[w] = x, x < e)) { - const K = d[x - 2] || (d[x - 2] = []); - K[K.length] = w; - } - } else { - p[w] = 1; - } - } - } - } - if (d) { - f || (g = p); - } else if (!q) { - return []; - } - f = p; - } - if (d) { - for (let n = d.length - 1, m, r; 0 <= n; n--) { - m = d[n]; - r = m.length; - for (let p = 0, q; p < r; p++) { - if (q = m[p], !f[q]) { - if (c) { - c--; - } else { - if (h[l++] = q, l === b) { - return h; - } - } - f[q] = 1; - } - } - } - } - return h; -} -;function I(a, b) { - if (!(this instanceof I)) { - return new I(a); - } - let c; - if (a) { - var d = a.charset; - c = a.lang; - "string" === typeof d && (-1 === d.indexOf(":") && (d += ":default"), d = G[d]); - "string" === typeof c && (c = F[c]); - } else { - a = {}; - } - let e, h, f = a.context || {}; - this.encode = a.encode || d && d.encode || E; - this.register = b || y(); - this.s = e = a.resolution || 9; - this.B = b = d && d.B || a.tokenize || "strict"; - this.i = "strict" === b && f.depth; - this.j = t(f.bidirectional); - this.g = h = t(a.optimize); - this.m = t(a.fastupdate); - this.h = a.minlength || 1; - this.C = a.boost; - this.map = h ? v(e) : y(); - this.v = e = f.resolution || 1; - this.l = h ? v(e) : y(); - this.u = d && d.u || a.rtl; - this.o = (b = a.matcher || c && c.o) && B(b, !1); - this.A = (b = a.stemmer || c && c.A) && B(b, !0); - if (a = b = a.filter || c && c.filter) { - a = b; - d = y(); - for (let g = 0, l = a.length; g < l; g++) { - d[a[g]] = 1; - } - a = d; - } - this.filter = a; -} -I.prototype.append = function(a, b) { - return this.add(a, b, !0); -}; -I.prototype.add = function(a, b, c, d) { - if (b && (a || 0 === a)) { - if (!d && !c && this.register[a]) { - return this.update(a, b); - } - b = this.encode(b); - if (d = b.length) { - const n = y(), m = y(), r = this.i, p = this.s; - for (let q = 0; q < d; q++) { - let k = b[this.u ? d - 1 - q : q]; - var e = k.length; - if (k && e >= this.h && (r || !m[k])) { - var h = L(p, d, q), f = ""; - switch(this.B) { + c = this.encoder.encode(c); + if (d = c.length) { + const r = v(), k = v(), p = this.depth, q = this.resolution; + for (let m = 0; m < d; m++) { + let n = c[this.rtl ? d - 1 - m : m]; + var e = n.length; + if (e && (p || !k[n])) { + var g = this.score ? this.score(c, n, m, null, 0) : Q(q, d, m), f = ""; + switch(this.tokenize) { case "full": if (2 < e) { - for (h = 0; h < e; h++) { - for (var g = e; g > h; g--) { - if (g - h >= this.h) { - var l = L(p, d, q, e, h); - f = k.substring(h, g); - M(this, m, f, l, a, c); - } + for (g = 0; g < e; g++) { + for (var h = e; h > g; h--) { + f = n.substring(g, h); + var l = this.score ? this.score(c, n, m, f, g) : Q(q, d, m, e, g); + R(this, k, f, l, a, b); } } break; } case "reverse": if (1 < e) { - for (g = e - 1; 0 < g; g--) { - f = k[g] + f, f.length >= this.h && M(this, m, f, L(p, d, q, e, g), a, c); + for (h = e - 1; 0 < h; h--) { + f = n[h] + f, l = this.score ? this.score(c, n, m, f, h) : Q(q, d, m, e, h), R(this, k, f, l, a, b); } f = ""; } case "forward": if (1 < e) { - for (g = 0; g < e; g++) { - f += k[g], f.length >= this.h && M(this, m, f, h, a, c); + for (h = 0; h < e; h++) { + f += n[h], R(this, k, f, g, a, b); } break; } default: - if (this.C && (h = Math.min(h / this.C(b, k, q) | 0, p - 1)), M(this, m, k, h, a, c), r && 1 < d && q < d - 1) { - for (e = y(), f = this.v, h = k, g = Math.min(r + 1, d - q), e[h] = 1, l = 1; l < g; l++) { - if ((k = b[this.u ? d - 1 - q - l : q + l]) && k.length >= this.h && !e[k]) { - e[k] = 1; - const u = this.j && k > h; - M(this, n, u ? h : k, L(f + (d / 2 > f ? 0 : 1), d, q, g - 1, l - 1), a, c, u ? k : h); + if (R(this, k, n, g, a, b), p && 1 < d && m < d - 1) { + for (e = v(), f = this.M, g = n, h = Math.min(p + 1, d - m), e[g] = 1, l = 1; l < h; l++) { + if ((n = c[this.rtl ? d - 1 - m - l : m + l]) && !e[n]) { + e[n] = 1; + const x = this.score ? this.score(c, g, m, n, l) : Q(f + (d / 2 > f ? 0 : 1), d, m, h - 1, l - 1), O = this.bidirectional && n > g; + R(this, r, O ? g : n, x, a, b, O ? n : g); } } } } } } - this.m || (this.register[a] = 1); + this.fastupdate || this.j.add(a); } } return this; }; -function L(a, b, c, d, e) { - return c && 1 < a ? b + (d || 0) <= a ? c + (e || 0) : (a - 1) / (b + (d || 0)) * (c + (e || 0)) + 1 | 0 : 0; +function R(a, c, b, d, e, g, f) { + let h = f ? a.B : a.map, l; + c[b] && f && (l = c[b])[f] || (f ? (c = l || (c[b] = v()), c[f] = 1, (l = h.get(f)) ? h = l : h.set(f, h = new Map())) : c[b] = 1, (l = h.get(b)) ? h = l : h.set(b, h = []), h = h[d] || (h[d] = []), g && h.includes(e) || (h.push(e), a.fastupdate && ((c = a.j.get(e)) ? c.push(h) : a.j.set(e, [h])))); } -function M(a, b, c, d, e, h, f) { - let g = f ? a.l : a.map; - if (!b[c] || f && !b[c][f]) { - a.g && (g = g[d]), f ? (b = b[c] || (b[c] = y()), b[f] = 1, g = g[f] || (g[f] = y())) : b[c] = 1, g = g[c] || (g[c] = []), a.g || (g = g[d] || (g[d] = [])), h && g.includes(e) || (g[g.length] = e, a.m && (a = a.register[e] || (a.register[e] = []), a[a.length] = g)); - } +function Q(a, c, b, d, e) { + return b && 1 < a ? c + (d || 0) <= a ? b + (e || 0) : (a - 1) / (c + (d || 0)) * (b + (e || 0)) + 1 | 0 : 0; } -I.prototype.search = function(a, b, c) { - c || (b || "object" !== typeof a ? "object" === typeof b && (c = b) : (c = a, a = c.query)); - let d = [], e; - let h, f = 0; - if (c) { - a = c.query || a; - b = c.limit; - f = c.offset || 0; - var g = c.context; - h = !1; +;function S(a, c, b) { + if (1 === a.length) { + return a = a[0], a = b || a.length > c ? c ? a.slice(b, b + c) : a.slice(b) : a; } - if (a && (a = this.encode("" + a), e = a.length, 1 < e)) { - c = y(); - var l = []; - for (let m = 0, r = 0, p; m < e; m++) { - if ((p = a[m]) && p.length >= this.h && !c[p]) { - if (this.g || h || this.map[p]) { - l[r++] = p, c[p] = 1; - } else { - return d; + let d = []; + for (let e = 0, g, f; e < a.length; e++) { + if ((g = a[e]) && (f = g.length)) { + if (b) { + if (b >= f) { + b -= f; + continue; + } + b < f && (g = c ? g.slice(b, b + c) : g.slice(b), f = g.length, b = 0); + } + if (d.length) { + f > c && (g = g.slice(0, c), f = g.length), d.push(g); + } else { + if (f >= c) { + return f > c && (g = g.slice(0, c)), g; + } + d = [g]; + } + c -= f; + if (!c) { + break; + } + } + } + return d.length ? d = 1 < d.length ? [].concat.apply([], d) : d[0] : d; +} +;function T(a, c, b, d) { + var e = a.length; + let g = [], f = 0, h, l, r; + d && (d = []); + for (let k = e - 1, p; 0 <= k; k--) { + r = a[k]; + e = v(); + p = !h; + for (let q = 0, m; q < r.length; q++) { + if ((m = r[q]) && m.length) { + for (let n = 0, x; n < m.length; n++) { + if (x = m[n], h) { + if (h[x]) { + if (!k) { + if (b) { + b--; + } else { + if (g[f++] = x, f === c) { + return g; + } + } + } + if (k || d) { + e[x] = 1; + } + p = !0; + } + d && !l[x] && (l[x] = 1, (d[q] || (d[q] = [])).push(x)); + } else { + e[x] = 1; + } } } } - a = l; - e = a.length; + if (d) { + h || (l = e); + } else if (!p) { + return []; + } + h = e; } - if (!e) { + if (d) { + for (let k = d.length - 1, p, q; 0 <= k; k--) { + p = d[k]; + q = p.length; + for (let m = 0, n; m < q; m++) { + if (n = p[m], !h[n]) { + if (b) { + b--; + } else { + if (g[f++] = n, f === c) { + return g; + } + } + h[n] = 1; + } + } + } + } + return g; +} +;P.prototype.search = function(a, c, b) { + b || (c || "object" !== typeof a ? "object" === typeof c && (b = c, c = 0) : (b = a, a = "")); + let d = []; + let e, g = 0; + if (b) { + a = b.query || a; + c = b.limit || c; + g = b.offset || 0; + var f = b.context; + e = !1; + } + a = this.encoder.encode(a); + b = a.length; + c || (c = 100); + if (1 === b) { + return U.call(this, a[0], "", c, g); + } + f = this.depth && !1 !== f; + if (2 === b && f && !e) { + return U.call(this, a[0], a[1], c, g); + } + let h = 0, l = 0; + if (1 < b) { + var r = v(); + const p = []; + for (let q = 0, m; q < b; q++) { + if ((m = a[q]) && !r[m]) { + if (e || V(this, m)) { + p.push(m), r[m] = 1; + } else { + return d; + } + const n = m.length; + h = Math.max(h, n); + l = l ? Math.min(l, n) : n; + } + } + a = p; + b = a.length; + } + if (!b) { return d; } - b || (b = 100); - g = this.i && 1 < e && !1 !== g; - c = 0; - let n; - g ? (n = a[0], c = 1) : 1 < e && a.sort(z); - for (let m, r; c < e; c++) { - r = a[c]; - g ? (m = N(this, d, h, b, f, 2 === e, r, n), h && !1 === m && d.length || (n = r)) : m = N(this, d, h, b, f, 1 === e, r); - if (m) { - return m; + r = 0; + let k; + if (1 === b) { + return U.call(this, a[0], "", c, g); + } + if (2 === b && f && !e) { + return U.call(this, a[0], a[1], c, g); + } + 1 < b && (f ? (k = a[0], r = 1) : 9 < h && 3 < h / l && a.sort(w)); + for (let p, q; r < b; r++) { + q = a[r]; + k ? (p = V(this, q, k), p = W(p, d, e, this.M, c, g, 2 === b), e && !1 === p && d.length || (k = q)) : (p = V(this, q), p = W(p, d, e, this.resolution, c, g, 1 === b)); + if (p) { + return p; } - if (h && c === e - 1) { - l = d.length; - if (!l) { - if (g) { - g = 0; - c = -1; + if (e && r === b - 1) { + f = d.length; + if (!f) { + if (k) { + k = ""; + r = -1; continue; } return d; } - if (1 === l) { - return O(d[0], b, f); + if (1 === f) { + return S(d[0], c, g); } } } - return H(d, b, f, h); + return T(d, c, g, e); }; -function N(a, b, c, d, e, h, f, g) { - let l = [], n = g ? a.l : a.map; - a.g || (n = P(n, f, g, a.j)); - if (n) { - let m = 0; - const r = Math.min(n.length, g ? a.v : a.s); - for (let p = 0, q = 0, k, u; p < r; p++) { - if (k = n[p]) { - if (a.g && (k = P(k, f, g, a.j)), e && k && h && (u = k.length, u <= e ? (e -= u, k = null) : (k = k.slice(e), e = 0)), k && (l[m++] = k, h && (q += k.length, q >= d))) { +function U(a, c, b, d) { + return (a = V(this, a, c)) && a.length ? S(a, b, d) : []; +} +function W(a, c, b, d, e, g, f) { + let h = []; + if (a) { + d = Math.min(a.length, d); + for (let l = 0, r = 0, k; l < d; l++) { + if (k = a[l]) { + if (g && k && f && (k.length <= g ? (g -= k.length, k = null) : (k = k.slice(g), g = 0)), k && (h[l] = k, f && (r += k.length, r >= e))) { break; } } } - if (m) { - if (h) { - return O(l, d, 0); + if (h.length) { + if (f) { + return S(h, e, 0); } - b[b.length] = l; + c.push(h); return; } } - return !c && l; + return !b && h; } -function O(a, b, c) { - a = 1 === a.length ? a[0] : [].concat.apply([], a); - return c || a.length > b ? a.slice(c, c + b) : a; -} -function P(a, b, c, d) { - c ? (d = d && b > c, a = (a = a[d ? b : c]) && a[d ? c : b]) : a = a[b]; +function V(a, c, b) { + let d; + b && (d = a.bidirectional && c > b); + a = b ? (a = a.B.get(d ? c : b)) && a.get(d ? b : c) : a.map.get(c); return a; } -I.prototype.contain = function(a) { - return !!this.register[a]; -}; -I.prototype.update = function(a, b) { - return this.remove(a).add(a, b); -}; -I.prototype.remove = function(a, b) { - const c = this.register[a]; - if (c) { - if (this.m) { - for (let d = 0, e; d < c.length; d++) { - e = c[d], e.splice(e.indexOf(a), 1); +;P.prototype.remove = function(a, c) { + const b = this.j.size && (this.fastupdate ? this.j.get(a) : this.j.has(a)); + if (b) { + if (this.fastupdate) { + for (let d = 0, e; d < b.length; d++) { + if (e = b[d]) { + if (2 > e.length) { + e.pop(); + } else { + const g = e.indexOf(a); + g === b.length - 1 ? e.pop() : e.splice(g, 1); + } + } } } else { - Q(this.map, a, this.s, this.g), this.i && Q(this.l, a, this.v, this.g); + X(this.map, a), this.depth && X(this.B, a); } - b || delete this.register[a]; + c || this.j.delete(a); } + this.cache && this.cache.remove(a); return this; }; -function Q(a, b, c, d, e) { - let h = 0; +function X(a, c) { + let b = 0; if (a.constructor === Array) { - if (e) { - b = a.indexOf(b), -1 !== b ? 1 < a.length && (a.splice(b, 1), h++) : h++; - } else { - e = Math.min(a.length, c); - for (let f = 0, g; f < e; f++) { - if (g = a[f]) { - h = Q(g, b, c, d, e), d || h || delete a[f]; + for (let d = 0, e, g; d < a.length; d++) { + if ((e = a[d]) && e.length) { + if (g = e.indexOf(c), 0 <= g) { + 1 < e.length ? (e.splice(g, 1), b++) : delete a[d]; + break; + } else { + b++; } } } } else { - for (let f in a) { - (h = Q(a[f], b, c, d, e)) || delete a[f]; + for (let d of a) { + const e = d[0], g = X(d[1], c); + g ? b += g : a.delete(e); } } - return h; + return b; } -;export default {Index:I, Document:null, Worker:null, registerCharset:function(a, b) { - G[a] = b; -}, registerLanguage:function(a, b) { - F[a] = b; -}}; +;function P(a, c) { + if (!(this instanceof P)) { + return new P(a); + } + if (a) { + var b = "string" === typeof a ? a : a.preset; + b && (N[b] || console.warn("Preset not found: " + b), a = Object.assign({}, N[b], a)); + } else { + a = {}; + } + b = a.context || {}; + const d = a.encode || a.encoder || M; + this.encoder = d.encode ? d : "object" === typeof d ? new D(d) : {encode:d}; + let e; + this.resolution = a.resolution || 9; + this.tokenize = e = a.tokenize || "strict"; + this.depth = "strict" === e && b.depth || 0; + this.bidirectional = !1 !== b.bidirectional; + this.fastupdate = !!a.fastupdate; + this.score = a.score || null; + this.map = (e = !1, new Map()); + this.B = e ? new G(e) : new Map(); + this.j = c || (this.fastupdate ? e ? new G(e) : new Map() : e ? new J(e) : new Set()); + this.M = b.resolution || 1; + this.rtl = d.rtl || a.rtl || !1; + this.cache = (e = a.cache || null) && new F(e); +} +t = P.prototype; +t.clear = function() { + this.map.clear(); + this.B.clear(); + this.j.clear(); + this.cache && this.cache.clear(); + return this; +}; +t.append = function(a, c) { + return this.add(a, c, !0); +}; +t.contain = function(a) { + return this.j.has(a); +}; +t.update = function(a, c) { + if (this.async) { + const b = this, d = this.remove(a); + return d.then ? d.then(() => b.add(a, c)) : this.add(a, c); + } + return this.remove(a).add(a, c); +}; +function Y(a) { + let c = 0; + if (a.constructor === Array) { + for (let b = 0, d; b < a.length; b++) { + (d = a[b]) && (c += d.length); + } + } else { + for (const b of a) { + const d = b[0], e = Y(b[1]); + e ? c += e : a.delete(d); + } + } + return c; +} +t.cleanup = function() { + if (!this.fastupdate) { + return console.info('Cleanup the index isn\'t required when not using "fastupdate".'), this; + } + Y(this.map); + this.depth && Y(this.B); + return this; +}; +t.searchCache = function(a, c, b) { + a = ("object" === typeof a ? "" + a.query : a).toLowerCase(); + let d = this.cache.get(a); + if (!d) { + d = this.search(a, c, b); + if (d instanceof Promise) { + const e = this; + d.then(function(g) { + e.cache.set(a, g); + }); + } + this.cache.set(a, d); + } + return d; +}; +export default {Index:P, Encoder:D, Charset:L, Language:K, Document:null, Worker:null, Resolver:null, IndexedDB:null}; +export const Index=P;export const Encoder=D;export const Charset=L;export const Language=K;export const Document=null;export const Worker=null;export const Resolver=null;export const IndexedDB=null; \ No newline at end of file diff --git a/dist/flexsearch.light.module.min.js b/dist/flexsearch.light.module.min.js index fe7705e..df51872 100644 --- a/dist/flexsearch.light.module.min.js +++ b/dist/flexsearch.light.module.min.js @@ -1,18 +1,29 @@ /**! - * FlexSearch.js v0.7.41 (Light.module) + * FlexSearch.js v0.8.0 (Bundle) * Author and Copyright: Thomas Wilkerling * Licence: Apache-2.0 * Hosted by Nextapps GmbH * https://github.com/nextapps-de/flexsearch */ -function t(a){return"undefined"!==typeof a?a:!0}function v(a){const b=Array(a);for(let c=0;c=this.h&&(r||!m[k])){var h=L(p,d,q),f="";switch(this.B){case "full":if(2h;g--)if(g-h>=this.h){var l=L(p,d,q,e,h);f=k.substring(h,g);M(this,m,f,l,a,c)}break}case "reverse":if(1=this.h&&M(this, -m,f,L(p,d,q,e,g),a,c);f=""}case "forward":if(1=this.h&&M(this,m,f,h,a,c);break}default:if(this.C&&(h=Math.min(h/this.C(b,k,q)|0,p-1)),M(this,m,k,h,a,c),r&&1=this.h&&!e[k]){e[k]=1;const u=this.j&&k>h;M(this,n,u?h:k,L(f+(d/2>f?0:1),d,q,g-1,l-1),a,c,u?k:h)}}}}this.m||(this.register[a]=1)}}return this}; -function L(a,b,c,d,e){return c&&1=this.h&&!c[p])if(this.g||h||this.map[p])l[r++]=p,c[p]=1;else return d;a=l;e=a.length}if(!e)return d;b||(b=100);g=this.i&&1=d)))break;if(m){if(h)return O(l,d,0);b[b.length]=l;return}}return!c&&l}function O(a,b,c){a=1===a.length?a[0]:[].concat.apply([],a);return c||a.length>b?a.slice(c,c+b):a} -function P(a,b,c,d){c?(d=d&&b>c,a=(a=a[d?b:c])&&a[d?c:b]):a=a[b];return a}I.prototype.contain=function(a){return!!this.register[a]};I.prototype.update=function(a,b){return this.remove(a).add(a,b)};I.prototype.remove=function(a,b){const c=this.register[a];if(c){if(this.m)for(let d=0,e;dthis.s.get(r)),l=1);this.m&&1this.m.get(r)), +l=1);f&&l&&(f.lengththis.L&&(this.A.clear(),this.h=this.h/1.1|0));f&&b.push(f)}this.D&&(b=this.D(b)||b);this.cache&&a.length<=this.g&& +(this.u.set(a,b),this.u.size>this.L&&(this.u.clear(),this.g=this.g/1.1|0));return b};function E(a){a.C=null;a.u.clear();a.A.clear()};function F(a){this.limit=a&&!0!==a?a:1E3;this.cache=new Map;this.g=""}F.prototype.set=function(a,c){this.cache.has(a)||(this.cache.set(this.g=a,c),this.limit&&this.cache.size>this.limit&&this.cache.delete(this.cache.keys().next().value))};F.prototype.get=function(a){const c=this.cache.get(a);c&&this.limit&&this.g!==a&&(this.cache.delete(a),this.cache.set(this.g=a,c));return c};F.prototype.remove=function(a){for(const c of this.cache){const b=c[0];c[1].includes(a)&&this.cache.delete(b)}}; +F.prototype.clear=function(){this.cache.clear();this.g=""};function G(a=8){if(!(this instanceof G))return new G(a);this.index=v();this.i=[];this.size=0;32g;h--){f=n.substring(g,h);var l=this.score?this.score(c,n,m,f,g):Q(q,d,m,e,g);R(this,k,f,l,a,b)}break}case "reverse":if(1< +e){for(h=e-1;0f?0:1),d,m,h-1,l-1),N=this.bidirectional&&n>g;R(this,r,N?g:n,x,a,b,N?n:g)}}}}this.fastupdate||this.j.add(a)}}return this}; +function R(a,c,b,d,e,g,f){let h=f?a.B:a.map,l;c[b]&&f&&(l=c[b])[f]||(f?(c=l||(c[b]=v()),c[f]=1,(l=h.get(f))?h=l:h.set(f,h=new Map)):c[b]=1,(l=h.get(b))?h=l:h.set(b,h=[]),h=h[d]||(h[d]=[]),g&&h.includes(e)||(h.push(e),a.fastupdate&&((c=a.j.get(e))?c.push(h):a.j.set(e,[h]))))}function Q(a,c,b,d,e){return b&&1c?c?a.slice(b,b+c):a.slice(b):a;let d=[];for(let e=0,g,f;e=f){b-=f;continue}bc&&(g=g.slice(0,c),f=g.length),d.push(g);else{if(f>=c)return f>c&&(g=g.slice(0,c)),g;d=[g]}c-=f;if(!c)break}return d.length?d=1=e)))break;if(h.length){if(f)return S(h,e,0);c.push(h);return}}return!b&&h}function V(a,c,b){let d;b&&(d=a.bidirectional&&c>b);a=b?(a=a.B.get(d?c:b))&&a.get(d?b:c):a.map.get(c);return a};P.prototype.remove=function(a,c){const b=this.j.size&&(this.fastupdate?this.j.get(a):this.j.has(a));if(b){if(this.fastupdate)for(let d=0,e;de.length)e.pop();else{const g=e.indexOf(a);g===b.length-1?e.pop():e.splice(g,1)}}else X(this.map,a),this.depth&&X(this.B,a);c||this.j.delete(a)}this.cache&&this.cache.remove(a);return this}; +function X(a,c){let b=0;if(a.constructor===Array)for(let d=0,e,g;db.add(a,c)):this.add(a,c)}return this.remove(a).add(a,c)};function Y(a){let c=0;if(a.constructor===Array)for(let b=0,d;b setTimeout(tick, 0, resolve)) + // ); + // + // // apply different performance budgets + // if(key === "update" || key === "remove" && this.fastupdate === false){ + // budget += 1000 * this.resolution; + // if(this.depth) + // budget += 1000 * this.resolution_ctx; + // } + // else if(key === "search"){ + // budget++; + // } + // else{ + // budget += 20 * this.resolution; + // if(this.depth) + // budget += 20 * this.resolution_ctx; + // } + // + // // wait for the event loop cycle + // if(budget >= 1e6){ + // await cycle; + // } + + const args = /*[].slice.call*/arguments, arg = args[args.length - 1]; let callback; - if (is_function(arg)) { - + if ("function" == typeof arg) { callback = arg; delete args[args.length - 1]; } - const promise = new Promise(function (resolve) { - - setTimeout(function () { - - self.async = !0; - const res = self[key].apply(self, args); - self.async = !1; - resolve(res); - }); - }); - - if (callback) { - - promise.then(callback); - return this; - } else { - - return promise; - } + this.async = !0; + const res = this[key].apply(this, args); + this.async = !1; + res.then ? res.then(callback) : callback(res); + return res; }; } \ No newline at end of file diff --git a/dist/module-debug/cache.js b/dist/module-debug/cache.js index 43262f7..a21e83a 100644 --- a/dist/module-debug/cache.js +++ b/dist/module-debug/cache.js @@ -1,168 +1,77 @@ -import { IndexInterface, DocumentInterface } from "./type.js"; -import { create_object, is_object } from "./common.js"; +import Index from "./index.js"; +import Document from "./document.js"; + +/** + * @param {string|Object} query + * @param {number|Object=} limit + * @param {Object=} options + * @this {Index|Document} + * @returns {Array|Promise} + */ + +export function searchCache(query, limit, options) { + + query = ("object" == typeof query ? "" + query.query : "" + query).toLowerCase(); + + //let encoded = this.encoder.encode(query).join(" "); + let cache = this.cache.get(query); + if (!cache) { + cache = this.search(query, limit, options); + if (cache instanceof Promise) { + const self = this; + cache.then(function (cache) { + self.cache.set(query, cache); + }); + } + this.cache.set(query, cache); + } + return cache; +} /** * @param {boolean|number=} limit * @constructor */ -function CacheClass(limit) { - +export default function CacheClass(limit) { /** @private */ - this.limit = !0 !== limit && limit; - + this.limit = !limit || !0 === limit ? 1000 : limit; /** @private */ - this.cache = create_object(); - + this.cache = new Map(); /** @private */ - this.queue = []; - - //this.clear(); + this.last = ""; } -export default CacheClass; - -/** - * @param {string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @this {IndexInterface} - * @returns {Array} - */ - -export function searchCache(query, limit, options) { - - if (is_object(query)) { - - query = query.query; - } - - let cache = this.cache.get(query); - - if (!cache) { - - cache = this.search(query, limit, options); - this.cache.set(query, cache); - } - - return cache; -} - -// CacheClass.prototype.clear = function(){ -// -// /** @private */ -// this.cache = create_object(); -// -// /** @private */ -// this.queue = []; -// }; - CacheClass.prototype.set = function (key, value) { - - if (!this.cache[key]) { - - // it is just a shame that native function array.shift() performs so bad - - // const length = this.queue.length; - // - // this.queue[length] = key; - // - // if(length === this.limit){ - // - // delete this.cache[this.queue.shift()]; - // } - - // the same bad performance - - // this.queue.unshift(key); - // - // if(this.queue.length === this.limit){ - // - // this.queue.pop(); - // } - - // fast implementation variant - - // let length = this.queue.length; - // - // if(length === this.limit){ - // - // length--; - // - // delete this.cache[this.queue[0]]; - // - // for(let x = 0; x < length; x++){ - // - // this.queue[x] = this.queue[x + 1]; - // } - // } - // - // this.queue[length] = key; - - // current fastest implementation variant - // theoretically that should not perform better compared to the example above - - let length = this.queue.length; - - if (length === this.limit) { - - delete this.cache[this.queue[length - 1]]; - } else { - - length++; + if (!this.cache.has(key)) { + this.cache.set(this.last = key, value); + if (this.limit && this.cache.size > this.limit) { + this.cache.delete(this.cache.keys().next().value); } - - for (let x = length - 1; 0 < x; x--) { - - this.queue[x] = this.queue[x - 1]; - } - - this.queue[0] = key; } - - this.cache[key] = value; }; CacheClass.prototype.get = function (key) { - - const cache = this.cache[key]; - - if (this.limit && cache) { - - // probably the indexOf() method performs faster when matched content is on front (left-to-right) - // using lastIndexOf() does not help, it performs almost slower - - const pos = this.queue.indexOf(key); - - // if(pos < this.queue.length - 1){ - // - // const tmp = this.queue[pos]; - // this.queue[pos] = this.queue[pos + 1]; - // this.queue[pos + 1] = tmp; - // } - - if (pos) { - - const tmp = this.queue[pos - 1]; - this.queue[pos - 1] = this.queue[pos]; - this.queue[pos] = tmp; - } + const cache = this.cache.get(key); + if (cache && this.limit && this.last !== key) { + this.cache.delete(key); + this.cache.set(this.last = key, cache); } - return cache; }; -CacheClass.prototype.del = function (id) { +CacheClass.prototype.remove = function (id) { + for (const item of this.cache) { + const key = item[0], + value = item[1]; - for (let i = 0, item, key; i < this.queue.length; i++) { - - key = this.queue[i]; - item = this.cache[key]; - - if (item.includes(id)) { - - this.queue.splice(i--, 1); - delete this.cache[key]; + if (value.includes(id)) { + this.cache.delete(key); } } +}; + +CacheClass.prototype.clear = function () { + this.cache.clear(); + this.last = ""; }; \ No newline at end of file diff --git a/dist/module-debug/common.js b/dist/module-debug/common.js index 95e4ace..b733012 100644 --- a/dist/module-debug/common.js +++ b/dist/module-debug/common.js @@ -1,6 +1,65 @@ -export function parse_option(value, default_value) { +/** + * @param {*} value + * @param {*} default_value + * @param {*=} merge_value + * @return {*} + */ - return "undefined" != typeof value ? value : default_value; +export function parse_option(value, default_value, merge_value) { + const type_merge = typeof merge_value, + type_value = typeof value; + + + if ("undefined" != type_merge) { + if ("undefined" != type_value) { + + if (merge_value) { + if ("function" == type_value && type_merge == type_value) { + return function (str) { + return (/** @type Function */value( + /** @type Function */merge_value(str)) + ); + }; + } + + const constructor_value = value.constructor, + constructor_merge = merge_value.constructor; + + + if (constructor_value === constructor_merge) { + + if (constructor_value === Array) { + return merge_value.concat(value); + } + + if (constructor_value === Map) { + const map = new Map( /** @type !Map */merge_value); + for (const item of /** @type !Map */value) { + const key = item[0], + val = item[1]; + + map.set(key, val); + } + return map; + } + + if (constructor_value === Set) { + const set = new Set( /** @type !Set */merge_value); + for (const val of /** @type !Set */value.values()) { + set.add(val); + } + return set; + } + } + } + + return value; + } else { + return merge_value; + } + } + + return "undefined" == type_value ? default_value : value; } /** @@ -13,19 +72,33 @@ export function create_object_array(count) { const array = Array(count); for (let i = 0; i < count; i++) { - array[i] = create_object(); } return array; } +/** + * @param {!number} count + * @returns {Array} + */ + +export function create_map_array(count) { + + const array = Array(count); + + for (let i = 0; i < count; i++) { + array[i] = new Map(); + } + + return array; +} + export function create_arrays(count) { const array = Array(count); for (let i = 0; i < count; i++) { - array[i] = []; } @@ -38,41 +111,75 @@ export function create_arrays(count) { */ export function get_keys(obj) { - return Object.keys(obj); } export function create_object() { - return Object.create(null); } export function concat(arrays) { - return [].concat.apply([], arrays); } export function sort_by_length_down(a, b) { - return b.length - a.length; } -export function is_array(val) { +export function sort_by_length_up(a, b) { + return a.length - b.length; +} +export function is_array(val) { return val.constructor === Array; } export function is_string(val) { - return "string" == typeof val; } export function is_object(val) { - return "object" == typeof val; } export function is_function(val) { - return "function" == typeof val; +} + +/** + * @param {Map|Set} val + * @param {boolean=} stringify + * @return {Array} + */ + +export function toArray(val, stringify) { + const result = []; + for (const key of val.keys()) { + result.push(stringify ? "" + key : key); + } + return result; +} + +// TODO support generic function created from string when tree depth > 1 +export function parse_simple(obj, tree) { + + if (is_string(tree)) { + obj = obj[tree]; + } else for (let i = 0; obj && i < tree.length; i++) { + obj = obj[tree[i]]; + } + + return obj; +} + +export function get_max_len(arr) { + let len = 0; + for (let i = 0, res; i < arr.length; i++) { + if (res = arr[i]) { + if (len < res.length) { + len = res.length; + } + } + } + return len; } \ No newline at end of file diff --git a/dist/module-debug/compress.js b/dist/module-debug/compress.js new file mode 100644 index 0000000..8efa91a --- /dev/null +++ b/dist/module-debug/compress.js @@ -0,0 +1,65 @@ +let table, timer; // = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/"; + +const cache = new Map(); + +function toRadix(number, radix = 255) { + + if (!table) { + table = Array(radix); + // the char code 0 could be a special marker + for (let i = 0; i < radix; i++) table[i] = i + 1; + table = String.fromCharCode.apply(null, table); + } + + let rixit, + residual = number, + result = ""; + + + while (!0) { + rixit = residual % radix; + result = table.charAt(rixit) + result; + residual = 0 | residual / radix; + if (!residual) break; + } + + return result; +} + +export default function (str) { + if (timer) { + if (cache.has(str)) { + return cache.get(str); + } + } else { + timer = setTimeout(clear); + } + + /* 2 ** ((level + 1.5) * 1.6 | 0) */ + + const result = toRadix("number" == typeof str ? str : lcg(str)); + + 2e5 < cache.size && cache.clear(); + cache.set(str, result); + + + return result; +} + +function lcg(str) { + let range = 4294967295; + if ("number" == typeof str) { + return str & range; + } + let crc = 0; + for (let i = 0; i < str.length; i++) { + crc = (crc * 33 ^ str.charCodeAt(i)) & range; + } + // shift up from Int32 to UInt32 range 0xFFFFFFFF + return crc + 2147483648; +} + +function clear() { + timer = null; + cache.clear(); +} \ No newline at end of file diff --git a/dist/module-debug/db/clickhouse/index.js b/dist/module-debug/db/clickhouse/index.js new file mode 100644 index 0000000..51bfa95 --- /dev/null +++ b/dist/module-debug/db/clickhouse/index.js @@ -0,0 +1,648 @@ + +import { ClickHouse } from "clickhouse"; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { concat, toArray } from "../../common.js"; +const defaults = { + host: "http://localhost", + port: "8123", + debug: !1, + basicAuth: null, + isUseGzip: !1, + trimQuery: !1, + usePost: !1, + format: "json", + raw: !1, + config: { + output_format_json_quote_64bit_integers: 0, + enable_http_compression: 0, + database: "default" + } +}, + VERSION = 1, + fields = ["map", "ctx", "tag", "reg", "cfg"], + types = { + text: "String", + char: "String", + varchar: "String", + string: "String", + number: "Int32", + numeric: "Int32", + integer: "Int32", + smallint: "Int16", + tinyint: "Int8", + mediumint: "Int32", + int: "Int32", + int8: "Int8", + uint8: "UInt8", + int16: "Int16", + uint16: "UInt16", + int32: "Int32", + uint32: "UInt32", + int64: "Int64", + uint64: "UInt64", + bigint: "Int64" +}; + + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +let DB; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function ClickhouseDB(name, config = {}) { + if (!(this instanceof ClickhouseDB)) { + return new ClickhouseDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + //field = "Test-456"; + this.id = "flexsearch" + (name ? "_" + sanitize(name) : ""); + this.field = config.field ? "_" + sanitize(config.field) : ""; + // Clickhouse does not support ALTER TABLE to upgrade + // the type of the ID when it is a part of the merge key + this.type = config.type ? types[config.type.toLowerCase()] : "String"; + if (!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); + //this.trx = false; + this.support_tag_search = !0; + this.db = DB || (DB = config.db || null); + Object.assign(defaults, config); + config.database && (defaults.config.database = config.database); + this.db && delete defaults.db; +} + +ClickhouseDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + defaults.resolution = Math.max(flexsearch.resolution, flexsearch.resolution_ctx); + flexsearch.db = this; + return this.open(); +}; + +ClickhouseDB.prototype.open = async function () { + + if (!this.db) { + this.db = DB || (DB = new ClickHouse(defaults)); + } + + const exists = await this.db.query(` + SELECT 1 FROM system.databases WHERE name = '${this.id}'; + `).toPromise(); + + if (!exists || !exists.length) { + await this.db.query(` + CREATE DATABASE IF NOT EXISTS ${this.id}; + `).toPromise(); + } + + for (let i = 0; i < fields.length; i++) { + switch (fields[i]) { + case "map": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.map${this.field}( + key String, + res ${255 >= defaults.resolution ? "UInt8" : "UInt16"}, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (key)*/ + ORDER BY (key, id); + `, { params: { name: this.id + ".map" + this.field } }).toPromise(); + break; + + case "ctx": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.ctx${this.field}( + ctx String, + key String, + res ${255 >= defaults.resolution ? "UInt8" : "UInt16"}, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (ctx, key)*/ + ORDER BY (ctx, key, id); + `).toPromise(); + break; + + case "tag": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.tag${this.field}( + tag String, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (ctx, key)*/ + ORDER BY (tag, id); + `).toPromise(); + break; + + case "reg": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.reg( + id ${this.type}, + doc Nullable(String) + ) + ENGINE = MergeTree + ORDER BY (id); + `).toPromise(); + break; + + case "cfg": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.cfg${this.field}( + cfg String + ) + ENGINE = TinyLog; + `).toPromise(); + break; + } + } + + return this.db; +}; + +ClickhouseDB.prototype.close = function () { + this.db.close(); + this.db = null; + return this; +}; + +ClickhouseDB.prototype.destroy = async function () { + await Promise.all([this.db.query(`DROP TABLE ${this.id}.map${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.ctx${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.tag${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.cfg${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.reg;`).toPromise()]); + this.close(); +}; + +ClickhouseDB.prototype.clear = function () { + return Promise.all([this.db.query(`TRUNCATE TABLE ${this.id}.map${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.ctx${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.tag${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.cfg${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.reg;`).toPromise()]); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + for (let i = 0; i < rows.length; i++) { + if (enrich) { + if (rows[i].doc) { + rows[i].doc = JSON.parse(rows[i].doc); + } + } else { + rows[i] = rows[i].id; + } + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +ClickhouseDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = !1, tags) { + let rows, + params = ctx ? { ctx, key } : { key }, + table = this.id + (ctx ? ".ctx" : ".map") + this.field; + + if (tags) { + for (let i = 0, count = 1; i < tags.length; i += 2) { + ` AND ${table}.id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = {tag${count}:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + if (ctx) { + rows = this.db.query(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE ctx = {ctx:String} AND key = {key:String} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, { params }).toPromise(); + } else { + rows = this.db.query(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE key = {key:String} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, { params }).toPromise(); + } + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +ClickhouseDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const table = this.id + ".tag" + this.field, + promise = this.db.query(` + SELECT ${table}.id + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE tag = {tag:String} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, { params: { tag } }).toPromise(); + + enrich || promise.then(function (rows) { + return create_result(rows, !0, !1); + }); + return promise; +}; + +ClickhouseDB.prototype.enrich = async function (ids) { + let MAXIMUM_QUERY_VARS = 1e5, + result = []; + + if ("object" != typeof ids) { + ids = [ids]; + } + for (let count = 0; count < ids.length;) { + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + let params = {}, + stmt = ""; + + for (let i = 0; i < chunk.length; i++) { + stmt += (stmt ? "," : "") + "{id" + (i + 1) + ":String}"; + params["id" + (i + 1)] = chunk[i]; + } + const res = await this.db.query(` + SELECT id, doc + FROM ${this.id}.reg + WHERE id IN (${stmt})`, { params }).toPromise(); + if (res && res.length) { + for (let i = 0, doc; i < res.length; i++) { + if (doc = res[i].doc) { + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + return 1 === result.length ? result[0] : 1 < result.length ? concat(result) : result; +}; + +ClickhouseDB.prototype.has = function (id) { + return this.db.query(` + SELECT EXISTS( + SELECT 1 + FROM ${this.id}.reg + WHERE id = {id:${this.type /*=== "number" ? "Int32" : "String"*/}} + LIMIT 1 + )`, { params: { id } }).toPromise(); +}; + +ClickhouseDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !0, tags) { + let rows; + if (1 < query.length && flexsearch.depth) { + let where = "", + params = {}, + keyword = query[0], + term; + + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + where += (where ? " OR " : "") + `(ctx = {ctx${i}:String} AND key = {key${i}:String})`; + params["ctx" + i] = swap ? term : keyword; + params["key" + i] = swap ? keyword : term; + keyword = term; + } + + if (tags) { + where = "(" + where + ")"; + for (let i = 0, count = 1; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = {tag${count}:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + + rows = this.db.query(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.ctx${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + (query.length - 1)} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, { params }).toPromise(); + + // for(let i = 1; i < query.length; i++){ + // where += (where ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM ${this.id}.ctx${this.field} + // WHERE ctx = {ctx${i}:String} AND key = {key${i}:String} + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params["ctx" + i] = swap ? term : keyword; + // params["key" + i] = swap ? keyword : term; + // keyword = term; + // } + // + // rows = await this.db.query(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, { params }).toPromise(); + } else { + let where = "", + params = {}; + + + for (let i = 0; i < query.length; i++) { + where += (where ? "," : "") + `{key${i}:String}`; + params["key" + i] = query[i]; + } + where = "key " + (1 < query.length ? "IN (" + where + ")" : "= " + where); + + if (tags) { + where = "(" + where + ")"; + for (let i = 0, count = 1; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = {tag${count}:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + + rows = this.db.query(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.map${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + query.length} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, { params }).toPromise(); + + // for(let i = 0; i < query.length; i++){ + // params["key" + i] = query[i]; + // where += (where ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM ${ this.id }.map${ this.field } + // WHERE key = {key${i}:String} + // `; + // } + // rows = await this.db.query(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, { params }).toPromise(); + } + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +ClickhouseDB.prototype.info = function () { + // todo +}; + +ClickhouseDB.prototype.transaction = function (task) { + + // not supported + return task.call(this); +}; + +ClickhouseDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + if (!flexsearch.reg.size) { + return; + } + + if (flexsearch.map.size) { + let data = []; + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + //this.type || (this.type = typeof ids[0]); + for (let j = 0; j < ids.length; j++) { + data.push({ + key: key, + res: i, + id: /*this.type === "number" + ? parseInt(ids[j], 10) + :*/ids[j] + }); + } + } + } + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.map${this.field} (key, res, id)`, data).toPromise(); + } + } + + if (flexsearch.ctx.size) { + let data = []; + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + for (let j = 0; j < ids.length; j++) { + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: /*this.type === "number" + ? parseInt(ids[j], 10) + :*/ids[j] + }); + } + } + } + } + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.ctx${this.field} (ctx, key, res, id)`, data).toPromise(); + } + } + + if (flexsearch.tag) { + let data = []; + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let j = 0; j < ids.length; j++) { + data.push({ tag, id: ids[j] }); + } + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.tag${this.field} (tag, id)`, data).toPromise(); + } + } + + if (flexsearch.store) { + let data = []; + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + data.push({ id, doc: doc && JSON.stringify(doc) }); + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.reg (id, doc)`, data).toPromise(); + } + } else if (!flexsearch.bypass) { + let data = toArray(flexsearch.reg); + for (let i = 0; i < data.length; i++) { + data[i] = { id: data[i] }; + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.reg (id)`, data).toPromise(); + } + } + + // TODO + // await this.db.insert(`INSERT INTO ${this.id}.cfg${this.field} (cfg)`, [{ + // cfg: JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }) + // }]).toPromise(); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); + + await Promise.all([this.db.query(`OPTIMIZE TABLE ${this.id}.map${this.field} FINAL`).toPromise(), this.db.query(`OPTIMIZE TABLE ${this.id}.ctx${this.field} FINAL`).toPromise(), this.db.query(`OPTIMIZE TABLE ${this.id}.tag${this.field} FINAL`).toPromise(), this.db.query(`OPTIMIZE TABLE ${this.id}.reg FINAL`).toPromise()]); +}; + +ClickhouseDB.prototype.remove = async function (ids) { + + if ("object" != typeof ids) { + ids = [ids]; + } + + while (ids.length) { + + let chunk = ids.slice(0, 1e5); + ids = ids.slice(1e5); + chunk = "String" === this.type ? "'" + chunk.join("','") + "'" : chunk.join(","); + + await Promise.all([this.db.query(` + ALTER TABLE ${this.id}.map${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise(), this.db.query(` + ALTER TABLE ${this.id}.ctx${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise(), this.db.query(` + ALTER TABLE ${this.id}.tag${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise(), this.db.query(` + ALTER TABLE ${this.id}.reg + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise()]); + } +}; \ No newline at end of file diff --git a/dist/module-debug/db/indexeddb/index.js b/dist/module-debug/db/indexeddb/index.js new file mode 100644 index 0000000..e7c6fee --- /dev/null +++ b/dist/module-debug/db/indexeddb/index.js @@ -0,0 +1,589 @@ + +import Document from "../../document.js"; + +const VERSION = 1, + IndexedDB = "undefined" != typeof window && (window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB), + IDBTransaction = "undefined" != typeof window && (window.IDBTransaction || window.webkitIDBTransaction || window.msIDBTransaction), + IDBKeyRange = "undefined" != typeof window && (window.IDBKeyRange || window.webkitIDBKeyRange || window.msIDBKeyRange), + fields = ["map", "ctx", "tag", "reg", "cfg"]; + +import StorageInterface from "../interface.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +/** + * @constructor + * @implements StorageInterface + */ + +export default function IdxDB(name, config = {}) { + if (!(this instanceof IdxDB)) { + return new IdxDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = "flexsearch" + (name ? ":" + sanitize(name) : ""); + this.field = config.field ? sanitize(config.field) : ""; + this.support_tag_search = !1; + this.db = null; + this.trx = {}; +} + +IdxDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +IdxDB.prototype.open = function () { + + let self = this; + + navigator.storage && navigator.storage.persist(); + + return this.db || new Promise(function (resolve, reject) { + + const req = IndexedDB.open(self.id + (self.field ? ":" + self.field : ""), VERSION); + + req.onupgradeneeded = function () { + + const db = self.db = this.result; + + // Using Indexes + IDBKeyRange on schema map => [key, res, id] performs + // too bad and blows up amazingly in size + // The schema map:key => [res][id] is currently used instead + // In fact that bypass the idea of a storage solution, + // IndexedDB is such a poor contribution :( + + fields.forEach(ref => { + db.objectStoreNames.contains(ref) || db.createObjectStore(ref); //{ autoIncrement: true /*keyPath: "id"*/ } + //.createIndex("idx", "ids", { multiEntry: true, unique: false }); + }); + + // switch(event.oldVersion){ // existing db version + // case 0: + // // version 0 means that the client had no database + // // perform initialization + // case 1: + // // client had version 1 + // // update + // } + }; + + req.onblocked = function (event) { + // this event shouldn't trigger if we handle onversionchange correctly + // it means that there's another open connection to the same database + // and it wasn't closed after db.onversionchange triggered for it + console.error("blocked", event); + reject(); + }; + + req.onerror = function (event) { + console.error(this.error, event); + reject(); + }; + + req.onsuccess = function () { + self.db = this.result; //event.target.result; + self.db.onversionchange = function () { + //database is outdated + self.close(); + }; + resolve(self); + }; + }); +}; + +IdxDB.prototype.close = function () { + this.db.close(); + this.db = null; +}; + +IdxDB.prototype.destroy = function () { + this.db && this.close(); + return IndexedDB.deleteDatabase(this.id + (this.field ? ":" + this.field : "")); +}; + +// IdxDB.prototype.set = function(ref, key, ctx, data){ +// const transaction = this.db.transaction(ref, "readwrite"); +// const map = transaction.objectStore(ref); +// const req = map.put(data, ctx ? ctx + ":" + key : key); +// return transaction;//promisfy(req, callback); +// }; + +// IdxDB.prototype.delete = function(ref, key, ctx){ +// const transaction = this.db.transaction(ref, "readwrite"); +// const map = transaction.objectStore(ref); +// const req = map.delete(ctx ? ctx + ":" + key : key); +// return transaction;//promisfy(req, callback); +// }; + +IdxDB.prototype.clear = function () { + const transaction = this.db.transaction(fields, "readwrite"); + for (let i = 0; i < fields.length; i++) { + transaction.objectStore(fields[i]).clear(); + } + return promisfy(transaction); +}; + +IdxDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = /* tag? */!0, enrich = !1) { + const transaction = this.db.transaction(ctx ? "ctx" : "map", "readonly"), + map = transaction.objectStore(ctx ? "ctx" : "map"), + req = map.get(ctx ? ctx + ":" + key : key), + self = this; + + return promisfy(req).then(function (res) { + let result = []; + if (!res || !res.length) return result; + if (resolve) { + if (!limit && !offset && 1 === res.length) { + return res[0]; + } + for (let i = 0, arr; i < res.length; i++) { + if ((arr = res[i]) && arr.length) { + if (offset >= arr.length) { + offset -= arr.length; + continue; + } + const end = limit ? offset + Math.min(arr.length - offset, limit) : arr.length; + for (let j = offset; j < end; j++) { + result.push(arr[j]); + } + offset = 0; + if (result.length === limit) { + break; + } + } + } + return enrich ? self.enrich(result) : result; + } else { + return res; + } + }); +}; + +IdxDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const transaction = this.db.transaction("tag", "readonly"), + map = transaction.objectStore("tag"), + req = map.get(tag), + self = this; + + return promisfy(req).then(function (ids) { + if (!ids || !ids.length || offset >= ids.length) return []; + if (!limit && !offset) return ids; + const result = ids.slice(offset, offset + limit); + return enrich ? self.enrich(result) : result; + }); +}; + + +IdxDB.prototype.enrich = function (ids) { + if ("object" != typeof ids) { + ids = [ids]; + } + const transaction = this.db.transaction("reg", "readonly"), + map = transaction.objectStore("reg"), + promises = []; + + for (let i = 0; i < ids.length; i++) { + promises[i] = promisfy(map.get(ids[i])); + } + return Promise.all(promises).then(function (docs) { + for (let i = 0; i < docs.length; i++) { + docs[i] = { + id: ids[i], + doc: docs[i] ? JSON.parse(docs[i]) : null + }; + } + return docs; + }); +}; + + +IdxDB.prototype.has = function (id) { + const transaction = this.db.transaction("reg", "readonly"), + map = transaction.objectStore("reg"), + req = map.getKey(id); + + return promisfy(req); +}; + +IdxDB.prototype.search = null; + +// IdxDB.prototype.has = function(ref, key, ctx){ +// const transaction = this.db.transaction(ref, "readonly"); +// const map = transaction.objectStore(ref); +// const req = map.getKey(ctx ? ctx + ":" + key : key); +// return promisfy(req); +// }; + +IdxDB.prototype.info = function () { + // todo +}; + +/** + * @param {!string} ref + * @param {!string} modifier + * @param {!Function} task + */ + +IdxDB.prototype.transaction = function (ref, modifier, task) { + + let store = this.trx[ref + ":" + modifier]; + if (store) return task.call(this, store); + + let transaction = this.db.transaction(ref, modifier); + this.trx[ref + ":" + modifier] = store = transaction.objectStore(ref); + + return new Promise((resolve, reject) => { + transaction.onerror = err => { + this.trx[ref + ":" + modifier] = null; + transaction.abort(); + transaction = store = null; + reject(err); + //db.close; + }; + transaction.oncomplete = res => { + this.trx[ref + ":" + modifier] = null; + transaction = store = null; + resolve(res || !0); + //db.close; + }; + return task.call(this, store); + }); +}; + +IdxDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction("map", "readwrite", function (store) { + + for (const item of flexsearch.map) { + const key = item[0], + value = item[1]; + + if (!value.length) continue; + + if (_replace) { + store.put(value, key); + continue; + } + + store.get(key).onsuccess = function () { + let result = this.result, + changed; + + if (result && result.length) { + const maxlen = Math.max(result.length, value.length); + for (let i = 0, res, val; i < maxlen; i++) { + val = value[i]; + if (val && val.length) { + res = result[i]; + if (res && res.length) { + for (let j = 0; j < val.length; j++) { + res.push(val[j]); + } + changed = 1; + //result[i] = res.concat(val); + //result[i] = new Set([...result[i], ...value[i]]); + //result[i] = result[i].union(new Set(value[i])); + } else { + result[i] = val; + changed = 1; + //result[i] = new Set(value[i]) + } + } + } + } else { + result = value; + changed = 1; + //result = []; + //for(let i = 0; i < value.length; i++){ + // if(value[i]) result[i] = new Set(value[i]); + //} + } + + changed && store.put(result, key); + }; + } + }); + + await this.transaction("ctx", "readwrite", function (store) { + + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + + for (const item of ctx_value) { + const key = item[0], + value = item[1]; + + if (!value.length) continue; + + if (_replace) { + store.put(value, ctx_key + ":" + key); + continue; + } + + store.get(ctx_key + ":" + key).onsuccess = function () { + let result = this.result, + changed; + + if (result && result.length) { + const maxlen = Math.max(result.length, value.length); + for (let i = 0, res, val; i < maxlen; i++) { + val = value[i]; + if (val && val.length) { + res = result[i]; + if (res && res.length) { + for (let j = 0; j < val.length; j++) { + res.push(val[j]); + } + //result[i] = res.concat(val); + changed = 1; + } else { + result[i] = val; + changed = 1; + } + } + } + } else { + result = value; + changed = 1; + } + + changed && store.put(result, ctx_key + ":" + key); + }; + } + } + }); + + if (flexsearch.store) { + await this.transaction("reg", "readwrite", function (store) { + for (const item of flexsearch.store) { + const id = item[0], + doc = item[1]; + + store.put("object" == typeof doc ? JSON.stringify(doc) : 1, id); + } + }); + } else if (!flexsearch.bypass) { + await this.transaction("reg", "readwrite", function (store) { + for (const id of flexsearch.reg.keys()) { + store.put(1, id); + } + }); + } + + if (flexsearch.tag) { + await this.transaction("tag", "readwrite", function (store) { + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + + store.get(tag).onsuccess = function () { + let result = this.result; + result = result && result.length ? result.concat(ids) : ids; + store.put(result, tag); + }; + } + }); + } + + // TODO + // await this.transaction("cfg", "readwrite", function(store){ + // store.put({ + // "charset": flexsearch.charset, + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "fastupdate": flexsearch.fastupdate, + // "compress": flexsearch.compress, + // "encoder": { + // "minlength": flexsearch.encoder.minlength + // }, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }, "current"); + // }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + + flexsearch.tag && flexsearch.tag.clear(); + + flexsearch.store && flexsearch.store.clear(); + + flexsearch.document || flexsearch.reg.clear(); +}; + +/** + * @param {IDBCursorWithValue} cursor + * @param {Array} ids + * @param {boolean=} _tag + */ + +function handle(cursor, ids, _tag) { + + const arr = cursor.value; + let changed, + parse, + count = 0; + + + for (let x = 0, result; x < arr.length; x++) { + // tags has no resolution layer + if (result = _tag ? arr : arr[x]) { + for (let i = 0, pos, id; i < ids.length; i++) { + id = ids[i]; + pos = result.indexOf(parse ? parseInt(id, 10) : id); + if (0 > pos && !parse && "string" == typeof id && !isNaN(id)) { + pos = result.indexOf(parseInt(id, 10)); + pos && (parse = 1); + } + if (0 <= pos) { + changed = 1; + if (1 < result.length) { + result.splice(pos, 1); + } else { + arr[x] = []; + break; + } + } + } + + count += result.length; + } + if (_tag) break; + } + + if (!count) { + + cursor.delete(); + //store.delete(cursor.key); + } else if (changed) { + + //await new Promise(resolve => { + cursor.update(arr); //.onsuccess = resolve; + //}); + } + + cursor.continue(); +} + +IdxDB.prototype.remove = function (ids) { + + if ("object" != typeof ids) { + ids = [ids]; + } + + return (/** @type {!Promise} */Promise.all([this.transaction("map", "readwrite", function (store) { + store.openCursor().onsuccess = function () { + const cursor = this.result; + cursor && handle(cursor, ids); + }; + }), this.transaction("ctx", "readwrite", function (store) { + store.openCursor().onsuccess = function () { + const cursor = this.result; + cursor && handle(cursor, ids); + }; + }), this.transaction("tag", "readwrite", function (store) { + store.openCursor().onsuccess = function () { + const cursor = this.result; + cursor && handle(cursor, ids, !0); + }; + }), + // let filtered = []; + this.transaction("reg", "readwrite", function (store) { + for (let i = 0; i < ids.length; i++) { + store.delete(ids[i]); + } + // return new Promise(resolve => { + // store.openCursor().onsuccess = function(){ + // const cursor = this.result; + // if(cursor){ + // const id = cursor.value; + // if(ids.includes(id)){ + // filtered.push(id); + // cursor.delete(); + // } + // cursor.continue(); + // } + // else{ + // resolve(); + // } + // }; + // }); + }) + // ids = filtered; + ]) + ); +}; + +/** + * @param {IDBRequest} req + * @param {Function=} callback + * @return {!Promise} + */ + +function promisfy(req, callback) { + return new Promise((resolve, reject) => { + /** @this IDBRequest */ + req.onsuccess = function () { + callback && callback(this.result); + resolve(this.result); + }; + /** @this IDBRequest */ + req.oncomplete = function () { + callback && callback(this.result); + resolve(this.result); + }; + req.onerror = reject; + req = null; + }); +} \ No newline at end of file diff --git a/dist/module-debug/db/mongo/index.js b/dist/module-debug/db/mongo/index.js new file mode 100644 index 0000000..0475df2 --- /dev/null +++ b/dist/module-debug/db/mongo/index.js @@ -0,0 +1,584 @@ +import { MongoClient } from "mongodb"; +const defaults = { + host: "localhost", + port: "27017", + user: null, + pass: null +}, + VERSION = 1, + fields = ["map", "ctx", "tag", "reg", "cfg"]; + +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +let CLIENT, + DB = Object.create(null); + + +/** + * @constructor + * @implements StorageInterface + */ + +export default function MongoDB(name, config = {}) { + if (!(this instanceof MongoDB)) { + return new MongoDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = "flexsearch" + (name ? "-" + sanitize(name) : ""); + this.field = config.field ? "-" + sanitize(config.field) : ""; + this.type = config.type || ""; + this.db = config.db || DB[this.id] || CLIENT || null; + this.trx = !1; + this.support_tag_search = /* tag? */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + Object.assign(defaults, config); + this.db && delete defaults.db; +} + +// MongoDB.mount = function(flexsearch){ +// return new this().mount(flexsearch); +// }; + +MongoDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +async function createCollection(db, ref, field) { + switch (ref) { + case "map": + await db.createCollection("map" + field); + await db.collection("map" + field).createIndex({ key: 1 }); + await db.collection("map" + field).createIndex({ id: 1 }); + break; + case "ctx": + await db.createCollection("ctx" + field); + await db.collection("ctx" + field).createIndex({ ctx: 1, key: 1 }); + await db.collection("ctx" + field).createIndex({ id: 1 }); + break; + case "tag": + await db.createCollection("tag" + field); + await db.collection("tag" + field).createIndex({ tag: 1 }); + await db.collection("tag" + field).createIndex({ id: 1 }); + break; + case "reg": + await db.createCollection("reg"); + await db.collection("reg").createIndex({ id: 1 }); + break; + case "cfg": + await db.createCollection("cfg" + field); + } +} + +MongoDB.prototype.open = async function () { + + if (!this.db) { + if (!(this.db = DB[this.id])) { + if (!(this.db = CLIENT)) { + + let url = defaults.url; + if (!url) { + url = defaults.user ? `mongodb://${defaults.user}:${defaults.pass}@${defaults.host}:${defaults.port}` : `mongodb://${defaults.host}:${defaults.port}`; + } + this.db = CLIENT = new MongoClient(url); + await this.db.connect(); + } + } + } + + if (this.db.db) { + this.db = DB[this.id] = this.db.db(this.id); + } + + const collections = await this.db.listCollections().toArray(); + + for (let i = 0, found; i < fields.length; i++) { + found = !1; + + for (let j = 0; j < collections.length; j++) { + if (collections[j].name === fields[i] + ("reg" !== fields[i] ? this.field : "")) { + found = !0; + break; + } + } + if (!found) { + await createCollection(this.db, fields[i], this.field); + } + } + + return this.db; +}; + +MongoDB.prototype.close = function () { + this.db.close(); + this.db = null; + return this; +}; + +MongoDB.prototype.destroy = async function () { + await Promise.all([this.db.dropCollection("map" + this.field), this.db.dropCollection("ctx" + this.field), this.db.dropCollection("tag" + this.field), this.db.dropCollection("cfg" + this.field), this.db.dropCollection("reg")]); + this.close(); +}; + +async function clear(ref) { + await this.db.dropCollection(ref); + await createCollection(this.db, ref, this.field); +} + +MongoDB.prototype.clear = function () { + return Promise.all([clear.call(this, "map" + this.field), clear.call(this, "ctx" + this.field), clear.call(this, "tag" + this.field), clear.call(this, "cfg" + this.field), clear.call(this, "reg")]); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + if (!enrich) for (let i = 0; i < rows.length; i++) { + rows[i] = rows[i].id; + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +MongoDB.prototype.get = async function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = !1, tags) { + let rows, + params = ctx ? { ctx, key } : { key }; + + if (!enrich && !tags) { + rows = await this.db.collection((ctx ? "ctx" : "map") + this.field).find(params, { projection: { _id: 0, res: 1, id: 1 }, limit, skip: offset }).toArray(); + } else { + const project = { _id: 0, id: 1 }, + stmt = [{ $match: params }]; + + + if (!resolve) { + project.res = 1; + } + + if (enrich) { + project.doc = "$doc.doc"; + stmt.push({ $lookup: { + from: "reg", + localField: "id", + foreignField: "id", + as: "doc" + } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }); + } + + if (tags) { + + const match = {}; + for (let i = 0, count = 1; i < tags.length; i += 2) { + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push({ $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "id", + foreignField: "id", + as: "tag" + count + } }); + count++; + } + + stmt.push({ $project: project }, { $match: match }, { $project: { id: 1, doc: 1 } }); + } else { + stmt.push({ $project: project }); + } + + stmt.push({ $sort: { res: 1 } }, { $skip: offset }, { $limit: limit }); + + rows = []; + const result = await this.db.collection((ctx ? "ctx" : "map") + this.field).aggregate(stmt); + + while (!0) { + const row = await result.next(); + if (row) rows.push(row);else break; + } + } + + return create_result(rows, resolve, enrich); +}; + +MongoDB.prototype.tag = async function (tag, limit = 0, offset = 0, enrich = !1) { + + if (!enrich) { + + const rows = await this.db.collection("tag" + this.field).find({ tag }, { projection: { _id: 0, id: 1 }, limit, skip: offset }).toArray(); + return create_result(rows, !0, !1); + } else { + + let rows = []; + const result = await this.db.collection("tag" + this.field).aggregate([{ $match: { tag } }, { $skip: offset }, { $limit: limit }, { $lookup: { + from: "reg", + localField: "id", + foreignField: "id", + as: "doc" + } }, { $project: { _id: 0, id: 1, doc: "$doc.doc" } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }]); + + while (!0) { + const row = await result.next(); + if (row) rows.push(row);else break; + } + + return rows; + } +}; + +MongoDB.prototype.enrich = function (ids) { + if ("object" != typeof ids) { + ids = [ids]; + } + return this.db.collection("reg").find({ id: { $in: ids } }, { projection: { _id: 0, id: 1, doc: 1 } }).toArray(); +}; + +MongoDB.prototype.has = function (id) { + return this.db.collection("reg").countDocuments({ id }, { limit: 1 }); +}; + +MongoDB.prototype.search = async function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let result = [], + rows; + + if (1 < query.length && flexsearch.depth) { + let params = [], + keyword = query[0], + term; + + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + params.push({ + ctx: swap ? term : keyword, + key: swap ? keyword : term + }); + keyword = term; + } + + let project = resolve ? { _id: 1 } : { _id: 1, res: 1 }; + + const stmt = [{ $match: { $or: params } }, { $group: { + _id: "$id", + res: suggest ? { $sum: 1 } : { $min: 1 }, + count: { $sum: 1 } + } }]; + + suggest || stmt.push({ $match: { count: query.length - 1 } }); + + if (enrich) { + project.doc = "$doc.doc"; + stmt.push({ $lookup: { + from: "reg", + localField: "_id", + foreignField: "id", + as: "doc" + } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }); + } + + if (tags) { + + const match = {}; + for (let i = 0, count = 1; i < tags.length; i += 2) { + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push({ $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "_id", + foreignField: "id", + as: "tag" + count + } }); + count++; + } + + stmt.push({ $project: project }, { $match: match }); + } else { + stmt.push({ $project: project }); + } + + stmt.push({ $sort: suggest ? { count: -1, res: 1 } : { res: 1 } }, { $skip: offset }, { $limit: limit }); + + if (tags) { + project = { _id: 1 }; + if (!resolve) project.res = 1; + if (enrich) project.doc = 1; + + stmt.push({ $project: project }); + } + + rows = await this.db.collection("ctx" + this.field).aggregate(stmt); + } else { + + let project = resolve ? { _id: 1 } : { _id: 1, res: 1 }; + + const stmt = [{ $match: { + key: { $in: query } + } }, { $group: { + _id: "$id", + res: suggest ? { $sum: 1 } : { $min: 1 }, + count: { $sum: 1 } + } }]; + + suggest || stmt.push({ $match: { count: query.length } }); + + if (enrich) { + project.doc = "$doc.doc"; + stmt.push({ $lookup: { + from: "reg", + localField: "_id", + foreignField: "id", + as: "doc" + } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }); + } + + if (tags) { + + const match = {}; + for (let i = 0, count = 1; i < tags.length; i += 2) { + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push({ $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "_id", + foreignField: "id", + as: "tag" + count + } }); + count++; + } + + stmt.push({ $project: project }, { $match: match }); + } else { + stmt.push({ $project: project }); + } + + stmt.push({ $sort: suggest ? { count: -1, res: 1 } : { res: 1 } }, { $skip: offset }, { $limit: limit }); + + if (tags) { + project = { _id: 1 }; + if (!resolve) project.res = 1; + if (enrich) project.doc = 1; + + stmt.push({ $project: project }); + } + + rows = await this.db.collection("map" + this.field).aggregate(stmt); + } + + while (!0) { + const row = await rows.next(); + if (row) { + if (resolve && !enrich) { + result.push(row._id); + } else { + row.id = row._id; + delete row._id; + result.push(row); + } + } else break; + } + + if (resolve && !enrich) { + return result; + } else { + return create_result(result, resolve, enrich); + } +}; + +MongoDB.prototype.info = function () { + // todo +}; + +MongoDB.prototype.transaction = function (task) { + // not supported + return task.call(this); +}; + +MongoDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + if (flexsearch.map.size) { + let data = []; + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + this.type || (this.type = typeof ids[0]); + for (let j = 0; j < ids.length; j++) { + data.push({ + key: key, + res: i, + id: ids[j] + }); + } + } + } + } + if (data.length) { + await this.db.collection("map" + this.field).insertMany(data); + flexsearch.map.clear(); + } + } + + if (flexsearch.ctx.size) { + let data = []; + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + for (let j = 0; j < ids.length; j++) { + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: ids[j] + }); + } + } + } + } + } + if (data.length) { + await this.db.collection("ctx" + this.field).insertMany(data); + flexsearch.ctx.clear(); + } + } + + if (flexsearch.tag) { + let data = []; + if (flexsearch.tag) { + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let j = 0; j < ids.length; j++) { + data.push({ tag, id: ids[j] }); + } + } + } + if (data.length) { + await this.db.collection("tag" + this.field).insertMany(data); + flexsearch.tag.clear(); + } + } + + let data = []; + if (flexsearch.store) { + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + data.push({ id, doc }); + } + } else if (!flexsearch.bypass) { + for (const id of flexsearch.reg.keys()) { + data.push({ id }); + } + } + if (data.length) { + await this.db.collection("reg").insertMany(data); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); + } + + // TODO + // await this.db.collection("cfg" + this.field).insertOne({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }); +}; + +MongoDB.prototype.remove = function (ids) { + + if (!ids && 0 !== ids) return; + if ("object" != typeof ids) { + ids = [ids]; + } + + // if(this.type !== "string" && typeof ids[0] !== "number"){ + // ids = ids.map(item => parseInt(item, 10)); + // } + + return Promise.all([this.db.collection("map" + this.field).deleteMany({ id: { $in: ids } }), this.db.collection("ctx" + this.field).deleteMany({ id: { $in: ids } }), this.db.collection("tag" + this.field).deleteMany({ id: { $in: ids } }), this.db.collection("reg").deleteMany({ id: { $in: ids } })]); +}; \ No newline at end of file diff --git a/dist/module-debug/db/postgres/index.js b/dist/module-debug/db/postgres/index.js new file mode 100644 index 0000000..33cb63a --- /dev/null +++ b/dist/module-debug/db/postgres/index.js @@ -0,0 +1,971 @@ + + +import pg_promise from "pg-promise"; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { concat, toArray } from "../../common.js"; +const defaults = { + schema: "flexsearch", + user: "postgres", + pass: "postgres", + name: "postgres", + host: "localhost", + port: "5432" +}, + pgp = pg_promise(), + VERSION = 1, + MAXIMUM_QUERY_VARS = 16000, + fields = ["map", "ctx", "reg", "tag", "cfg"], + types = { + text: "text", + char: "text", + varchar: "text", + string: "text", + number: "int", + numeric: "int", + integer: "int", + smallint: "int", + tinyint: "int", + mediumint: "int", + int: "int", + int8: "int", + uint8: "int", + int16: "int", + uint16: "int", + int32: "int", + uint32: "bigint", + int64: "bigint", + bigint: "bigint" +}; + + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +let DB, TRX; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function PostgresDB(name, config = {}) { + if (!(this instanceof PostgresDB)) { + return new PostgresDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = (config.schema ? sanitize(config.schema) : defaults.schema) + (name ? "_" + sanitize(name) : ""); + this.field = config.field ? "_" + sanitize(config.field) : ""; + this.type = config.type ? types[config.type.toLowerCase()] : "text"; + this.support_tag_search = /* tag? */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + if (!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); + this.db = DB || (DB = config.db || null); + Object.assign(defaults, config); + this.db && delete defaults.db; +} + +PostgresDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +PostgresDB.prototype.open = async function () { + + if (!this.db) { + this.db = DB || (DB = pgp(`postgres://${defaults.user}:${encodeURIComponent(defaults.pass)}@${defaults.host}:${defaults.port}/${defaults.name}`)); + } + + const exist = await this.db.oneOrNone(` + SELECT EXISTS ( + SELECT 1 + FROM information_schema.schemata + WHERE schema_name = '${this.id}' + ); + `); + if (!exist || !exist.exists) { + await this.db.none(`CREATE SCHEMA IF NOT EXISTS ${this.id};`); + } + + for (let i = 0; i < fields.length; i++) { + const exist = await this.db.oneOrNone(` + SELECT EXISTS ( + SELECT 1 FROM pg_tables + WHERE schemaname = '${this.id}' AND tablename = '${fields[i] + ("reg" !== fields[i] ? this.field : "")}' + ); + `); + if (exist && exist.exists) continue; + + const type = "text" === this.type ? "varchar(128)" : this.type; + + switch (fields[i]) { + case "map": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.map${this.field}( + key varchar(128) NOT NULL, + res smallint NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_map_index${this.field} + ON ${this.id}.map${this.field} (key); + CREATE INDEX IF NOT EXISTS ${this.id}_map_id${this.field} + ON ${this.id}.map${this.field} (id); + `); + break; + + case "ctx": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.ctx${this.field}( + ctx varchar(128) NOT NULL, + key varchar(128) NOT NULL, + res smallint NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_ctx_index${this.field} + ON ${this.id}.ctx${this.field} (ctx, key); + CREATE INDEX IF NOT EXISTS ${this.id}_ctx_id${this.field} + ON ${this.id}.ctx${this.field} (id); + `); + break; + + case "tag": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.tag${this.field}( + tag varchar(128) NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_tag_index${this.field} + ON ${this.id}.tag${this.field} (tag); + CREATE INDEX IF NOT EXISTS ${this.id}_tag_id${this.field} + ON ${this.id}.tag${this.field} (id); + `); + break; + + case "reg": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.reg( + id ${type} NOT NULL + CONSTRAINT ${this.id}_reg_pk + PRIMARY KEY, + doc text DEFAULT NULL + ); + `).catch(() => { + // document indexes will try to create same registry table + // and the "IF NOT EXISTS" did not apply on parallel + }); + break; + + case "cfg": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.cfg${this.field}( + cfg text NOT NULL + ); + `); + break; + } + } + + return this.db; +}; + +PostgresDB.prototype.close = function () { + this.db.close && this.db.close(); + this.db = DB = null; + return this; +}; + +PostgresDB.prototype.destroy = async function () { + await this.db.none(` + DROP TABLE IF EXISTS ${this.id}.map${this.field}; + DROP TABLE IF EXISTS ${this.id}.ctx${this.field}; + DROP TABLE IF EXISTS ${this.id}.tag${this.field}; + DROP TABLE IF EXISTS ${this.id}.cfg${this.field}; + DROP TABLE IF EXISTS ${this.id}.reg; + `); + this.close(); +}; + +PostgresDB.prototype.clear = function () { + return this.db.none(` + TRUNCATE TABLE ${this.id}.map${this.field}; + TRUNCATE TABLE ${this.id}.ctx${this.field}; + TRUNCATE TABLE ${this.id}.tag${this.field}; + TRUNCATE TABLE ${this.id}.cfg${this.field}; + TRUNCATE TABLE ${this.id}.reg; + `); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + for (let i = 0; i < rows.length; i++) { + if (enrich) { + if (rows[i].doc) { + rows[i].doc = JSON.parse(rows[i].doc); + } + } else { + rows[i] = rows[i].id; + } + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +PostgresDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = !1, tags) { + let rows, + stmt = '', + params = ctx ? [ctx, key] : [key], + table = this.id + (ctx ? ".ctx" : ".map") + this.field; + + if (tags) { + for (let i = 0, count = params.length + 1; i < tags.length; i += 2) { + stmt += ` AND ${table}.id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = $${count++})`; + params.push(tags[i + 1]); + } + } + if (ctx) { + rows = this.db.any(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE ctx = $1 AND key = $2 ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, params); + } else { + rows = this.db.any(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE key = $1 ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, params); + } + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +PostgresDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const table = this.id + ".tag" + this.field, + promise = this.db.any(` + SELECT ${table}.id + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE tag = $1 + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, [tag]); + + enrich || promise.then(function (rows) { + return create_result(rows, !0, !1); + }); + return promise; +}; + +PostgresDB.prototype.enrich = async function (ids) { + let result = []; + if ("object" != typeof ids) { + ids = [ids]; + } + for (let count = 0; count < ids.length;) { + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + let stmt = ""; + for (let i = 1; i <= chunk.length; i++) { + stmt += (stmt ? "," : "") + "$" + i; + } + const res = await this.db.any(` + SELECT id, doc + FROM ${this.id}.reg + WHERE id IN (${stmt})`, ids); + if (res && res.length) { + for (let i = 0, doc; i < res.length; i++) { + if (doc = res[i].doc) { + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + return 1 === result.length ? result[0] : 1 < result.length ? concat(result) : result; +}; + +PostgresDB.prototype.has = function (id) { + return this.db.oneOrNone("SELECT EXISTS(SELECT 1 FROM " + this.id + ".reg WHERE id = $1)", [id]); +}; + +PostgresDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let rows; + + if (1 < query.length && flexsearch.depth) { + let where = "", + params = [], + keyword = query[0], + term, + count = 1; + + // variant new + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + where += (where ? " OR " : "") + `(ctx = $${count++} AND key = $${count++})`; + params.push(swap ? term : keyword, swap ? keyword : term); + keyword = term; + } + + if (tags) { + where = "(" + where + ")"; + for (let i = 0; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = $${count++})`; + params.push(tags[i + 1]); + } + } + + rows = this.db.any(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.ctx${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + (query.length - 1)} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, params); + + // variant 1 + + // for(let i = 1, count = 1; i < query.length; i++){ + // where += (where ? " UNION " : "") + ` + // SELECT id, res + // FROM ${this.id}.ctx${this.field} + // WHERE ctx = $${count++} AND key = $${count++} + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params.push( + // swap ? term : keyword, + // swap ? keyword : term + // ); + // keyword = term; + // } + // + // rows = await db.any(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, params); + } else { + let where = "", + count = 1, + query_length = query.length; + + for (let i = 0; i < query_length; i++) { + where += (where ? "," : "") + "$" + count++; + } + where = "key " + (1 < query_length ? "IN (" + where + ")" : "= " + where); + + if (tags) { + where = "(" + where + ")"; + for (let i = 0; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = $${count++})`; + query.push(tags[i + 1]); + } + } + + rows = this.db.any(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.map${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + query_length} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, query); + + // variant 1 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " UNION " : "") + ` + // SELECT id, res + // FROM ${ this.id }.map${ this.field } + // WHERE key = $${i} + // `; + // } + // rows = await db.any(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, query); + + // variant 2 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " AND EXISTS " : "") + `(SELECT FROM ${this.id}.map${this.field} as d WHERE d.id = t.id AND d.key = $` + i + ")"; + // } + // rows = await db.any(` + // SELECT t.id, min(t.res) + // FROM ${this.id}.map${this.field} AS t + // WHERE EXISTS ${where} + // GROUP BY t.id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + + // variant 3 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " INTERSECT " : "") + `SELECT id FROM ${this.id}.map${this.field} WHERE key = $` + i; + // } + // rows = await db.any(` + // WITH filtering (id) AS(${where}) + // SELECT t.id, min(t.res) + // FROM ${this.id}.map${this.field} AS t + // JOIN filtering USING (id) + // GROUP BY t.id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + + // variant 4 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " INTERSECT " : "") + `SELECT id FROM ${this.id}.map${this.field} WHERE key = $` + i; + // } + // rows = await db.any(` + // SELECT id, min(res) + // FROM ${this.id}.map${this.field} + // WHERE id IN (${where}) + // GROUP BY id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + } + + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +PostgresDB.prototype.info = function () { + // todo +}; + +// PostgresDB.prototype.transaction = async function(task){ +// const self = this; +// if(TRX){ +// return TRX.then(function(){ +// return self.transaction(task); +// //task.call(self, TRX); +// }); +// } +// TRX = await this.db.tx(async function(trx){ +// await task.call(self, trx); +// }); +// TRX = null; +// }; + +PostgresDB.prototype.transaction = function (task) { + if (TRX) { + return task.call(this, TRX); + } + const self = this; + return (TRX || this.db).tx(function (trx) { + return task.call(self, TRX = trx); + }).finally(function () { + TRX = null; + }); +}; + +PostgresDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + + //console.log("tasks:", tasks) + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction(function (trx) { + + const batch = []; + + // Datastore + Registry + + if (flexsearch.store) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["id", "doc"], { + table: this.id + ".reg" + }); + + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + // const migration = checkMigration.call(this, id); + // migration && await migration; + data.push({ id, doc: doc && JSON.stringify(doc) }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + // while(data.length){ + // let next; + // if(data.length > MAXIMUM_QUERY_VARS){ + // next = data.slice(MAXIMUM_QUERY_VARS); + // data = data.slice(0, MAXIMUM_QUERY_VARS); + // } + // let insert = pgp.helpers.insert(data, stmt); + // trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')); + // if(next) data = next; + // else break; + // } + } + + // Registry Only + + else if (!flexsearch.bypass) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["id"], { + table: this.id + ".reg" + }); + + for (const id of flexsearch.reg.keys()) { + // const migration = checkMigration.call(this, id); + // migration && await migration; + data.push({ id }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Default Index + + if (flexsearch.map.size) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["key", "res", "id"], { + table: this.id + ".map" + this.field + }); + + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + //this.type || (this.type = typeof ids[0]); + // let stmt = "($1,$2,$3)"; + // let params = [key, i, ids[0]]; + for (let j = 0; j < ids.length; j++) { + // stmt += ",($1,$2,$3)"; + // params.push(key, i, ids[j]); + //trx.none(`INSERT INTO ${config.schema}.map${self.field} (key, res, id) VALUES ($1,$2,$3)`, [key, i, ids[j]]); + data.push({ + key: key, + res: i, + id: ids[j] + }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + } + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Context Index + + if (flexsearch.ctx.size) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["ctx", "key", "res", "id"], { + table: this.id + ".ctx" + this.field + }); + + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + // let stmt = "(?,?,?)"; + // let params = [ctx_key + ":" + key, i, ids[0]]; + for (let j = 0; j < ids.length; j++) { + // stmt += ",(?,?,?)"; + // params.push(ctx_key + ":" + key, i, ids[j]); + //trx.none("INSERT INTO " + config.schema + ".ctx" + self.field + " (ctx, key, res, id) VALUES ($1,$2,$3,$4)", [ctx_key, key, i, ids[j]]); + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: ids[j] + }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + } + } + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Tag Index + + if (flexsearch.tag) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["tag", "id"], { + table: this.id + ".tag" + this.field + }); + + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let j = 0; j < ids.length; j++) { + data.push({ tag, id: ids[j] }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Field Configuration + + // TODO + // trx.none("INSERT INTO " + this.id + ".cfg" + this.field + " (cfg) VALUES ($1)", [ + // JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }) + // ]); + + //return Promise.all(batch); + + if (batch.length) { + return trx.batch(batch); + } + }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); +}; + +PostgresDB.prototype.remove = function (ids) { + + if (!ids && 0 !== ids) { + return; + } + if ("object" != typeof ids) { + ids = [ids]; + } + if (!ids.length) { + return; + } + + // ids = [ids]; + // return this.db.none( + // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1);", [ids] + // ); + + // ids = [ids]; + // return Promise.all([ + // this.db.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids) + // ]); + + return this.transaction(function (trx) { + + //console.log("remove:", ids); + + // ids = [ids]; + // trx.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids); + + // ids = [ids]; + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", ids); + // return; + + // return trx.none( + // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1);", [ids] + // ); + + // while(ids.length){ + // let next; + // let stmt = ""; + // let chunk = 0; + // let migration; + // for(let i = 0; i < ids.length; i++){ + // stmt += (stmt ? "," : "") + "$" + (i + 1); // + "::text"; + // // maximum count of variables supported + // if(++chunk === MAXIMUM_QUERY_VARS){ + // next = ids.slice(MAXIMUM_QUERY_VARS); + // ids = ids.slice(0, MAXIMUM_QUERY_VARS); + // break; + // } + // } + // + // trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id IN (" + stmt + ")", ids) + // ]); + // + // // trx.none( + // // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".reg WHERE id IN (" + stmt + ");", ids + // // ); + // + // if(next) ids = next; + // else break; + // } + + ids = [ids]; + return trx.batch([trx.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), trx.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), trx.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), trx.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids)]); + + // ids = [ids]; + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", ids) + // ]); + + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id IN ($1:csv)", [ids]) + // ]); + + // const type = self.type === "text" ? "text" : "int"; + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + self.field + " WHERE id = ANY($1::" + type + "[])", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + self.field + " WHERE id = ANY($1::" + type + "[])", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY($1::" + type + "[])", [ids]) + // ]); + + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + self.field + " WHERE id = ANY($1)", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + self.field + " WHERE id = ANY($1)", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY($1)", [ids]) + // ]); + }); +}; + +// if(this.type === "int" && typeof ids[0] === "string"){ +// ids = ids.map(item => parseInt(item, 10)); +// } +// if(this.type === "bigint" && typeof ids[0] === "string"){ +// ids = ids.map(item => BigInt(item)); +// } +// if(this.type === "string" && typeof ids[0] === "number"){ +// ids = ids.map(item => item + ""); +// } +// this.type !== "string" && checkMigration.call(this, ids[0]); + +// function checkMigration(id){ +// if(this.type !== "string"){ +// if(typeof id === "object"){ +// id = id[0]; +// } +// if(typeof id === "string"){ +// this.type = "string"; +// return upgradeTextId.call(this); +// } +// if(this.type !== "bigint"){ +// if(id > 2 ** 31 - 1){ +// this.type = "bigint"; +// return upgradeBigIntId.call(this); +// } +// } +// } +// } +// +// function upgradeTextId(){ +// return this.db.none(` +// ALTER TABLE ${this.id}.map${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.ctx${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.tag${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.reg +// ALTER COLUMN id type varchar(128) +// USING id::text; +// `); +// } +// +// function upgradeBigIntId(){ +// return this.db.none(` +// ALTER TABLE ${this.id}.map${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.ctx${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.tag${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.reg +// ALTER COLUMN id type bigint +// USING id::bigint; +// `); +// } \ No newline at end of file diff --git a/dist/module-debug/db/redis/index.js b/dist/module-debug/db/redis/index.js new file mode 100644 index 0000000..dbf7c83 --- /dev/null +++ b/dist/module-debug/db/redis/index.js @@ -0,0 +1,500 @@ + + +import { createClient } from "redis"; +const defaults = { + host: "localhost", + port: "6379", + user: null, + pass: null +}, + VERSION = 1, + fields = ["map", "ctx", "reg", "tag", "doc", "cfg"]; + +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +let DB, TRX; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function RedisDB(name, config = {}) { + if (!(this instanceof RedisDB)) { + return new RedisDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = (name ? sanitize(name) : "flexsearch") + "|"; + this.field = config.field ? "-" + sanitize(config.field) : ""; + this.type = config.type || ""; + this.fastupdate = /* tag? */ /* stringify */ /* stringify */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + this.db = config.db || DB || null; + this.support_tag_search = !0; + //this.trx = false; + Object.assign(defaults, config); + this.db && delete defaults.db; +} + +// RedisDB.mount = function(flexsearch){ +// return new this().mount(flexsearch); +// }; + +RedisDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + // todo support + //this.fastupdate = flexsearch.fastupdate; + return this.open(); +}; + +RedisDB.prototype.open = async function () { + if (this.db) { + return this.db; + } + + let url = defaults.url; + if (!url) { + url = defaults.user ? `redis://${defaults.user}:${defaults.pass}@${defaults.host}:${defaults.port}` : `redis://${defaults.host}:${defaults.port}`; + } + return this.db = await createClient(url).on("error", err => console.error(err)).connect(); +}; + +RedisDB.prototype.close = async function () { + await this.db.disconnect(); // this.db.client.disconnect(); + this.db = null; + return this; +}; + +RedisDB.prototype.clear = function () { + return this.db.unlink([this.id + "map" + this.field, this.id + "ctx" + this.field, this.id + "tag" + this.field, this.id + "cfg" + this.field, this.id + "doc", this.id + "reg"]); +}; + +function create_result(range, type, resolve, enrich) { + if (resolve) { + for (let i = 0, tmp, id; i < range.length; i++) { + tmp = range[i]; + id = "number" === type ? parseInt(tmp.value || tmp, 10) : tmp.value || tmp; + range[i] = /*enrich + ? { id, doc: tmp.doc } + :*/id; + } + return range; + } else { + let result = []; + for (let i = 0, tmp, id, score; i < range.length; i++) { + tmp = range[i]; + id = "number" === type ? parseInt(tmp.value, 10) : tmp.value; + score = tmp.score; + result[score] || (result[score] = []); + result[score].push(enrich ? { id, doc: tmp.doc } : id); + } + return result; + } +} + +RedisDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = /* suggest */!1, tags) { + + if (tags) { + // flexsearch dummy + const query = ctx ? [ctx, key] : [key]; + // keyword first + return this.search({ depth: !!ctx }, query, limit, offset, !1, resolve, enrich, tags); + } + + const type = this.type, + self = this; + + let result; + + if (ctx) { + result = this.db[resolve ? "zRange" : "zRangeWithScores"](this.id + "ctx" + this.field + ":" + ctx + ":" + key, "" + offset, "" + (offset + limit - 1), { REV: !0 }); + } else { + result = this.db[resolve ? "zRange" : "zRangeWithScores"](this.id + "map" + this.field + ":" + key, "" + offset, "" + (offset + limit - 1), { REV: !0 }); + } + + return result.then(async function (range) { + if (!range.length) return range; + if (enrich) range = await self.enrich(range); + return create_result(range, type, resolve, enrich); + }); +}; + +RedisDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const self = this; + return this.db.sMembers(this.id + "tag" + this.field + ":" + tag).then(function (ids) { + if (!ids || !ids.length || offset >= ids.length) return []; + if (!limit && !offset) return ids; + const result = ids.slice(offset, offset + limit); + return enrich ? self.enrich(result) : result; + }); +}; + +RedisDB.prototype.enrich = function (ids) { + if ("object" != typeof ids) { + ids = [ids]; + } + return this.db.hmGet(this.id + "doc", ids).then(function (res) { + for (let i = 0; i < res.length; i++) { + res[i] = { + id: ids[i], + doc: res[i] && JSON.parse(res[i]) + }; + } + return res; + }); +}; + +RedisDB.prototype.has = function (id) { + return this.db.sIsMember(this.id + "reg", "" + id); +}; + +RedisDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let result; + + if (1 < query.length && flexsearch.depth) { + + const key = this.id + "ctx" + this.field + ":"; + let params = [], + keyword = query[0], + term; + + + for (let i = 1, swap; i < query.length; i++) { + term = query[i]; + swap = flexsearch.bidirectional && term > keyword; + params.push(key + (swap ? term : keyword) + ":" + (swap ? keyword : term)); + keyword = term; + } + query = params; + } else { + + const key = this.id + "map" + this.field + ":"; + for (let i = 0; i < query.length; i++) { + query[i] = key + query[i]; + } + } + + const type = this.type; + let key = this.id + "tmp:" + Math.random(); + + + if (suggest) { + if (tags) for (let i = 0; i < tags.length; i += 2) { + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + + const multi = this.db.multi().zUnionStore(key, query, { AGGREGATE: "SUM" }); + // Strict Tag Intersection: it does not put tags into union, instead it calculates the + // intersection against the term match union. This was the default behavior + // of Tag-Search. But putting everything into union will also provide suggestions derived + // from tags when no term was matched. + + result = multi[resolve ? "zRange" : "zRangeWithScores"](key, "" + offset, "" + (offset + limit - 1), { REV: !0 }).unlink(key).exec(); + } else { + if (tags) for (let i = 0; i < tags.length; i += 2) { + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + result = this.db.multi().zInterStore(key, query, { AGGREGATE: "MIN" })[resolve ? "zRange" : "zRangeWithScores"](key, "" + offset, "" + (offset + limit - 1), { REV: !0 }).unlink(key).exec(); + } + + const self = this; + return result.then(async function (range) { + range = suggest && tags + // take the 3rd result from batch return + ? range[2] + // take the 2nd result from batch return + : range[1]; + if (!range.length) return range; + if (enrich) range = await self.enrich(range); + return create_result(range, type, resolve, enrich); + }); +}; + +RedisDB.prototype.info = function () { + // todo +}; + +RedisDB.prototype.transaction = async function (task, callback) { + + if (TRX) { + return task.call(this, TRX); + } + + TRX = this.db.multi(); + let promise1 = /*await*/task.call(this, TRX), + promise2 = TRX.exec(); + + TRX = null; + callback && promise.then(callback); + await Promise.all([promise1, promise2]); +}; + +RedisDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = "" + task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg, !0)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction(function (trx) { + + let refs = new Map(); + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + + let result = []; + for (let j = 0; j < ids.length; j++) { + result.push({ + score: i, + value: "" + ids[j] + }); + } + if ("number" == typeof ids[0]) { + this.type = "number"; + } + + const ref = this.id + "map" + this.field + ":" + key; + trx.zAdd(ref, result); + // if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + // trx.sAdd("ref" + this.field + ":" + ids[j], ref); + // } + if (this.fastupdate) for (let j = 0, id; j < ids.length; j++) { + // Map performs bad when pushing numeric-like values as key + // id = ids[j]; + // let tmp = refs.get(id); + // tmp || refs.set(id, tmp = []); + // tmp.push(ref); + id = ids[j]; + let tmp = refs.get(id); + tmp || refs.set(id, tmp = []); + tmp.push(ref); + } + } + } + } + // if(this.fastupdate) for(let item of refs){ + // const key = item[0]; + // const value = item[1]; + // trx.sAdd("ref" + this.field + ":" + key, value); + // } + if (this.fastupdate) for (const item of refs) { + const key = item[0], + value = item[1]; + + trx.sAdd(this.id + "ref" + this.field + ":" + key, value); + } + + refs = new Map(); + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + let result = []; + for (let j = 0; j < ids.length; j++) { + result.push({ score: i, value: "" + ids[j] }); + } + if ("number" == typeof ids[0]) { + this.type = "number"; + } + const ref = this.id + "ctx" + this.field + ":" + ctx_key + ":" + key; + trx.zAdd(ref, result); + // if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + // trx.sAdd("ref" + this.field + ":" + ids[j], ref); + // } + if (this.fastupdate) for (let j = 0, id; j < ids.length; j++) { + // Map performs bad when pushing numeric-like values as key + // id = ids[j]; + // let tmp = refs.get(id); + // tmp || refs.set(id, tmp = []); + // tmp.push(ref); + id = ids[j]; + let tmp = refs.get(id); + tmp || refs.set(id, tmp = []); + tmp.push(ref); + } + } + } + } + } + + if (this.fastupdate) for (const item of refs) { + const key = item[0], + value = item[1]; + + trx.sAdd(this.id + "ref" + this.field + ":" + key, value); + } + + if (flexsearch.store) { + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + doc && trx.hSet(this.id + "doc", "" + id, JSON.stringify(doc)); + } + } + if (!flexsearch.bypass) { + let ids = toArray(flexsearch.reg, !0); + if (ids.length) { + trx.sAdd(this.id + "reg", ids); + } + } + + if (flexsearch.tag) { + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + let result = []; + // for(let i = 0; i < ids.length; i++){ + // result.push({ + // score: 0, + // value: "" + ids[i] + // }); + // } + if ("number" == typeof ids[0]) { + for (let i = 0; i < ids.length; i++) { + result[i] = "" + ids[i]; + } + } else { + result = ids; + } + trx.sAdd(this.id + "tag" + this.field + ":" + tag, result); + } + } + + // TODO + // trx.set(this.id + "cfg" + this.field, JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // })); + }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); +}; + +RedisDB.prototype.remove = function (ids) { + + if (!ids && 0 !== ids) { + return; + } + if ("object" != typeof ids) { + ids = [ids]; + } + if (!ids.length) { + return; + } + + return this.transaction(async function (trx) { + + while (ids.length) { + let next; + if (10000 < ids.length) { + next = ids.slice(10000); + ids = ids.slice(0, 10000); + } + + if ("number" == typeof ids[0]) { + for (let i = 0; i < ids.length; i++) { + ids[i] = "" + ids[i]; + } + } + + const check = await this.db.smIsMember(this.id + "reg", ids); + + for (let i = 0, id; i < ids.length; i++) { + if (!check[i]) continue; + id = "" + ids[i]; + + if (this.fastupdate) { + // const refs = new Map(); + const ref = await this.db.sMembers(this.id + "ref" + this.field + ":" + id); + if (ref) { + for (let j = 0; j < ref.length; j++) { + // let tmp = refs.get(ref[j]); + // tmp || refs.set(ref[j], tmp = []); + // tmp.push(id); + trx.zRem(ref[j], id); + } + trx.unlink(this.id + "ref" + this.field + ":" + id); + } + // for(let item of refs){ + // //console.log(item[0], item[1]) + // trx.zRem(item[0], item[1]); + // } + } + + trx.hDel(this.id + "doc", id); + trx.sRem(this.id + "reg", id); + } + + if (next) ids = next;else break; + } + }); +}; \ No newline at end of file diff --git a/dist/module-debug/db/sqlite/index.js b/dist/module-debug/db/sqlite/index.js new file mode 100644 index 0000000..c71a6dd --- /dev/null +++ b/dist/module-debug/db/sqlite/index.js @@ -0,0 +1,794 @@ + + +//const sqlite3 = require("sqlite3").verbose(); +import sqlite3 from "sqlite3"; +import path from "path"; +import StorageInterface from "../interface.js"; +import { concat, toArray } from "../../common.js"; +import Document from "../../document.js"; + +const VERSION = 1, + MAXIMUM_QUERY_VARS = 16000, + fields = ["map", "ctx", "reg", "tag", "cfg"], + types = { + text: "text", + char: "text", + varchar: "text", + string: "text", + number: "int", + numeric: "int", + integer: "int", + smallint: "int", + tinyint: "int", + mediumint: "int", + int: "int", + int8: "int", + uint8: "int", + int16: "int", + uint16: "int", + int32: "int", + uint32: "bigint", + int64: "bigint", + bigint: "bigint" +}; + + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +// global transaction to keep track of database lock +const TRX = Object.create(null), + DB = Object.create(null); + + +/** + * @constructor + * @implements StorageInterface + */ + +export default function SqliteDB(name, config = {}) { + if (!(this instanceof SqliteDB)) { + return new SqliteDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + //field = "Test-456"; + this.id = config.path || (":memory:" === name ? name : "flexsearch" + (name ? "-" + sanitize(name) : "") + ".sqlite"); + this.field = config.field ? "_" + sanitize(config.field) : ""; + this.support_tag_search = /* tag? */ /* stringify */ /* stringify */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + this.db = config.db || DB[this.id] || null; + this.type = config.type ? types[config.type.toLowerCase()] : "string"; + if (!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); +} + +SqliteDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +SqliteDB.prototype.open = async function () { + + if (!this.db) { + + if (!(this.db = DB[this.id])) { + + let filepath = this.id; + if (":memory:" !== filepath) { + // skip absolute path + if ("/" !== filepath[0] && "\\" !== filepath[0]) { + // current working directory + const dir = process.cwd(); + filepath = path.join(dir, this.id); + } + } + + this.db = DB[this.id] = new sqlite3.Database(filepath); + } + } + + const db = this.db; + + for (let i = 0; i < fields.length; i++) { + const exist = await this.promisfy({ + method: "get", + stmt: "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?) as exist", + params: [fields[i] + ("reg" === fields[i] ? "" : this.field)] + }); + if (!exist || !exist.exist) { + let stmt, stmt_index; + switch (fields[i]) { + case "map": + stmt = ` + CREATE TABLE IF NOT EXISTS main.map${this.field}( + key TEXT NOT NULL, + res INTEGER NOT NULL, + id ${this.type} NOT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS map_key_index${this.field} + ON map${this.field} (key); + CREATE INDEX IF NOT EXISTS map_id_index${this.field} + ON map${this.field} (id); + `; + break; + + case "ctx": + stmt = ` + CREATE TABLE IF NOT EXISTS main.ctx${this.field}( + ctx TEXT NOT NULL, + key TEXT NOT NULL, + res INTEGER NOT NULL, + id ${this.type} NOT NULL + ); + + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS ctx_key_index${this.field} + ON ctx${this.field} (ctx, key); + CREATE INDEX IF NOT EXISTS ctx_id_index${this.field} + ON ctx${this.field} (id); + `; + break; + + case "tag": + stmt = ` + CREATE TABLE IF NOT EXISTS main.tag${this.field}( + tag TEXT NOT NULL, + id ${this.type} NOT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS tag_index${this.field} + ON tag${this.field} (tag); + CREATE INDEX IF NOT EXISTS tag_id_index${this.field} + ON tag${this.field} (id); + `; + break; + + case "reg": + stmt = ` + CREATE TABLE IF NOT EXISTS main.reg( + id ${this.type} NOT NULL + CONSTRAINT reg_pk${this.field} + PRIMARY KEY, + doc TEXT DEFAULT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS reg_index + ON reg (id); + `; + break; + + case "cfg": + stmt = ` + CREATE TABLE IF NOT EXISTS main.cfg${this.field} ( + cfg TEXT NOT NULL + ); + `; + break; + } + + await new Promise(function (resolve, reject) { + db.exec(stmt, function (err, rows) { + if (err) return reject(err); + stmt_index ? db.exec(stmt_index, function (err, rows) { + if (err) return reject(err); + resolve(rows); + }) : resolve(rows); + }); + }); + } + } + + db.exec("PRAGMA optimize = 0x10002"); + + return db; +}; + +SqliteDB.prototype.close = function () { + this.db.close(); + this.db = null; + return this; +}; + +SqliteDB.prototype.destroy = async function () { + await this.transaction(function () { + this.db.run("DROP TABLE IF EXISTS main.map" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.ctx" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.tag" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.cfg" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.reg;"); + }); + this.close(); +}; + +SqliteDB.prototype.clear = function () { + return this.transaction(function () { + this.db.run("DELETE FROM main.map" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.ctx" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.tag" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.cfg" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.reg WHERE 1;"); + }); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + for (let i = 0; i < rows.length; i++) { + if (enrich) { + if (rows[i].doc) { + rows[i].doc = JSON.parse(rows[i].doc); + } + } else { + rows[i] = rows[i].id; + } + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +SqliteDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = /* suggest */!1, tags) { + let result, + stmt = '', + params = ctx ? [ctx, key] : [key], + table = "main." + (ctx ? "ctx" : "map") + this.field; + + if (tags) { + for (let i = 0; i < tags.length; i += 2) { + stmt += ` AND ${table}.id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + params.push(tags[i + 1]); + } + } + if (ctx) { + result = this.promisfy({ + method: "all", + stmt: ` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${table}.id + ` : ""} + WHERE ctx = ? AND key = ? ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params + }); + } else { + result = this.promisfy({ + method: "all", + stmt: ` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${table}.id + ` : ""} + WHERE key = ? ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params + }); + } + return result.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +SqliteDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const table = "main.tag" + this.field, + promise = this.promisfy({ + method: "all", + stmt: ` + SELECT ${table}.id + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${table}.id + ` : ""} + WHERE tag = ? + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params: [tag] + }); + + enrich || promise.then(function (rows) { + return create_result(rows, !0, !1); + }); + return promise; +}; + +function build_params(length) { + + let stmt = ",?"; + for (let i = 1; i < length;) { + if (i <= length - i) { + stmt += stmt; + i *= 2; + } else { + stmt += stmt.substring(0, 2 * (length - i)); + break; + } + } + return stmt.substring(1); +} + +SqliteDB.prototype.enrich = function (ids) { + const result = [], + promises = []; + + if ("object" != typeof ids) { + ids = [ids]; + } + + for (let count = 0; count < ids.length;) { + + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + + // let stmt = "?"; + // for(let i = 1; i < chunk.length; i++){ + // stmt += ",?"; + // } + + // 10x faster string concatenation + let stmt = build_params(chunk.length); + + promises.push(this.promisfy({ + method: "all", + stmt: `SELECT id, doc FROM main.reg WHERE id IN (${stmt})`, + params: chunk + })); + } + + return Promise.all(promises).then(function (promises) { + + for (let i = 0, res; i < promises.length; i++) { + res = promises[i]; + if (res && res.length) { + for (let i = 0, doc; i < res.length; i++) { + if (doc = res[i].doc) { + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + + return 1 === result.length ? result[0] : 1 < result.length ? concat(result) : result; + }); +}; + +SqliteDB.prototype.has = function (id) { + return this.promisfy({ + method: "get", + stmt: `SELECT EXISTS(SELECT 1 FROM main.reg WHERE id = ?) as exist`, + params: [id] + }).then(function (result) { + return result && result.exist; + }); +}; + +SqliteDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let rows; + + if (1 < query.length && flexsearch.depth) { + let stmt = "", + params = [], + keyword = query[0], + term; + + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + stmt += (stmt ? " OR " : "") + `(ctx = ? AND key = ?)`; + params.push(swap ? term : keyword, swap ? keyword : term); + keyword = term; + } + + if (tags) { + stmt = "(" + stmt + ")"; + for (let i = 0; i < tags.length; i += 2) { + stmt += ` AND id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + params.push(tags[i + 1]); + } + } + + rows = this.promisfy({ + method: "all", + stmt: ` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM main.ctx${this.field} + WHERE ${stmt} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + (query.length - 1)} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params + }); + + // variant 1 + // for(let i = 1; i < query.length; i++){ + // stmt += (stmt ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM main.ctx${this.field} + // WHERE ctx = ? AND key = ? + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params.push(swap ? term : keyword, swap ? keyword : term); + // keyword = term; + // } + // + // rows = await this.promisfy({ + // method: "all", + // stmt: ` + // SELECT id/*, res */ + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${stmt}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, + // params + // }); + } else { + let stmt = "", + query_length = query.length; + + for (let i = 0; i < query_length; i++) { + stmt += (stmt ? " OR " : "") + `key = ?`; + } + + if (tags) { + stmt = "(" + stmt + ")"; + for (let i = 0; i < tags.length; i += 2) { + stmt += ` AND id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + query.push(tags[i + 1]); + } + } + + rows = this.promisfy({ + method: "all", + stmt: ` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM main.map${this.field} + WHERE ${stmt} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + query_length} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params: query + }); + + // variant 1 + // for(let i = 0; i < query.length; i++){ + // stmt += (stmt ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM main.map${ this.field } + // WHERE key = ? + // `; + // } + // + // rows = await this.promisfy({ + // method: "all", + // stmt: ` + // SELECT id/*, res*/ + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${stmt}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, + // params: query + // }); + } + + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +SqliteDB.prototype.info = function () { + // todo +}; + +SqliteDB.prototype.transaction = function (task, callback) { + + const self = this; + + if (TRX[this.id]) { + return TRX[this.id].then(function () { + return self.transaction(task, callback); + }); + } + + const db = this.db; + + return TRX[this.id] = new Promise(function (resolve, reject) { + db.exec("PRAGMA optimize"); + db.exec('PRAGMA busy_timeout = 5000'); + db.exec("BEGIN"); + db.parallelize(function () { + task.call(self); + }); + db.exec("COMMIT", function (err, rows) { + TRX[self.id] = null; + if (err) return reject(err); + callback && callback(rows); + resolve(rows); + db.exec("PRAGMA shrink_memory"); + }); + }); +}; + +SqliteDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction(function () { + + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + let stmt = "", + params = []; + + + for (let j = 0; j < ids.length; j++) { + stmt += (stmt ? "," : "") + "(?,?,?)"; + params.push(key, i, ids[j]); + // maximum count of variables supported + if (j === ids.length - 1 || params.length + 3 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.map" + this.field + " (key, res, id) VALUES " + stmt, params); + stmt = ""; + params = []; + } + //this.db.run("INSERT INTO map (key, res, id) VALUES (?, ?, ?) ON CONFLICT DO NOTHING", [key, i, ids[j]]); + } + } + } + } + //}); + //await this.transaction(function(){ + + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + let stmt = "", + params = []; + + + for (let j = 0; j < ids.length; j++) { + stmt += (stmt ? "," : "") + "(?,?,?,?)"; + params.push(ctx_key, key, i, ids[j]); + // maximum count of variables supported + if (j === ids.length - 1 || params.length + 4 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.ctx" + this.field + " (ctx, key, res, id) VALUES " + stmt, params); + stmt = ""; + params = []; + } + } + } + } + } + } + //}); + //await this.transaction(function(){ + + if (flexsearch.store) { + let stmt = "", + chunk = []; + + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + stmt += (stmt ? "," : "") + "(?,?)"; + chunk.push(id, "object" == typeof doc ? JSON.stringify(doc) : doc || null); + if (chunk.length + 2 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.reg (id, doc) VALUES " + stmt, chunk); + stmt = ""; + chunk = []; + } + } + if (chunk.length) { + this.db.run("INSERT INTO main.reg (id, doc) VALUES " + stmt, chunk); + } + } else if (!flexsearch.bypass) { + let ids = toArray(flexsearch.reg); + for (let count = 0; count < ids.length;) { + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + const stmt = build_params(chunk.length); + this.db.run("INSERT INTO main.reg (id) VALUES (" + stmt + ")", chunk); + } + } + //}); + //await this.transaction(function(){ + + if (flexsearch.tag) { + let stmt = "", + chunk = []; + + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let i = 0; i < ids.length; i++) { + stmt += (stmt ? "," : "") + "(?,?)"; + chunk.push(tag, ids[i]); + } + if (chunk.length + 2 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.tag" + this.field + " (tag, id) VALUES " + stmt, chunk); + stmt = ""; + chunk = []; + } + } + if (chunk.length) { + this.db.run("INSERT INTO main.tag" + this.field + " (tag, id) VALUES " + stmt, chunk); + } + } + }); + + // TODO + //await this.transaction(function(){ + // this.db.run("INSERT INTO main.cfg" + this.field + " (cfg) VALUES (?)", [JSON.stringify({ + // "charset": flexsearch.charset, + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "fastupdate": flexsearch.fastupdate, + // "compress": flexsearch.compress, + // "encoder": { + // "minlength": flexsearch.encoder.minlength + // }, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // })]); + //}); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); +}; + +SqliteDB.prototype.remove = function (ids) { + + if ("object" != typeof ids) { + ids = [ids]; + } + + let next; + // maximum count of variables supported + if (ids.length > MAXIMUM_QUERY_VARS) { + next = ids.slice(MAXIMUM_QUERY_VARS); + ids = ids.slice(0, MAXIMUM_QUERY_VARS); + } + + const self = this; + return this.transaction(function () { + const stmt = build_params(ids.length); + this.db.run("DELETE FROM main.map" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.ctx" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.tag" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.reg WHERE id IN (" + stmt + ")", ids); + }).then(function (res) { + return next ? self.remove(next) : res; + }); +}; + +SqliteDB.prototype.promisfy = function (opt) { + const db = this.db; + return new Promise(function (resolve, reject) { + db[opt.method](opt.stmt, opt.params || [], function (err, rows) { + opt.callback && opt.callback(rows); + err ? reject(err) : resolve(rows); + }); + }); +}; \ No newline at end of file diff --git a/dist/module-debug/document.js b/dist/module-debug/document.js index e0f7196..5c2a8df 100644 --- a/dist/module-debug/document.js +++ b/dist/module-debug/document.js @@ -6,68 +6,168 @@ * https://github.com/nextapps-de/flexsearch */ +import { DocumentOptions } from "./type.js"; import Index from "./index.js"; -import { DocumentInterface } from "./type.js"; -import Cache, { searchCache } from "./cache.js"; -import { create_object, is_array, is_string, is_object, parse_option, get_keys } from "./common.js"; -import apply_async from "./async.js"; -import { intersect, intersect_union } from "./intersect.js"; -import { exportDocument, importDocument } from "./serialize.js"; import WorkerIndex from "./worker/index.js"; +import Cache, { searchCache } from "./cache.js"; +import { is_string, is_object, parse_simple } from "./common.js"; +import apply_async from "./async.js"; +import { exportDocument, importDocument } from "./serialize.js"; +import { KeystoreMap, KeystoreSet } from "./keystore.js"; +import "./document/add.js"; +import "./document/search.js"; /** * @constructor - * @implements {DocumentInterface} - * @param {Object=} options - * @return {Document} + * @param {DocumentOptions=} options */ -function Document(options) { +export default function Document(options) { if (!(this instanceof Document)) { - return new Document(options); } const document = options.document || options.doc || options; - let opt; + let tmp, keystore; this.tree = []; this.field = []; this.marker = []; - this.register = create_object(); - this.key = (opt = document.key || document.id) && parse_tree(opt, this.marker) || "id"; - this.fastupdate = parse_option(options.fastupdate, /* append: */ /* skip update: */ /* skip_update: */!0); + this.key = (tmp = document.key || document.id) && parse_tree(tmp, this.marker) || "id"; - this.storetree = (opt = document.store) && !0 !== opt && []; - this.store = opt && create_object(); + keystore = options.keystore || 0; + keystore && (this.keystore = keystore); + this.fastupdate = !!options.fastupdate; + this.reg = this.fastupdate ? keystore ? new KeystoreMap(keystore) : new Map() : keystore ? new KeystoreSet(keystore) : new Set(); + // todo support custom filter function + this.storetree = (tmp = document.store || null) && /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ !== tmp && []; + this.store = tmp && (keystore ? new KeystoreMap(keystore) : new Map()); - // TODO case-insensitive tags - - this.tag = (opt = document.tag) && parse_tree(opt, this.marker); - this.tagindex = opt && create_object(); - - - this.cache = (opt = options.cache) && new Cache(opt); - - // do not apply cache again for the indexes - - options.cache = !1; - + this.cache = (tmp = options.cache || null) && new Cache(tmp); + // do not apply cache again for the indexes since .searchCache() + // is just a wrapper over .search() + options.cache = /* suggest */ /* append: */ /* enrich */!1; this.worker = options.worker; - // this switch is used by recall of promise callbacks - this.async = !1; /** @export */ this.index = parse_descriptor.call(this, options, document); + + this.tag = null; + // TODO case-insensitive tags? + if (tmp = document.tag) { + if ("string" == typeof tmp) { + tmp = [tmp]; + } + if (tmp.length) { + this.tag = new Map(); + this.tagtree = []; + this.tagfield = []; + for (let i = 0, params, field; i < tmp.length; i++) { + params = tmp[i]; + field = params.field || params; + if (!field) { + throw new Error("The tag field from the document descriptor is undefined."); + } + if (params.custom) { + this.tagtree[i] = params.custom; + } else { + this.tagtree[i] = parse_tree(field, this.marker); + if (params.filter) { + if ("string" == typeof this.tagtree[i]) { + // it needs an object to put a property to it + this.tagtree[i] = new String(this.tagtree[i]); + } + this.tagtree[i]._filter = params.filter; + } + } + // the tag fields needs to be hold by indices + this.tagfield[i] = field; + this.tag.set(field, new Map()); + } + } + } + + options.db && this.mount(options.db); } -export default Document; +Document.prototype.mount = function (db) { + + let fields = this.field; + + if (this.tag) { + // tag indexes are referenced by field + // move tags to their field indexes respectively + for (let i = 0, field; i < this.tagfield.length; i++) { + field = this.tagfield[i]; + let index = this.index.get(field); + if (!index) { + // create raw index when not exists + this.index.set(field, index = new Index({}, this.reg)); + // copy and push to the field selection + if (fields === this.field) { + fields = fields.slice(0); + } + // tag indexes also needs to be upgraded to db instances + fields.push(field); + } + // assign reference + index.tag = this.tag.get(field); + } + } + + const promises = [], + config = { + db: db.db, + type: db.type, + fastupdate: db.fastupdate + }; + + + // upgrade all indexes to db instances + for (let i = 0, index, field; i < fields.length; i++) { + config.field = field = fields[i]; + index = this.index.get(field); + const dbi = new db.constructor(db.id, config); + // take over the storage id + dbi.id = db.id; + promises[i] = dbi.mount(index); + // add an identification property + index.document = !0; + if (i) { + // the register has to export just one time + // also it's needed by the index for ID contain check + index.bypass = !0; + } else { + // the datastore has to export one time + index.store = this.store; + } + } + + this.async = !0; + this.db = !0; + return Promise.all(promises); +}; + +Document.prototype.commit = async function (replace, append) { + // parallel: + const promises = []; + for (const index of this.index.values()) { + promises.push(index.db.commit(index, replace, append)); + } + await Promise.all(promises); + this.reg.clear(); + // queued: + // for(const index of this.index.values()){ + // await index.db.commit(index, replace, append); + // } + // this.reg.clear(); +}; /** * @this Document @@ -75,11 +175,10 @@ export default Document; function parse_descriptor(options, document) { - const index = create_object(); + const index = new Map(); let field = document.index || document.field || document; if (is_string(field)) { - field = [field]; } @@ -88,7 +187,6 @@ function parse_descriptor(options, document) { key = field[i]; if (!is_string(key)) { - opt = key; key = key.field; } @@ -96,36 +194,55 @@ function parse_descriptor(options, document) { opt = is_object(opt) ? Object.assign({}, options, opt) : options; if (this.worker) { - - index[key] = new WorkerIndex(opt); - - if (!index[key].worker) { - + const worker = new WorkerIndex(opt); + index.set(key, worker); + if (!worker.worker) { + // fallback when not supported this.worker = !1; } } if (!this.worker) { - - index[key] = new Index(opt, this.register); + index.set(key, new Index(opt, this.reg)); + } + + if (opt.custom) { + this.tree[i] = opt.custom; + } else { + this.tree[i] = parse_tree(key, this.marker); + if (opt.filter) { + if ("string" == typeof this.tree[i]) { + // it needs an object to put a property to it + this.tree[i] = new String(this.tree[i]); + } + this.tree[i]._filter = opt.filter; + } } - this.tree[i] = parse_tree(key, this.marker); this.field[i] = key; } if (this.storetree) { - let store = document.store; + let stores = document.store; + if (is_string(stores)) stores = [stores]; - if (is_string(store)) { - - store = [store]; - } - - for (let i = 0; i < store.length; i++) { - - this.storetree[i] = parse_tree(store[i], this.marker); + for (let i = 0, store, field; i < stores.length; i++) { + store = stores[i]; + field = store.field || store; + if (store.custom) { + this.storetree[i] = store.custom; + store.custom._field = field; + } else { + this.storetree[i] = parse_tree(field, this.marker); + if (store.filter) { + if ("string" == typeof this.storetree[i]) { + // it needs an object to put a property to it + this.storetree[i] = new String(this.storetree[i]); + } + this.storetree[i]._filter = store.filter; + } + } } } @@ -138,291 +255,57 @@ function parse_tree(key, marker) { let count = 0; for (let i = 0; i < tree.length; i++) { - key = tree[i]; - - if (0 <= key.indexOf("[]")) { - + // todo apply some indexes e.g. [0], [-1], [0-2] + if ("]" === key[key.length - 1]) { key = key.substring(0, key.length - 2); - if (key) { - marker[count] = !0; } } - if (key) { - tree[count++] = key; } } if (count < tree.length) { - tree.length = count; } return 1 < count ? tree : tree[0]; } -// TODO support generic function created from string when tree depth > 1 - -function parse_simple(obj, tree) { - - if (is_string(tree)) { - - obj = obj[tree]; - } else { - - for (let i = 0; obj && i < tree.length; i++) { - - obj = obj[tree[i]]; - } - } - - return obj; -} - -// TODO support generic function created from string when tree depth > 1 - -function store_value(obj, store, tree, pos, key) { - - obj = obj[key]; - - // reached target field - - if (pos === tree.length - 1) { - - // store target value - - store[key] = obj; - } else if (obj) { - - if (is_array(obj)) { - - store = store[key] = Array(obj.length); - - for (let i = 0; i < obj.length; i++) { - - // do not increase pos (an array is not a field) - store_value(obj, store, tree, pos, i); - } - } else { - - store = store[key] || (store[key] = create_object()); - key = tree[++pos]; - - store_value(obj, store, tree, pos, key); - } - } -} - -function add_index(obj, tree, marker, pos, index, id, key, _append) { - - obj = obj[key]; - - if (obj) { - - // reached target field - - if (pos === tree.length - 1) { - - // handle target value - - if (is_array(obj)) { - - // append array contents so each entry gets a new scoring context - - if (marker[pos]) { - - for (let i = 0; i < obj.length; i++) { - - index.add(id, obj[i], !0, !0); - } - - return; - } - - // or join array contents and use one scoring context - - obj = obj.join(" "); - } - - index.add(id, obj, _append, !0); - } else { - - if (is_array(obj)) { - - for (let i = 0; i < obj.length; i++) { - - // do not increase index, an array is not a field - - add_index(obj, tree, marker, pos, index, id, i, _append); - } - } else { - - key = tree[++pos]; - - add_index(obj, tree, marker, pos, index, id, key, _append); - } - } - } -} - -/** - * - * @param id - * @param content - * @param {boolean=} _append - * @returns {Document|Promise} - */ - -Document.prototype.add = function (id, content, _append) { - - if (is_object(id)) { - - content = id; - id = parse_simple(content, this.key); - } - - if (content && (id || 0 === id)) { - - if (!_append && this.register[id]) { - - return this.update(id, content); - } - - for (let i = 0, tree, field; i < this.field.length; i++) { - - field = this.field[i]; - tree = this.tree[i]; - - if (is_string(tree)) { - - tree = [tree]; - } - - add_index(content, tree, this.marker, 0, this.index[field], id, tree[0], _append); - } - - if (this.tag) { - let tag = parse_simple(content, this.tag), - dupes = create_object(); - - - if (is_string(tag)) { - - tag = [tag]; - } - - for (let i = 0, key, arr; i < tag.length; i++) { - - key = tag[i]; - - if (!dupes[key]) { - - dupes[key] = 1; - arr = this.tagindex[key] || (this.tagindex[key] = []); - - if (!_append || !arr.includes(id)) { - - arr[arr.length] = id; - - // add a reference to the register for fast updates - - if (this.fastupdate) { - - const tmp = this.register[id] || (this.register[id] = []); - tmp[tmp.length] = arr; - } - } - } - } - } - - // TODO: how to handle store when appending contents? - - if (this.store && (!_append || !this.store[id])) { - - let store; - - if (this.storetree) { - - store = create_object(); - - for (let i = 0, tree; i < this.storetree.length; i++) { - - tree = this.storetree[i]; - - if (is_string(tree)) { - - store[tree] = content[tree]; - } else { - - store_value(content, store, tree, 0, tree[0]); - } - } - } - - this.store[id] = store || content; - } - } - - return this; -}; - Document.prototype.append = function (id, content) { - return this.add(id, content, !0); }; Document.prototype.update = function (id, content) { - return this.remove(id).add(id, content); }; Document.prototype.remove = function (id) { if (is_object(id)) { - id = parse_simple(id, this.key); } - if (this.register[id]) { + for (const index of this.index.values()) { + index.remove(id, !0); + } - for (let i = 0; i < this.field.length; i++) { - - // workers does not share the register - - this.index[this.field[i]].remove(id, !this.worker); - - if (this.fastupdate) { - - // when fastupdate was enabled all ids are removed - - break; - } - } + if (this.reg.has(id)) { if (this.tag) { - // when fastupdate was enabled all ids are already removed - if (!this.fastupdate) { + for (let field of this.tag.values()) { + for (let item of field) { + const tag = item[0], + ids = item[1], + pos = ids.indexOf(id); - for (let key in this.tagindex) { - const tag = this.tagindex[key], - pos = tag.indexOf(id); - - - if (-1 !== pos) { - - if (1 < tag.length) { - - tag.splice(pos, 1); - } else { - - delete this.tagindex[key]; + if (-1 < pos) { + 1 < ids.length ? ids.splice(pos, 1) : field.delete(tag); } } } @@ -430,297 +313,84 @@ Document.prototype.remove = function (id) { } if (this.store) { - - delete this.store[id]; + this.store.delete(id); } - delete this.register[id]; + this.reg.delete(id); + } + + // the cache could be used outside the InMemory store + if (this.cache) { + this.cache.remove(id); } return this; }; -/** - * @param {!string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @param {Array=} _resolve For internal use only. - * @returns {Promise|Array} - */ +Document.prototype.clear = function () { -Document.prototype.search = function (query, limit, options, _resolve) { + //const promises = []; - if (!options) { + for (const index of this.index.values()) { + // db index will add clear task + index.clear(); + // const promise = index.clear(); + // if(promise instanceof Promise){ + // promises.push(promise); + // } + } - if (!limit && is_object(query)) { - - options = /** @type {Object} */query; - query = ""; - } else if (is_object(limit)) { - - options = /** @type {Object} */limit; - limit = 0; + if (this.tag) { + for (const tags of this.tag.values()) { + tags.clear(); } } - let result = [], - result_field = [], - pluck, - enrich, - field, - tag, - bool, - offset, - count = 0; - - - if (options) { - - if (is_array(options)) { - - field = options; - options = null; - } else { - - query = options.query || query; - pluck = options.pluck; - field = pluck || options.index || options.field /*|| (is_string(options) && [options])*/; - tag = options.tag; - enrich = this.store && options.enrich; - bool = "and" === options.bool; - limit = options.limit || limit || 100; - offset = options.offset || 0; - - if (tag) { - - if (is_string(tag)) { - - tag = [tag]; - } - - // when tags is used and no query was set, - // then just return the tag indexes - - if (!query) { - - for (let i = 0, res; i < tag.length; i++) { - - res = get_tag.call(this, tag[i], limit, offset, enrich); - - if (res) { - - result[result.length] = res; - count++; - } - } - - return count ? result : []; - } - } - - if (is_string(field)) { - - field = [field]; - } - } + if (this.store) { + this.store.clear(); } - field || (field = this.field); - bool = bool && (1 < field.length || tag && 1 < tag.length); - - const promises = !_resolve && (this.worker || this.async) && []; - - // TODO solve this in one loop below - - for (let i = 0, res, key, len; i < field.length; i++) { - - let field_options; - - key = field[i]; - - if (!is_string(key)) { - - field_options = key; - key = field_options.field; - query = field_options.query || query; - limit = field_options.limit || limit; - enrich = field_options.enrich || enrich; - } - - if (promises) { - - promises[i] = this.index[key].searchAsync(query, limit, field_options || options); - - // just collect and continue - - continue; - } else if (_resolve) { - - res = _resolve[i]; - } else { - - // inherit options also when search? it is just for laziness, Object.assign() has a cost - - res = this.index[key].search(query, limit, field_options || options); - } - - len = res && res.length; - - if (tag && len) { - - const arr = []; - let count = 0; - - if (bool) { - - // prepare for intersection - - arr[0] = [res]; - } - - for (let y = 0, key, res; y < tag.length; y++) { - - key = tag[y]; - res = this.tagindex[key]; - len = res && res.length; - - if (len) { - - count++; - arr[arr.length] = bool ? [res] : res; - } - } - - if (count) { - - if (bool) { - - res = intersect(arr, limit || 100, offset || 0); - } else { - - res = intersect_union(res, arr); - } - - len = res.length; - } - } - - if (len) { - - result_field[count] = key; - result[count++] = res; - } else if (bool) { - - return []; - } - } - - if (promises) { - - const self = this; - - // anyone knows a better workaround of optionally having async promises? - // the promise.all() needs to be wrapped into additional promise, - // otherwise the recursive callback wouldn't run before return - - return new Promise(function (resolve) { - - Promise.all(promises).then(function (result) { - - resolve(self.search(query, limit, options, result)); - }); - }); - } - - if (!count) { - - // fast path "not found" - - return []; - } - - if (pluck && (!enrich || !this.store)) { - - // fast path optimization - - return result[0]; - } - - for (let i = 0, res; i < result_field.length; i++) { - - res = result[i]; - - if (res.length) { - - if (enrich) { - - res = apply_enrich.call(this, res); - } - } - - if (pluck) { - - return res; - } - - result[i] = { - - field: result_field[i], - result: res - }; - } - - return result; + return this; /*promises.length + ? Promise.all(promises) + :*/ }; -/** - * @this Document - */ - -function get_tag(key, limit, offset) { - let res = this.tagindex[key], - len = res && res.length - offset; -} - -/** - * @this Document - */ - -function apply_enrich(res) { - - const arr = Array(res.length); - - for (let x = 0, id; x < res.length; x++) { - - id = res[x]; - - arr[x] = { - - id: id, - doc: this.store[id] - }; - } - - return arr; -} - Document.prototype.contain = function (id) { - return !!this.register[id]; + if (this.db) { + return this.index.get(this.field[0]).db.has(id); + } + + return this.reg.has(id); +}; + +Document.prototype.cleanup = function () { + + for (const index of this.index.values()) { + index.cleanup(); + } + + return this; }; Document.prototype.get = function (id) { - return this.store[id]; + if (this.db) { + return this.index.get(this.field[0]).db.enrich(id).then(function (result) { + return result[0] && result[0].doc; + }); + } + + return this.store.get(id); }; -Document.prototype.set = function (id, data) { +Document.prototype.set = function (id, store) { - this.store[id] = data; + this.store.set(id, store); return this; }; - +// todo mo Document.prototype.searchCache = searchCache; diff --git a/dist/module-debug/document/add.js b/dist/module-debug/document/add.js new file mode 100644 index 0000000..e0cf5c9 --- /dev/null +++ b/dist/module-debug/document/add.js @@ -0,0 +1,244 @@ + +import { create_object, is_array, is_object, is_string, parse_simple } from "../common.js"; +import { KeystoreArray } from "../keystore.js"; +import Document from "../document.js"; + +/** + * + * @param id + * @param content + * @param {boolean=} _append + * @returns {Document|Promise} + */ + +Document.prototype.add = function (id, content, _append) { + + if (is_object(id)) { + + content = id; + id = parse_simple(content, this.key); + } + + if (content && (id || 0 === id)) { + + if (!_append && this.reg.has(id)) { + return this.update(id, content); + } + + // this.field does not include db tag indexes + for (let i = 0, tree; i < this.field.length; i++) { + + tree = this.tree[i]; + + const index = this.index.get(this.field[i]); + if ("function" == typeof tree) { + const tmp = tree(content); + if (tmp) { + index.add(id, tmp, /* suggest */ /* append: */!1, /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/); + } + } else { + const filter = tree._filter; + if (filter && !filter(content)) { + continue; + } + if (tree instanceof String) { + tree = ["" + tree]; + } else if (is_string(tree)) { + tree = [tree]; + } + add_index(content, tree, this.marker, 0, index, id, tree[0], _append); + } + } + + if (this.tag) { + + //console.log(this.tag, this.tagtree) + + for (let x = 0; x < this.tagtree.length; x++) { + let tree = this.tagtree[x], + field = this.tagfield[x], + ref = this.tag.get(field), + dupes = create_object(), + tags; + + + if ("function" == typeof tree) { + tags = tree(content); + if (!tags) continue; + } else { + const filter = tree._filter; + if (filter && !filter(content)) { + continue; + } + if (tree instanceof String) { + tree = "" + tree; + } + tags = parse_simple(content, tree); + } + + if (!ref || !tags) { + ref || console.warn("Tag '" + field + "' was not found"); + continue; + } + + if (is_string(tags)) { + tags = [tags]; + } + + for (let i = 0, tag, arr; i < tags.length; i++) { + + tag = tags[i]; + //console.log(this.tag, tag, key, field) + + if (!dupes[tag]) { + dupes[tag] = 1; + + let tmp = ref.get(tag); + + tmp ? arr = tmp : ref.set(tag, arr = []); + + if (!_append || ! /** @type {!Array|KeystoreArray} */arr.includes(id)) { + + // auto-upgrade to keystore array if max size exceeded + if (2147483647 === arr.length /*|| !(arr instanceof KeystoreArray)*/) { + const keystore = new KeystoreArray(arr); + if (this.fastupdate) { + for (let value of this.reg.values()) { + if (value.includes(arr)) { + value[value.indexOf(arr)] = keystore; + } + } + } + ref.set(tag, arr = keystore); + } + + + arr.push(id); + + // add a reference to the register for fast updates + if (this.fastupdate) { + const tmp = this.reg.get(id); + tmp ? tmp.push(arr) : this.reg.set(id, [arr]); + } + } + } + } + } + } + + if (this.store && (!_append || !this.store.has(id))) { + + let payload; + + if (this.storetree) { + + payload = create_object(); + + for (let i = 0, tree; i < this.storetree.length; i++) { + tree = this.storetree[i]; + + const filter = tree._filter; + if (filter && !filter(content)) { + continue; + } + let custom; + if ("function" == typeof tree) { + custom = tree(content); + if (!custom) continue; + tree = [tree._field]; + } else if (is_string(tree) || tree instanceof String) { + payload[tree] = content[tree]; + continue; + } + + store_value(content, payload, tree, 0, tree[0], custom); + } + } + + this.store.set(id, payload || content); + } + } + + return this; +}; + +// TODO support generic function created from string when tree depth > 1 + +/** + * @param obj + * @param store + * @param tree + * @param pos + * @param key + * @param {*=} custom + */ + +function store_value(obj, store, tree, pos, key, custom) { + + obj = obj[key]; + + // reached target field + if (pos === tree.length - 1) { + + // store target value + store[key] = custom || obj; + } else if (obj) { + + if (is_array(obj)) { + + store = store[key] = Array(obj.length); + + for (let i = 0; i < obj.length; i++) { + // do not increase pos (an array is not a field) + store_value(obj, store, tree, pos, i); + } + } else { + + store = store[key] || (store[key] = create_object()); + key = tree[++pos]; + store_value(obj, store, tree, pos, key); + } + } +} + +function add_index(obj, tree, marker, pos, index, id, key, _append) { + + if (obj = obj[key]) { + + // reached target field + if (pos === tree.length - 1) { + + // handle target value + if (is_array(obj)) { + + // append array contents so each entry gets a new scoring context + if (marker[pos]) { + for (let i = 0; i < obj.length; i++) { + index.add(id, obj[i], !0, !0); + } + return; + } + + // or join array contents and use one scoring context + obj = obj.join(" "); + } + + index.add(id, obj, _append, !0); + } else { + + if (is_array(obj)) { + for (let i = 0; i < obj.length; i++) { + // do not increase index, an array is not a field + add_index(obj, tree, marker, pos, index, id, i, _append); + } + } else { + key = tree[++pos]; + add_index(obj, tree, marker, pos, index, id, key, _append); + } + } + } else { + if (index.db) { + index.remove(id); + } + } +} \ No newline at end of file diff --git a/dist/module-debug/document/search.js b/dist/module-debug/document/search.js new file mode 100644 index 0000000..b482118 --- /dev/null +++ b/dist/module-debug/document/search.js @@ -0,0 +1,442 @@ + + +import { DocumentSearchOptions } from "../type.js"; +import { create_object, is_array, is_object, is_string } from "../common.js"; +import { intersect_union } from "../intersect.js"; +import Document from "../document.js"; + +let debug = /* suggest */ /* append: */ /* enrich */!1; + +/** + * @param {!string|DocumentSearchOptions} query + * @param {number|DocumentSearchOptions=} limit + * @param {DocumentSearchOptions=} options + * @param {Array=} _resolve For internal use only. + * @returns {Promise|Array} + */ + +Document.prototype.search = function (query, limit, options, _resolve) { + + if (!options) { + if (!limit && is_object(query)) { + options = /** @type {DocumentSearchOptions} */query; + query = ""; + } else if (is_object(limit)) { + options = /** @type {DocumentSearchOptions} */limit; + limit = 0; + } + } + + let result = [], + result_field = [], + pluck, + enrich, + merge, + suggest, + field, + tag, + offset, + count = 0; + + + if (options) { + + // todo remove support? + if (is_array(options)) { + field = options; + options = null; + } else { + + query = options.query || query; + pluck = options.pluck; + merge = options.merge; + field = pluck || options.field || options.index; + tag = this.tag && options.tag; + enrich = this.store && options.enrich; + suggest = options.suggest; + limit = options.limit || limit; + offset = options.offset || 0; + limit || (limit = 100); + + if (tag && (!this.db || !_resolve)) { + + if (tag.constructor !== Array) { + tag = [tag]; + } + + // Tag-Search + // ----------------------------- + + let pairs = []; + + for (let i = 0, field; i < tag.length; i++) { + field = tag[i]; + if (is_string(field)) { + throw new Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + // default array notation + if (field.field && field.tag) { + const value = field.tag; + if (value.constructor === Array) { + for (let k = 0; k < value.length; k++) { + pairs.push(field.field, value[k]); + } + } else { + pairs.push(field.field, value); + } + } + // shorter object notation + else { + const keys = Object.keys(field); + for (let j = 0, key, value; j < keys.length; j++) { + key = keys[j]; + value = field[key]; + if (value.constructor === Array) { + for (let k = 0; k < value.length; k++) { + pairs.push(key, value[k]); + } + } else { + pairs.push(key, value); + } + } + } + } + + if (!pairs.length) { + throw new Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + + // tag used as pairs from this point + tag = pairs; + + // when tags is used and no query was set, + // then just return the tag indexes + if (!query) { + + let promises = []; + if (pairs.length) for (let j = 0; j < pairs.length; j += 2) { + let ids; + if (this.db) { + const index = this.index.get(pairs[j]); + if (!index) { + console.warn("Tag '" + pairs[j] + ":" + pairs[j + 1] + "' will be skipped because there is no field '" + pairs[j] + "'."); + + continue; + } + + promises.push(ids = index.db.tag(pairs[j + 1], limit, offset, enrich)); + } else { + ids = get_tag.call(this, pairs[j], pairs[j + 1], limit, offset, enrich); + } + result.push({ + field: pairs[j], + tag: pairs[j + 1], + result: ids + }); + } + + if (promises.length) { + return Promise.all(promises).then(function (promises) { + for (let j = 0; j < promises.length; j++) { + result[j].result = promises[j]; + } + return result; + }); + } + + return result; + } + } + + // extend to multi field search by default + if (is_string(field)) { + field = [field]; + } + } + } + + field || (field = this.field); + let promises = !_resolve && (this.worker || this.async) && [], + db_tag_search; + + + // multi field search + // field could be a custom set of selected fields by this query + // db tag indexes are also included in this field list + for (let i = 0, res, key, len; i < field.length; i++) { + + key = field[i]; + + if (this.db && this.tag) { + // tree is missing when it is a tag-only index (db) + if (!this.tree[i]) { + continue; + } + } + + let field_options; + + if (!is_string(key)) { + field_options = key; + key = field_options.field; + query = field_options.query || query; + limit = field_options.limit || limit; + //offset = field_options.offset || offset; + suggest = field_options.suggest || suggest; + //enrich = SUPPORT_STORE && this.store && (field_options.enrich || enrich); + } + + if (_resolve) { + res = _resolve[i]; + } else { + let opt = field_options || options, + index = this.index.get(key); + + + if (tag) { + if (this.db) { + opt.tag = tag; + db_tag_search = index.db.support_tag_search; + opt.field = field; + } + if (!db_tag_search) { + opt.enrich = !1; + } + } + if (promises) { + promises[i] = index.searchAsync(query, limit, opt); + // restore enrich state + opt && enrich && (opt.enrich = enrich); + // just collect and continue + continue; + } else { + res = index.search(query, limit, opt); + // restore enrich state + opt && enrich && (opt.enrich = enrich); + } + } + + len = res && res.length; + + // todo when no term was matched but tag was retrieved extend suggestion to tags + // every field has to intersect against all selected tag fields + if (tag && len) { + + const arr = []; + let count = 0; + + // tags are only applied in resolve phase when it's a db + if (this.db && _resolve) { + if (!db_tag_search) { + + // retrieve tag results assigned to it's field + for (let y = field.length; y < _resolve.length; y++) { + let ids = _resolve[y], + len = ids && ids.length; + + + if (len) { + count++; + arr.push(ids); + } else if (!suggest) { + // no tags found + return result; + } + } + } + } else { + + // tag[] are pairs at this line + for (let y = 0, ids, len; y < tag.length; y += 2) { + ids = this.tag.get(tag[y]); + + if (!ids) { + console.warn("Tag '" + tag[y] + ":" + tag[y + 1] + "' will be skipped because there is no field '" + tag[y] + "'."); + + if (suggest) { + continue; + } else { + return result; + } + } + + ids = ids && ids.get(tag[y + 1]); + len = ids && ids.length; + + if (len) { + count++; + arr.push(ids); + } else if (!suggest) { + // no tags found + return result; + } + } + } + + if (count) { + res = intersect_union(res, arr); // intersect(arr, limit, offset) + len = res.length; + if (!len && !suggest) { + // nothing matched + return result; + } + // move counter back by 1 + count--; + } + } + + if (len) { + result_field[count] = key; + result.push(res); + count++; + } else if (1 === field.length) { + // fast path: nothing matched + return result; + } + } + + if (promises) { + if (this.db) { + // todo when a tag index is never a search index this could be extracted + // push tag promises to the end + if (tag && tag.length && !db_tag_search) { + for (let y = 0; y < tag.length; y += 2) { + // it needs to retrieve data from tag pairs + const index = this.index.get(tag[y]); + if (!index) { + console.warn("Tag '" + tag[y] + ":" + tag[y + 1] + "' was not found because there is no field '" + tag[y] + "'."); + + if (suggest) { + continue; + } else { + return result; + } + } + + promises.push(index.db.tag(tag[y + 1], limit, offset, !1)); + } + } + } + + const self = this; + + // TODO unroll this recursion + return Promise.all(promises).then(function (result) { + return result.length ? self.search(query, limit, options, /* resolve: */result) : result; + }); + } + + if (!count) { + return result; + } + if (pluck && (!enrich || !this.store)) { + return result[0]; + } + + promises = []; + + for (let i = 0, res; i < result_field.length; i++) { + + res = result[i]; + + if (enrich && res.length && !res[0].doc) { + if (!this.db) { + if (res.length) { + res = apply_enrich.call(this, res); + } + } else { + promises.push(res = this.index.get(this.field[0]).db.enrich(res)); + } + } + + if (pluck) { + return res; + } + + result[i] = { + field: result_field[i], + result: res + }; + } + + if (enrich && /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ && this.db && promises.length) { + return Promise.all(promises).then(function (promises) { + for (let j = 0; j < promises.length; j++) { + result[j].result = promises[j]; + } + return merge ? merge_fields(result, limit, offset) : result; + }); + } + + return merge ? merge_fields(result, limit, offset) : result; +}; + +// todo support Resolver +// todo when searching through multiple fields each term should +// be found at least by one field to get a valid match without +// using suggestion explicitly + +function merge_fields(fields, limit) { + const final = [], + set = create_object(); + + for (let i = 0, field, res; i < fields.length; i++) { + field = fields[i]; + res = field.result; + for (let j = 0, id, entry, tmp; j < res.length; j++) { + entry = res[j]; + id = entry.id; + tmp = set[id]; + if (!tmp) { + // offset was already applied on field indexes + // if(offset){ + // offset--; + // continue; + // } + // apply limit from last round, because just fields could + // be pushed without adding new results + if (final.length === limit) { + return final; + } + entry.field = set[id] = [field.field]; + final.push(entry); + } else { + tmp.push(field.field); + } + } + } + return final; +} + +/** + * @this Document + */ + +function get_tag(tag, key, limit, offset) { + let res = this.tag.get(tag); + if (!res) { + console.warn("Tag '" + tag + "' was not found"); + return []; + } + res = res && res.get(key); + res && res.length - offset; +} + +/** + * @this Document + */ + +function apply_enrich(res) { + + const arr = Array(res.length); + + for (let x = 0, id; x < res.length; x++) { + id = res[x]; + arr[x] = { + id: id, + doc: this.store.get(id) + }; + } + + return arr; +} \ No newline at end of file diff --git a/dist/module-debug/encoder.js b/dist/module-debug/encoder.js new file mode 100644 index 0000000..0b18c42 --- /dev/null +++ b/dist/module-debug/encoder.js @@ -0,0 +1,568 @@ + +import { parse_option } from "./common.js"; + +/* + +Custom Encoder +---------------- + +// Split a passed string into an Array of words: +function englishEncoder(string){ + return string.toLowerCase().split(/[^a-z]+/) +} + +// For CJK split a passed string into an Array of chars: +function chineseEncoder(string){ + return string.replace(/\s+/, "").split("") +} + +// Alternatively do not split the input: +function fixedEncoder(string){ + return [string] +} + +Built-in Encoder (Workflow) +---------------------------- +Pipeline: + 1. apply this.normalize: charset normalization: + applied on the whole input string e.g. lowercase, + will also apply on: filter, matcher, stemmer, mapper + 2. apply this.split: split input into terms (includes/excludes) + 3. apply this.filter (pre-filter) + 4. apply this.matcher (replace terms) + 5. apply this.stemmer (replace term endings) + 6. apply this.filter (post-filter) + 7. apply this.mapper (replace chars) + 8. apply this.replacer (custom regex) + 9. apply this.dedupe (letter deduplication) + 10. apply this.finalize +*/ + +const whitespace = /[^\p{L}\p{N}]+/u, + numeric_split_length = /(\d{3})/g, + numeric_split_prev_char = /(\D)(\d{3})/g, + numeric_split_next_char = /(\d{3})(\D)/g, + normalize = /[\u0300-\u036f]/g, + normalize_mapper = !normalize && new Map([ + +// Charset Normalization + +["ª", "a"], ["²", "2"], ["³", "3"], ["¹", "1"], ["º", "o"], ["¼", "1⁄4"], ["½", "1⁄2"], ["¾", "3⁄4"], ["à", "a"], ["á", "a"], ["â", "a"], ["ã", "a"], ["ä", "a"], ["å", "a"], ["ç", "c"], ["è", "e"], ["é", "e"], ["ê", "e"], ["ë", "e"], ["ì", "i"], ["í", "i"], ["î", "i"], ["ï", "i"], ["ñ", "n"], ["ò", "o"], ["ó", "o"], ["ô", "o"], ["õ", "o"], ["ö", "o"], ["ù", "u"], ["ú", "u"], ["û", "u"], ["ü", "u"], ["ý", "y"], ["ÿ", "y"], ["ā", "a"], ["ă", "a"], ["ą", "a"], ["ć", "c"], ["ĉ", "c"], ["ċ", "c"], ["č", "c"], ["ď", "d"], ["ē", "e"], ["ĕ", "e"], ["ė", "e"], ["ę", "e"], ["ě", "e"], ["ĝ", "g"], ["ğ", "g"], ["ġ", "g"], ["ģ", "g"], ["ĥ", "h"], ["ĩ", "i"], ["ī", "i"], ["ĭ", "i"], ["į", "i"], ["ij", "ij"], ["ĵ", "j"], ["ķ", "k"], ["ĺ", "l"], ["ļ", "l"], ["ľ", "l"], ["ŀ", "l"], ["ń", "n"], ["ņ", "n"], ["ň", "n"], ["ʼn", "n"], ["ō", "o"], ["ŏ", "o"], ["ő", "o"], ["ŕ", "r"], ["ŗ", "r"], ["ř", "r"], ["ś", "s"], ["ŝ", "s"], ["ş", "s"], ["š", "s"], ["ţ", "t"], ["ť", "t"], ["ũ", "u"], ["ū", "u"], ["ŭ", "u"], ["ů", "u"], ["ű", "u"], ["ų", "u"], ["ŵ", "w"], ["ŷ", "y"], ["ź", "z"], ["ż", "z"], ["ž", "z"], ["ſ", "s"], ["ơ", "o"], ["ư", "u"], ["dž", "dz"], ["lj", "lj"], ["nj", "nj"], ["ǎ", "a"], ["ǐ", "i"], ["ǒ", "o"], ["ǔ", "u"], ["ǖ", "u"], ["ǘ", "u"], ["ǚ", "u"], ["ǜ", "u"], ["ǟ", "a"], ["ǡ", "a"], ["ǣ", "ae"], ["æ", "ae"], ["ǽ", "ae"], ["ǧ", "g"], ["ǩ", "k"], ["ǫ", "o"], ["ǭ", "o"], ["ǯ", "ʒ"], ["ǰ", "j"], ["dz", "dz"], ["ǵ", "g"], ["ǹ", "n"], ["ǻ", "a"], ["ǿ", "ø"], ["ȁ", "a"], ["ȃ", "a"], ["ȅ", "e"], ["ȇ", "e"], ["ȉ", "i"], ["ȋ", "i"], ["ȍ", "o"], ["ȏ", "o"], ["ȑ", "r"], ["ȓ", "r"], ["ȕ", "u"], ["ȗ", "u"], ["ș", "s"], ["ț", "t"], ["ȟ", "h"], ["ȧ", "a"], ["ȩ", "e"], ["ȫ", "o"], ["ȭ", "o"], ["ȯ", "o"], ["ȱ", "o"], ["ȳ", "y"], ["ʰ", "h"], ["ʱ", "h"], ["ɦ", "h"], ["ʲ", "j"], ["ʳ", "r"], ["ʴ", "ɹ"], ["ʵ", "ɻ"], ["ʶ", "ʁ"], ["ʷ", "w"], ["ʸ", "y"], ["ˠ", "ɣ"], ["ˡ", "l"], ["ˢ", "s"], ["ˣ", "x"], ["ˤ", "ʕ"], ["ΐ", "ι"], ["ά", "α"], ["έ", "ε"], ["ή", "η"], ["ί", "ι"], ["ΰ", "υ"], ["ϊ", "ι"], ["ϋ", "υ"], ["ό", "ο"], ["ύ", "υ"], ["ώ", "ω"], ["ϐ", "β"], ["ϑ", "θ"], ["ϒ", "Υ"], ["ϓ", "Υ"], ["ϔ", "Υ"], ["ϕ", "φ"], ["ϖ", "π"], ["ϰ", "κ"], ["ϱ", "ρ"], ["ϲ", "ς"], ["ϵ", "ε"], ["й", "и"], ["ѐ", "е"], ["ё", "е"], ["ѓ", "г"], ["ї", "і"], ["ќ", "к"], ["ѝ", "и"], ["ў", "у"], ["ѷ", "ѵ"], ["ӂ", "ж"], ["ӑ", "а"], ["ӓ", "а"], ["ӗ", "е"], ["ӛ", "ә"], ["ӝ", "ж"], ["ӟ", "з"], ["ӣ", "и"], ["ӥ", "и"], ["ӧ", "о"], ["ӫ", "ө"], ["ӭ", "э"], ["ӯ", "у"], ["ӱ", "у"], ["ӳ", "у"], ["ӵ", "ч"] + +// Term Separators + +// ["'", ""], // it's -> its +// ["´", ""], +// ["`", ""], +// ["’", ""], +// ["ʼ", ""], + +// Numeric-Separators Chars Removal + +// [",", ""], +// [".", ""] + +// Non-Whitespace Separators + +// already was split by default via p{P} +// ["-", " "], +// [":", " "], +// ["_", " "], +// ["|", " "], +// ["/", " "], +// ["\\", " "] +]); // /[\p{Z}\p{S}\p{P}\p{C}]+/u; +//const numeric_split = /(\d{3})/g; + +//.replace(/(\d{3})/g, "$1 ") +//.replace(/([^\d])([\d])/g, "$1 $2") +//.replace(/([\d])([^\d])/g, "$1 $2") +// '´`’ʼ., + +/** + * @param options + * @constructor + */ + +export default function Encoder(options = {}) { + + if (!(this instanceof Encoder)) { + return new Encoder(...arguments); + } + + for (let i = 0; i < arguments.length; i++) { + this.assign(arguments[i]); + } +} + +Encoder.prototype.assign = function (options) { + + // if(options.assign){ + // //options = Object.assign({}, options.assign, options); + // this.assign(options.assign); + // } + + // let tmp = options["normalize"]; + // if(typeof tmp === "function"){ + // const old = this.normalize; + // if(typeof old === "function"){ + // const current = tmp; + // tmp = function(){ + // old(); + // current(); + // } + // } + // } + + /** + * pre-processing string input + * @type {Function|boolean} + */ + this.normalize = /** @type {Function|boolean} */parse_option(options.normalize, /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, this.normalize); + + // { + // letter: true, + // number: true, + // whitespace: true, + // symbol: true, + // punctuation: true, + // control: true, + // char: "" + // } + + let include = options.include, + tmp = include || options.exclude || options.split; + + + if ("object" == typeof tmp) { + let numeric = !include, + regex = ""; + + // split on whitespace by default + options.include || (regex += "\\p{Z}"); + if (tmp.letter) { + regex += "\\p{L}"; + } + if (tmp.number) { + regex += "\\p{N}"; + numeric = !!include; + } + if (tmp.symbol) { + regex += "\\p{S}"; + } + if (tmp.punctuation) { + regex += "\\p{P}"; + } + if (tmp.control) { + regex += "\\p{C}"; + } + if (tmp = tmp.char) { + regex += "object" == typeof tmp ? tmp.join("") : tmp; + } + + this.split = new RegExp("[" + (include ? "^" : "") + regex + "]+", "u"); + this.numeric = numeric; + } else { + + /** + * split string input into terms + * @type {string|RegExp|boolean|null} + */ + this.split = /** @type {string|RegExp|boolean} */parse_option(tmp, whitespace, this.split); + this.numeric = parse_option(this.numeric, !0); + } + + /** + * post-processing terms + * @type {Function|null} + */ + this.prepare = /** @type {Function|null} */parse_option(options.prepare, null, this.prepare); + /** + * final processing + * @type {Function|null} + */ + this.finalize = /** @type {Function|null} */parse_option(options.finalize, null, this.finalize); + + // options + + this.rtl = options.rtl || /* suggest */ /* append: */ /* enrich */!1; + this.dedupe = parse_option(options.dedupe, !0, this.dedupe); + this.filter = parse_option((tmp = options.filter) && new Set(tmp), null, this.filter); + this.matcher = parse_option((tmp = options.matcher) && new Map(tmp), null, this.matcher); + this.mapper = parse_option((tmp = options.mapper) && new Map(tmp), null, this.mapper); + this.stemmer = parse_option((tmp = options.stemmer) && new Map(tmp), null, this.stemmer); + this.replacer = parse_option(options.replacer, null, this.replacer); + this.minlength = parse_option(options.minlength, 1, this.minlength); + this.maxlength = parse_option(options.maxlength, 0, this.maxlength); + + // minimum required tokenizer by this encoder + //this.tokenize = options["tokenize"] || ""; + + // auto-balanced cache + this.cache = tmp = parse_option(options.cache, !0, this.cache); + if (tmp) { + this.timer = null; + this.cache_size = "number" == typeof tmp ? tmp : 2e5; + this.cache_enc = new Map(); + this.cache_prt = new Map(); + this.cache_enc_length = 128; + this.cache_prt_length = 128; + } + + // regex temporary state + this.matcher_str = ""; + this.matcher_test = null; + this.stemmer_str = ""; + this.stemmer_test = null; + + // prebuilt + // if(this.filter && this.split){ + // for(const key of this.filter){ + // const tmp = key.replace(this.split, ""); + // if(key !== tmp){ + // this.filter.delete(key); + // this.filter.add(tmp); + // } + // } + // } + if (this.matcher) { + for (const key of this.matcher.keys()) { + this.matcher_str += (this.matcher_str ? "|" : "") + key; + } + } + if (this.stemmer) { + for (const key of this.stemmer.keys()) { + this.stemmer_str += (this.stemmer_str ? "|" : "") + key; + } + } + + // if(SUPPORT_COMPRESSION){ + // this.compression = parse_option(options.compress || options.compression, 0, this.compression); + // if(this.compression && !table){ + // table = new Array(radix); + // for(let i = 0; i < radix; i++) table[i] = i + 33; + // table = String.fromCharCode.apply(null, table); + // } + // } + + return this; +}; + +Encoder.prototype.addMatcher = function (match, replace) { + // regex: + if ("object" == typeof match) { + return this.addReplacer(match, replace); + } + // a single char: + if (2 > match.length) { + return this.addMapper(match, replace); + } + this.matcher || (this.matcher = new Map()); + this.matcher.set(match, replace); + this.matcher_str += (this.matcher_str ? "|" : "") + match; + this.matcher_test = null; //new RegExp("(" + this.matcher_str + ")"); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addStemmer = function (match, replace) { + this.stemmer || (this.stemmer = new Map()); + this.stemmer.set(match, replace); + this.stemmer_str += (this.stemmer_str ? "|" : "") + match; + this.stemmer_test = null; //new RegExp("(" + this.stemmer_str + ")"); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addFilter = function (str) { + this.filter || (this.filter = new Set()); + this.filter.add(str); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addMapper = function (char_match, char_replace) { + // regex: + if ("object" == typeof char_match) { + return this.addReplacer(char_match, char_replace); + } + // not a char: + if (1 < char_match.length) { + return this.addMatcher(char_match, char_replace); + } + this.mapper || (this.mapper = new Map()); + this.mapper.set(char_match, char_replace); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addReplacer = function (match, replace) { + if ("string" == typeof match) match = new RegExp(match, "g"); + this.replacer || (this.replacer = []); + this.replacer.push(match, replace || ""); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.invalidate = function () { + this.cache_enc.clear(); + this.cache_prt.clear(); +}; + + +Encoder.prototype.encode = function (str) { + + //if(!str) return str; + // todo remove dupe terms + + if (this.cache && str.length <= this.cache_enc_length) { + if (this.timer) { + if (this.cache_enc.has(str)) { + return this.cache_enc.get(str); + } + } else { + this.timer = setTimeout(clear, 0, this); + } + } + + // 1. apply charset normalization + if (this.normalize) { + if ("function" == typeof this.normalize) { + str = this.normalize(str); + } else if (normalize) { + str = str.normalize("NFKD").replace(normalize, "").toLowerCase(); + } else { + str = str.toLowerCase(); + + this.mapper = this.mapper + // todo replace spread + ? new Map([... /** @type {!Iterable} */normalize_mapper, ...this.mapper]) : new Map( /** @type {Map} */normalize_mapper); + } + //if(!str) return str; + } + + // 2. apply custom encoder (can replace split) + if (this.prepare) { + str = this.prepare(str); + } + + // 3. split numbers into triplets + if (this.numeric && 3 < str.length) { + str = str.replace(numeric_split_prev_char, "$1 $2").replace(numeric_split_next_char, "$1 $2").replace(numeric_split_length, "$1 "); + } + + // if(this.matcher && (str.length > 1)){ + // this.matcher_test || ( + // this.matcher_test = new RegExp("(" + this.matcher_str + ")", "g") + // ); + // str = str.replace(this.matcher_test, match => this.matcher.get(match)); + // } + // if(this.stemmer){ + // this.stemmer_test || ( + // this.stemmer_test = new RegExp("(?!\\b)(" + this.stemmer_str + ")(\\b|_)", "g") + // ); + // str = str.replace(this.stemmer_test, match => this.stemmer.get(match)); + // } + + const skip = !(this.dedupe || this.mapper || this.filter || this.matcher || this.stemmer || this.replacer); + let final = [], + words = this.split || "" === this.split ? str.split( /** @type {string|RegExp} */this.split) : str; + //[str]; + + for (let i = 0, word, base; i < words.length; i++) { + // filter empty entries + if (!(word = base = words[i])) { + continue; + } + if (word.length < this.minlength) { + continue; + } + if (skip) { + final.push(word); + continue; + } + + // 1. pre-filter before cache + if (this.filter && this.filter.has(word)) { + continue; + } + + if (this.cache && word.length <= this.cache_prt_length) { + if (this.timer) { + const tmp = this.cache_prt.get(word); + //if(this.cache_prt.has(word)){ + if (tmp || "" === tmp) { + //word = this.cache_prt.get(word); + tmp && final.push(tmp); + //word ? words[i] = word : words.splice(i--, 1); + continue; + } + } else { + this.timer = setTimeout(clear, 0, this); + } + } + + let postfilter; + + // if(this.normalize === true && normalize){ + // word = word.normalize("NFKD").replace(normalize, ""); + // postfilter = 1; + // } + + // if(this.normalize){ + // if(typeof this.normalize === "function"){ + // word = this.normalize(word); + // } + // else if(normalize){ + // word = word.normalize("NFKD").replace(normalize, "").toLowerCase(); + // } + // else{ + // word = word.toLowerCase(); + // this.mapper = this.mapper + // ? new Map([...normalize_mapper, ...this.mapper]) + // : new Map(/** @type {Map} */ normalize_mapper); + // } + // postfilter = 1; + // //if(!str) return str; + // } + + // 2. apply stemmer after matcher + if (this.stemmer && 2 < word.length) { + // for(const item of this.stemmer){ + // const key = item[0]; + // const value = item[1]; + // + // if(word.length > key.length && word.endsWith(key)){ + // word = word.substring(0, word.length - key.length) + value; + // break; + // } + // + // // const position = word.length - key.length; + // // if(position > 0 && word.substring(position) === key){ + // // word = word.substring(0, position) + value; + // // break; + // // } + // } + this.stemmer_test || (this.stemmer_test = new RegExp("(?!^)(" + this.stemmer_str + ")$")); + word = word.replace(this.stemmer_test, match => this.stemmer.get(match)); + postfilter = 1; + } + + // 3. apply matcher + if (this.matcher && 1 < word.length) { + this.matcher_test || (this.matcher_test = new RegExp("(" + this.matcher_str + ")", "g")); + word = word.replace(this.matcher_test, match => this.matcher.get(match)); + postfilter = 1; + } + + // 4. post-filter after matcher and stemmer was applied + if (word && postfilter && (word.length < this.minlength || this.filter && this.filter.has(word))) { + word = ""; + } + + // 5. apply mapper and collapsing + if (word && (this.mapper || this.dedupe && 1 < word.length)) { + //word = this.replace_dedupe(word); + //word = replace_deduped(word, this.mapper, true); + let final = ""; + for (let i = 0, prev = "", char, tmp; i < word.length; i++) { + char = word.charAt(i); + if (char !== prev || !this.dedupe) { + tmp = this.mapper && this.mapper.get(char); + if (!tmp && "" !== tmp) final += prev = char;else if ((tmp !== prev || !this.dedupe) && (prev = tmp)) final += tmp; + } + } + word = final; + } + + // apply custom regex + if (word && this.replacer) { + for (let i = 0; word && i < this.replacer.length; i += 2) { + word = word.replace(this.replacer[i], this.replacer[i + 1]); + } + } + + // slower variants for removing same chars in a row: + //word = word.replace(/([^0-9])\1+/g, "$1"); + //word = word.replace(/(.)\1+/g, "$1"); + //word = word.replace(/(?<=(.))\1+/g, ""); + + // if(word){ + // words[i] = word; + // } + + if (this.cache && base.length <= this.cache_prt_length) { + this.cache_prt.set(base, word); + if (this.cache_prt.size > this.cache_size) { + this.cache_prt.clear(); + this.cache_prt_length = 0 | this.cache_prt_length / 1.1; + } + } + + //word || words.splice(i--, 1); + word && final.push(word); + } + + //words = final; + // else if(this.filter){ + // for(let i = 0, word; i < words.length; i++){ + // if((word = words[i]) && !this.filter.has(word)){ + // //filtered.push(word); + // words.splice(i--, 1); + // } + // } + // } + + if (this.finalize) { + final = this.finalize(final) || final; + } + + if (this.cache && str.length <= this.cache_enc_length) { + this.cache_enc.set(str, final); + if (this.cache_enc.size > this.cache_size) { + this.cache_enc.clear(); + this.cache_enc_length = 0 | this.cache_enc_length / 1.1; + } + } + + return final; +}; + +// Encoder.prototype.compress = function(str) { +// +// //return str; +// //if(!str) return str; +// +// if(SUPPORT_CACHE && this.cache && str.length <= this.cache_prt_length){ +// if(this.timer){ +// if(this.cache_cmp.has(str)){ +// return this.cache_cmp.get(str); +// } +// } +// else{ +// this.timer = setTimeout(clear, 0, this); +// } +// } +// +// const result = typeof this.compression === "function" +// ? this.compression(str) +// : hash(str); //window.hash(str); +// +// if(SUPPORT_CACHE && this.cache && str.length <= this.cache_prt_length){ +// this.cache_cmp.set(str, result); +// this.cache_cmp.size > this.cache_size && +// this.cache_cmp.clear(); +// } +// +// return result; +// }; + +// function hash(str){ +// return str; +// } + +function clear(self) { + self.timer = null; + self.cache_enc.clear(); + self.cache_prt.clear(); +} \ No newline at end of file diff --git a/dist/module-debug/engine.js b/dist/module-debug/engine.js deleted file mode 100644 index d460eda..0000000 --- a/dist/module-debug/engine.js +++ /dev/null @@ -1,35 +0,0 @@ - -import { searchCache } from "./cache"; - -/** - * @constructor - * @abstract - */ - -function Engine(index) { - - //if(this.constructor === Engine){ - if (this instanceof Engine) { - - throw new Error("Can't instantiate abstract class!"); - } - - - index.prototype.searchCache = searchCache; - - - index.prototype.addAsync = addAsync; - index.prototype.appendAsync = appendAsync; - index.prototype.searchAsync = searchAsync; - index.prototype.updateAsync = updateAsync; - index.prototype.removeAsync = removeAsync; -} - -Engine.prototype.searchCache = searchCache; - - -Engine.prototype.addAsync = addAsync; -Engine.prototype.appendAsync = appendAsync; -Engine.prototype.searchAsync = searchAsync; -Engine.prototype.updateAsync = updateAsync; -Engine.prototype.removeAsync = removeAsync; \ No newline at end of file diff --git a/dist/module-debug/global.js b/dist/module-debug/global.js index bd7da48..fc9a5ce 100644 --- a/dist/module-debug/global.js +++ b/dist/module-debug/global.js @@ -1,5 +1,7 @@ -export const global_lang = {}; -export const global_charset = {}; +import { create_object } from "./common.js"; + +export const global_lang = create_object(); +export const global_charset = create_object(); /** * @param {!string} name @@ -7,7 +9,6 @@ export const global_charset = {}; */ export function registerCharset(name, charset) { - global_charset[name] = charset; } @@ -17,6 +18,5 @@ export function registerCharset(name, charset) { */ export function registerLanguage(name, lang) { - global_lang[name] = lang; } \ No newline at end of file diff --git a/dist/module-debug/index.js b/dist/module-debug/index.js index d6f8703..8f18dda 100644 --- a/dist/module-debug/index.js +++ b/dist/module-debug/index.js @@ -6,90 +6,152 @@ * https://github.com/nextapps-de/flexsearch */ -import { IndexInterface } from "./type.js"; -import { encode as default_encoder } from "./lang/latin/default.js"; -import { create_object, create_object_array, concat, sort_by_length_down, is_array, is_string, is_object, parse_option } from "./common.js"; -import { pipeline, init_stemmer_or_matcher, init_filter } from "./lang.js"; -import { global_lang, global_charset } from "./global.js"; -import apply_async from "./async.js"; -import { intersect } from "./intersect.js"; +import { IndexOptions } from "./type.js"; +import Encoder from "./encoder.js"; import Cache, { searchCache } from "./cache.js"; -import apply_preset from "./preset.js"; +import { KeystoreMap, KeystoreSet } from "./keystore.js"; +import { is_array, is_string } from "./common.js"; import { exportIndex, importIndex } from "./serialize.js"; +import { global_lang, global_charset } from "./global.js"; +import default_encoder from "./lang/latin/default.js"; +import apply_preset from "./preset.js"; +import apply_async from "./async.js"; +import tick from "./profiler.js"; +import "./index/add.js"; +import "./index/search.js"; +import "./index/remove.js"; /** * @constructor - * @implements IndexInterface - * @param {Object=} options - * @param {Object=} _register - * @return {Index} + * @param {IndexOptions|string=} options + * @param {Map|Set|KeystoreSet|KeystoreMap=} _register */ -function Index(options, _register) { +export default function Index(options, _register) { if (!(this instanceof Index)) { - return new Index(options); } - let charset, lang, tmp; - if (options) { options = apply_preset(options); + // charset = options.charset; + // // lang = options.lang; + // + // if(is_string(charset)){ + // + // if(!charset.includes(":")){ + // charset += ":default"; + // } + // + // charset = global_charset[charset]; + // } - - charset = options.charset; - lang = options.lang; - - if (is_string(charset)) { - - if (-1 === charset.indexOf(":")) { - - charset += ":default"; - } - - charset = global_charset[charset]; - } - - if (is_string(lang)) { - - lang = global_lang[lang]; - } + // if(is_string(lang)){ + // + // lang = global_lang[lang]; + // } } else { - options = {}; } - let resolution, - optimize, - context = options.context || {}; + // let charset, lang, tmp; - this.encode = options.encode || charset && charset.encode || default_encoder; - this.register = _register || create_object(); - this.resolution = resolution = options.resolution || 9; - this.tokenize = tmp = charset && charset.tokenize || options.tokenize || "strict"; - this.depth = "strict" === tmp && context.depth; - this.bidirectional = parse_option(context.bidirectional, /* append: */ /* skip update: */ /* skip_update: */!0); - this.optimize = optimize = parse_option(options.optimize, !0); - this.fastupdate = parse_option(options.fastupdate, !0); - this.minlength = options.minlength || 1; - this.boost = options.boost; + const context = options.context || {}, + encoder = options.encode || options.encoder || default_encoder; - // when not using the memory strategy the score array should not pre-allocated to its full length + this.encoder = encoder.encode ? encoder : "object" == typeof encoder ? new Encoder(encoder) : { encode: encoder }; - this.map = optimize ? create_object_array(resolution) : create_object(); - this.resolution_ctx = resolution = context.resolution || 1; - this.ctx = optimize ? create_object_array(resolution) : create_object(); - this.rtl = charset && charset.rtl || options.rtl; - this.matcher = (tmp = options.matcher || lang && lang.matcher) && init_stemmer_or_matcher(tmp, !1); - this.stemmer = (tmp = options.stemmer || lang && lang.stemmer) && init_stemmer_or_matcher(tmp, !0); - this.filter = (tmp = options.filter || lang && lang.filter) && init_filter(tmp); + this.compress = options.compress || options.compression || /* suggest */ /* append: */ /* enrich */!1; - this.cache = (tmp = options.cache) && new Cache(tmp); + + let tmp; + this.resolution = options.resolution || 9; + this.tokenize = tmp = options.tokenize || "strict"; + this.depth = "strict" === tmp && context.depth || 0; + this.bidirectional = !1 !== context.bidirectional; + this.fastupdate = !!options.fastupdate; + this.score = options.score || null; + + tmp = options.keystore || 0; + tmp && (this.keystore = tmp); + + this.map = tmp ? new KeystoreMap(tmp) : new Map(); + this.ctx = tmp ? new KeystoreMap(tmp) : new Map(); + this.reg = _register || (this.fastupdate ? tmp ? new KeystoreMap(tmp) : new Map() : tmp ? new KeystoreSet(tmp) : new Set()); + this.resolution_ctx = context.resolution || 1; + this.rtl = encoder.rtl || options.rtl || !1; + + this.cache = (tmp = options.cache || null) && new Cache(tmp); + + this.resolve = !1 !== options.resolve; + + if (tmp = options.db) { + this.db = tmp.mount(this); + } + this.commit_auto = !1 !== options.commit; + this.commit_task = []; + this.commit_timer = null; } -export default Index; +Index.prototype.mount = function (db) { + if (this.commit_timer) { + clearTimeout(this.commit_timer); + this.commit_timer = null; + } + return db.mount(this); +}; +Index.prototype.commit = function (replace, append) { + if (this.commit_timer) { + clearTimeout(this.commit_timer); + this.commit_timer = null; + } + return this.db.commit(this, replace, append); +}; + +// if(SUPPORT_RESOLVER){ +// Index.prototype.resolve = function(params){ +// return new Resolver(params); +// }; +// } + +/** + * @param {!Index} self + * @param {boolean=} replace + * @param {boolean=} append + */ + +export function autoCommit(self, replace, append) { + if (!self.commit_timer) { + self.commit_timer = setTimeout(function () { + self.commit_timer = null; + self.db.commit(self, replace, append); + }, 0); + } +} + +Index.prototype.clear = function () { + + //this.map = new Map(); + //this.ctx = new Map(); + //this.reg = this.fastupdate ? new Map() : new Set(); + this.map.clear(); + this.ctx.clear(); + this.reg.clear(); + + this.cache && this.cache.clear(); + + + if (this.db) { + this.commit_timer && clearTimeout(this.commit_timer); + this.commit_timer = null; + this.commit_task = [{ clear: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ }]; + //return this.db.clear(); + } + + return this; +}; //Index.prototype.pipeline = pipeline; @@ -99,681 +161,68 @@ export default Index; */ Index.prototype.append = function (id, content) { - return this.add(id, content, !0); }; -// TODO: -// string + number as text -// boolean, null, undefined as ? - -/** - * @param {!number|string} id - * @param {!string} content - * @param {boolean=} _append - * @param {boolean=} _skip_update - */ - -Index.prototype.add = function (id, content, _append, _skip_update) { - - if (content && (id || 0 === id)) { - - if (!_skip_update && !_append && this.register[id]) { - - return this.update(id, content); - } - - content = this.encode("" + content); - const length = content.length; - - if (length) { - - // check context dupes to skip all contextual redundancy along a document - - const dupes_ctx = create_object(), - dupes = create_object(), - depth = this.depth, - resolution = this.resolution; - - - for (let i = 0; i < length; i++) { - let term = content[this.rtl ? length - 1 - i : i], - term_length = term.length; - - - // skip dupes will break the context chain - - if (term && term_length >= this.minlength && (depth || !dupes[term])) { - let score = get_score(resolution, length, i), - token = ""; - - - switch (this.tokenize) { - - case "full": - - if (2 < term_length) { - - for (let x = 0; x < term_length; x++) { - - for (let y = term_length; y > x; y--) { - - if (y - x >= this.minlength) { - - const partial_score = get_score(resolution, length, i, term_length, x); - token = term.substring(x, y); - this.push_index(dupes, token, partial_score, id, _append); - } - } - } - - break; - } - - // fallthrough to next case when term length < 3 - - case "reverse": - - // skip last round (this token exist already in "forward") - - if (1 < term_length) { - - for (let x = term_length - 1; 0 < x; x--) { - - token = term[x] + token; - - if (token.length >= this.minlength) { - - const partial_score = get_score(resolution, length, i, term_length, x); - this.push_index(dupes, token, partial_score, id, _append); - } - } - - token = ""; - } - - // fallthrough to next case to apply forward also - - case "forward": - - if (1 < term_length) { - - for (let x = 0; x < term_length; x++) { - - token += term[x]; - - if (token.length >= this.minlength) { - - this.push_index(dupes, token, score, id, _append); - } - } - - break; - } - - // fallthrough to next case when token has a length of 1 - - default: - // case "strict": - - if (this.boost) { - - score = Math.min(0 | score / this.boost(content, term, i), resolution - 1); - } - - this.push_index(dupes, term, score, id, _append); - - // context is just supported by tokenizer "strict" - - if (depth) { - - if (1 < length && i < length - 1) { - - // check inner dupes to skip repeating words in the current context - - const dupes_inner = create_object(), - resolution = this.resolution_ctx, - keyword = term, - size = Math.min(depth + 1, length - i); - - - dupes_inner[keyword] = 1; - - for (let x = 1; x < size; x++) { - - term = content[this.rtl ? length - 1 - i - x : i + x]; - - if (term && term.length >= this.minlength && !dupes_inner[term]) { - - dupes_inner[term] = 1; - - const context_score = get_score(resolution + (length / 2 > resolution ? 0 : 1), length, i, size - 1, x - 1), - swap = this.bidirectional && term > keyword; - - this.push_index(dupes_ctx, swap ? keyword : term, context_score, id, _append, swap ? term : keyword); - } - } - } - } - } - } - } - - this.fastupdate || (this.register[id] = 1); - } - } - - return this; -}; - -/** - * @param {number} resolution - * @param {number} length - * @param {number} i - * @param {number=} term_length - * @param {number=} x - * @returns {number} - */ - -function get_score(resolution, length, i, term_length, x) { - - // console.log("resolution", resolution); - // console.log("length", length); - // console.log("term_length", term_length); - // console.log("i", i); - // console.log("x", x); - // console.log((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1); - - // the first resolution slot is reserved for the best match, - // when a query matches the first word(s). - - // also to stretch score to the whole range of resolution, the - // calculation is shift by one and cut the floating point. - // this needs the resolution "1" to be handled additionally. - - // do not stretch the resolution more than the term length will - // improve performance and memory, also it improves scoring in - // most cases between a short document and a long document - - return i && 1 < resolution ? length + (term_length || 0) <= resolution ? i + (x || 0) : 0 | (resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1 : 0; -} - -/** - * @private - * @param dupes - * @param value - * @param score - * @param id - * @param {boolean=} append - * @param {string=} keyword - */ - -Index.prototype.push_index = function (dupes, value, score, id, append, keyword) { - - let arr = keyword ? this.ctx : this.map; - - if (!dupes[value] || keyword && !dupes[value][keyword]) { - - if (this.optimize) { - - arr = arr[score]; - } - - if (keyword) { - - dupes = dupes[value] || (dupes[value] = create_object()); - dupes[keyword] = 1; - - arr = arr[keyword] || (arr[keyword] = create_object()); - } else { - - dupes[value] = 1; - } - - arr = arr[value] || (arr[value] = []); - - if (!this.optimize) { - - arr = arr[score] || (arr[score] = []); - } - - if (!append || !arr.includes(id)) { - - arr[arr.length] = id; - - // add a reference to the register for fast updates - - if (this.fastupdate) { - - const tmp = this.register[id] || (this.register[id] = []); - tmp[tmp.length] = arr; - } - } - } -}; - -/** - * @param {string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @returns {Array} - */ - -Index.prototype.search = function (query, limit, options) { - - if (!options) { - - if (!limit && is_object(query)) { - - options = /** @type {Object} */query; - query = options.query; - } else if (is_object(limit)) { - - options = /** @type {Object} */limit; - } - } - - let result = [], - length, - context, - suggest, - offset = 0; - - - if (options) { - - query = options.query || query; - limit = options.limit; - offset = options.offset || 0; - context = options.context; - suggest = options.suggest; - } - - if (query) { - - query = /** @type {Array} */this.encode("" + query); - length = query.length; - - // TODO: solve this in one single loop below - - if (1 < length) { - const dupes = create_object(), - query_new = []; - - - for (let i = 0, count = 0, term; i < length; i++) { - - term = query[i]; - - if (term && term.length >= this.minlength && !dupes[term]) { - - // this fast path can just apply when not in memory-optimized mode - - if (!this.optimize && !suggest && !this.map[term]) { - - // fast path "not found" - - return result; - } else { - - query_new[count++] = term; - dupes[term] = 1; - } - } - } - - query = query_new; - length = query.length; - } - } - - if (!length) { - - return result; - } - - limit || (limit = 100); - - let depth = this.depth && 1 < length && !1 !== context, - index = 0, - keyword; - - - if (depth) { - - keyword = query[0]; - index = 1; - } else { - - if (1 < length) { - - query.sort(sort_by_length_down); - } - } - - for (let arr, term; index < length; index++) { - - term = query[index]; - - // console.log(keyword); - // console.log(term); - // console.log(""); - - if (depth) { - - arr = this.add_result(result, suggest, limit, offset, 2 === length, term, keyword); - - // console.log(arr); - // console.log(result); - - // when suggestion enabled just forward keyword if term was found - // as long as the result is empty forward the pointer also - - if (!suggest || !1 !== arr || !result.length) { - - keyword = term; - } - } else { - - arr = this.add_result(result, suggest, limit, offset, 1 === length, term); - } - - if (arr) { - - return (/** @type {Array} */arr - ); - } - - // apply suggestions on last loop or fallback - - if (suggest && index == length - 1) { - - let length = result.length; - - if (!length) { - - if (depth) { - - // fallback to non-contextual search when no result was found - - depth = 0; - index = -1; - - continue; - } - - return result; - } else if (1 === length) { - - // fast path optimization - - return single_result(result[0], limit, offset); - } - } - } - - return intersect(result, limit, offset, suggest); -}; - -/** - * Returns an array when the result is done (to stop the process immediately), - * returns false when suggestions is enabled and no result was found, - * or returns nothing when a set was pushed successfully to the results - * - * @private - * @param {Array} result - * @param {Array} suggest - * @param {number} limit - * @param {number} offset - * @param {boolean} single_term - * @param {string} term - * @param {string=} keyword - * @return {Array>|boolean|undefined} - */ - -Index.prototype.add_result = function (result, suggest, limit, offset, single_term, term, keyword) { - let word_arr = [], - arr = keyword ? this.ctx : this.map; - - - if (!this.optimize) { - - arr = get_array(arr, term, keyword, this.bidirectional); - } - - if (arr) { - - let count = 0; - const arr_len = Math.min(arr.length, keyword ? this.resolution_ctx : this.resolution); - - // relevance: - for (let x = 0, size = 0, tmp, len; x < arr_len; x++) { - - tmp = arr[x]; - - if (tmp) { - - if (this.optimize) { - - tmp = get_array(tmp, term, keyword, this.bidirectional); - } - - if (offset) { - - if (tmp && single_term) { - - len = tmp.length; - - if (len <= offset) { - - offset -= len; - tmp = null; - } else { - - tmp = tmp.slice(offset); - offset = 0; - } - } - } - - if (tmp) { - - // keep score (sparse array): - //word_arr[x] = tmp; - - // simplified score order: - word_arr[count++] = tmp; - - if (single_term) { - - size += tmp.length; - - if (size >= limit) { - - // fast path optimization - - break; - } - } - } - } - } - - if (count) { - - if (single_term) { - - // fast path optimization - // offset was already applied at this point - - return single_result(word_arr, limit, 0); - } - - result[result.length] = word_arr; - return; - } - } - - // return an empty array will stop the loop, - // to prevent stop when using suggestions return a false value - - return !suggest && word_arr; -}; - -function single_result(result, limit, offset) { - - if (1 === result.length) { - - result = result[0]; - } else { - - result = concat(result); - } - - return offset || result.length > limit ? result.slice(offset, offset + limit) : result; -} - -function get_array(arr, term, keyword, bidirectional) { - - if (keyword) { - - // the frequency of the starting letter is slightly less - // on the last half of the alphabet (m-z) in almost every latin language, - // so we sort downwards (https://en.wikipedia.org/wiki/Letter_frequency) - - const swap = bidirectional && term > keyword; - - arr = arr[swap ? term : keyword]; - arr = arr && arr[swap ? keyword : term]; - } else { - - arr = arr[term]; - } - - return arr; -} - Index.prototype.contain = function (id) { - return !!this.register[id]; + if (this.db) { + return this.db.has(id); + } + + return this.reg.has(id); }; Index.prototype.update = function (id, content) { + // todo check the async part + if (this.async /*|| (SUPPORT_PERSISTENT && this.db)*/) { + const self = this, + res = this.remove(id); + + return res.then ? res.then(() => self.add(id, content)) : this.add(id, content); + } + return this.remove(id).add(id, content); }; -/** - * @param {boolean=} _skip_deletion - */ - -Index.prototype.remove = function (id, _skip_deletion) { - - const refs = this.register[id]; - - if (refs) { - - if (this.fastupdate) { - - // fast updates performs really fast but did not fully cleanup the key entries - - for (let i = 0, tmp; i < refs.length; i++) { - - tmp = refs[i]; - tmp.splice(tmp.indexOf(id), 1); - } - } else { - - remove_index(this.map, id, this.resolution, this.optimize); - - if (this.depth) { - - remove_index(this.ctx, id, this.resolution_ctx, this.optimize); - } - } - - _skip_deletion || delete this.register[id]; - - if (this.cache) { - - this.cache.del(id); - } - } - - return this; -}; - /** * @param map - * @param id - * @param res - * @param optimize - * @param {number=} resolution * @return {number} */ -function remove_index(map, id, res, optimize, resolution) { +function cleanup_index(map) { let count = 0; if (is_array(map)) { - - // the first array is the score array in both strategies - - if (!resolution) { - - resolution = Math.min(map.length, res); - - for (let x = 0, arr; x < resolution; x++) { - - arr = map[x]; - - if (arr) { - - count = remove_index(arr, id, res, optimize, resolution); - - if (!optimize && !count) { - - // when not memory optimized the score index should removed - - delete map[x]; - } - } - } - } else { - - const pos = map.indexOf(id); - - if (-1 !== pos) { - - // fast path, when length is 1 or lower then the whole field gets deleted - - if (1 < map.length) { - - map.splice(pos, 1); - count++; - } - } else { - - count++; - } + for (let i = 0, arr; i < map.length; i++) { + (arr = map[i]) && (count += arr.length); } - } else { + } else for (const item of map) { + const key = item[0], + value = item[1], + tmp = cleanup_index(value); - for (let key in map) { - - count = remove_index(map[key], id, res, optimize, resolution); - - if (!count) { - - delete map[key]; - } - } + tmp ? count += tmp : map.delete(key); } return count; } +Index.prototype.cleanup = function () { + + if (!this.fastupdate) { + console.info("Cleanup the index isn't required when not using \"fastupdate\"."); + return this; + } + + cleanup_index(this.map); + this.depth && cleanup_index(this.ctx); + + return this; +}; + Index.prototype.searchCache = searchCache; diff --git a/dist/module-debug/index/add.js b/dist/module-debug/index/add.js new file mode 100644 index 0000000..0f47234 --- /dev/null +++ b/dist/module-debug/index/add.js @@ -0,0 +1,261 @@ + +import { create_object } from "../common.js"; +import Index, { autoCommit } from "../index.js"; +import default_compress from "../compress.js"; +import { KeystoreArray } from "../keystore.js"; + +// TODO: +// string + number as text +// boolean, null, undefined as ? + + +/** + * @param {!number|string} id + * @param {!string} content + * @param {boolean=} _append + * @param {boolean=} _skip_update + */ + +Index.prototype.add = function (id, content, _append, _skip_update) { + + if (content && (id || 0 === id)) { + + // todo check skip_update + //_skip_update = false; + + if (!_skip_update && !_append) { + if (this.reg.has(id)) { + return this.update(id, content); + } + } + + // do not force a string as input + // https://github.com/nextapps-de/flexsearch/issues/432 + content = this.encoder.encode(content); + const word_length = content.length; + + if (word_length) { + + // check context dupes to skip all contextual redundancy along a document + + const dupes_ctx = create_object(), + dupes = create_object(), + depth = this.depth, + resolution = this.resolution; + + + for (let i = 0; i < word_length; i++) { + let term = content[this.rtl ? word_length - 1 - i : i], + term_length = term.length; + + + // skip dupes will break the context chain + + if (term_length /*&& (term_length >= this.minlength)*/ && (depth || !dupes[term])) { + let score = this.score ? this.score(content, term, i, null, 0) : get_score(resolution, word_length, i), + token = ""; + + + switch (this.tokenize) { + + case "full": + if (2 < term_length) { + for (let x = 0; x < term_length; x++) { + for (let y = term_length; y > x; y--) { + + //if((y - x) >= this.minlength){ + token = term.substring(x, y); + const partial_score = this.score ? this.score(content, term, i, token, x) : get_score(resolution, word_length, i, term_length, x); + this.push_index(dupes, token, partial_score, id, _append); + //} + } + } + break; + } + // fallthrough to next case when term length < 3 + case "reverse": + // skip last round (this token exist already in "forward") + if (1 < term_length) { + for (let x = term_length - 1; 0 < x; x--) { + token = term[x] + token; + //if(token.length >= this.minlength){ + const partial_score = this.score ? this.score(content, term, i, token, x) : get_score(resolution, word_length, i, term_length, x); + this.push_index(dupes, token, partial_score, id, _append); + //} + } + token = ""; + } + + // fallthrough to next case to apply forward also + case "forward": + if (1 < term_length) { + for (let x = 0; x < term_length; x++) { + token += term[x]; + //if(token.length >= this.minlength){ + this.push_index(dupes, token, score, id, _append); + //} + } + break; + } + + // fallthrough to next case when token has a length of 1 + default: + // case "strict": + + // todo move boost to search + // if(this.boost){ + // score = Math.min((score / this.boost(content, term, i)) | 0, resolution - 1); + // } + + this.push_index(dupes, term, score, id, _append); + + // context is just supported by tokenizer "strict" + if (depth) { + + if (1 < word_length && i < word_length - 1) { + + // check inner dupes to skip repeating words in the current context + const dupes_inner = create_object(), + resolution = this.resolution_ctx, + keyword = term, + size = Math.min(depth + 1, word_length - i); + + + dupes_inner[keyword] = 1; + + for (let x = 1; x < size; x++) { + + term = content[this.rtl ? word_length - 1 - i - x : i + x]; + + if (term /*&& (term.length >= this.minlength)*/ && !dupes_inner[term]) { + + dupes_inner[term] = 1; + + const context_score = this.score ? this.score(content, keyword, i, term, x) : get_score(resolution + (word_length / 2 > resolution ? 0 : 1), word_length, i, size - 1, x - 1), + swap = this.bidirectional && term > keyword; + + this.push_index(dupes_ctx, swap ? keyword : term, context_score, id, _append, swap ? term : keyword); + } + } + } + } + } + } + } + + this.fastupdate || this.reg.add(id); + } else { + content = ""; + } + } + + if (this.db) { + // when the term has no valid content (e.g. empty), + // then it was not added to the ID registry for removal + content || this.commit_task.push({ del: id }); + this.commit_auto && autoCommit(this); + } + + return this; +}; + +/** + * @private + * @param dupes + * @param term + * @param score + * @param id + * @param {boolean=} append + * @param {string=} keyword + */ + +Index.prototype.push_index = function (dupes, term, score, id, append, keyword) { + let arr = keyword ? this.ctx : this.map, + tmp; + + + if (!dupes[term] || !keyword || !(tmp = dupes[term])[keyword]) { + + if (keyword) { + + dupes = tmp || (dupes[term] = create_object()); + dupes[keyword] = 1; + + if (this.compress) { + keyword = default_compress(keyword); + } + + tmp = arr.get(keyword); + tmp ? arr = tmp : arr.set(keyword, arr = new Map()); + } else { + + dupes[term] = 1; + } + + if (this.compress) { + term = default_compress(term); + } + + tmp = arr.get(term); + tmp ? arr = tmp : arr.set(term, arr = tmp = []); + // the ID array will be upgraded dynamically + arr = arr[score] || (arr[score] = []); + + if (!append || !arr.includes(id)) { + + // auto-upgrade to keystore array if max size exceeded + if (2147483647 === arr.length /*|| !(arr instanceof KeystoreArray)*/) { + const keystore = new KeystoreArray(arr); + if (this.fastupdate) { + for (let value of this.reg.values()) { + if (value.includes(arr)) { + value[value.indexOf(arr)] = keystore; + } + } + } + tmp[score] = arr = keystore; + } + + + arr.push(id); + + // add a reference to the register for fast updates + if (this.fastupdate) { + const tmp = this.reg.get(id); + tmp ? tmp.push(arr) : this.reg.set(id, [arr]); + } + } + } +}; + +/** + * @param {number} resolution + * @param {number} length + * @param {number} i + * @param {number=} term_length + * @param {number=} x + * @returns {number} + */ + +function get_score(resolution, length, i, term_length, x) { + + // console.log("resolution", resolution); + // console.log("length", length); + // console.log("term_length", term_length); + // console.log("i", i); + // console.log("x", x); + // console.log((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1); + + // the first resolution slot is reserved for the best match, + // when a query matches the first word(s). + + // also to stretch score to the whole range of resolution, the + // calculation is shift by one and cut the floating point. + // this needs the resolution "1" to be handled additionally. + + // do not stretch the resolution more than the term length will + // improve performance and memory, also it improves scoring in + // most cases between a short document and a long document + + return i && 1 < resolution ? length + (term_length || 0) <= resolution ? i + (x || 0) : 0 | (resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1 : 0; +} \ No newline at end of file diff --git a/dist/module-debug/index/remove.js b/dist/module-debug/index/remove.js new file mode 100644 index 0000000..b79ad2c --- /dev/null +++ b/dist/module-debug/index/remove.js @@ -0,0 +1,137 @@ + +import { create_object, is_array } from "../common.js"; +import Index, { autoCommit } from "../index.js"; +import default_compress from "../compress.js"; + +/** + * @param {boolean=} _skip_deletion + */ + +Index.prototype.remove = function (id, _skip_deletion) { + + const refs = this.reg.size && (this.fastupdate ? this.reg.get(id) : this.reg.has(id)); + + if (refs) { + + if (this.fastupdate) { + + // fast updates did not fully cleanup the key entries + + for (let i = 0, tmp; i < refs.length; i++) { + if (tmp = refs[i]) { + // todo check + //if(tmp.length < 1) throw new Error("invalid length"); + //if(tmp.indexOf(id) < 0) throw new Error("invalid id"); + if (2 > tmp.length) { + tmp.pop(); + } else { + const index = tmp.indexOf(id); + index === refs.length - 1 ? tmp.pop() : tmp.splice(index, 1); + } + } + } + + // todo variation which cleans up, requires to push [ctx, key] instead of arr to the index.reg + // for(let i = 0, arr, term, keyword; i < refs.length; i++){ + // arr = refs[i]; + // if(typeof arr === "string"){ + // arr = this.map.get(term = arr); + // } + // else{ + // arr = this.ctx.get(keyword = arr[0]); + // arr && (arr = arr.get(arr[1])); + // } + // let counter = 0, found; + // if(arr && arr.length){ + // for(let j = 0, tmp; j < arr.length; j++){ + // if((tmp = arr[j])){ + // if(!found && tmp.length){ + // const index = tmp.indexOf(id); + // if(index >= 0){ + // tmp.splice(index, 1); + // // the index [ctx, key]:[res, id] is unique + // found = 1; + // } + // } + // if(tmp.length){ + // counter++; + // if(found){ + // break; + // } + // } + // else{ + // delete arr[j]; + // } + // } + // } + // } + // if(!counter){ + // keyword + // ? this.ctx.delete(keyword) + // : this.map.delete(term); + // } + // } + } else { + + remove_index(this.map, id /*, this.resolution*/); + this.depth && remove_index(this.ctx, id /*, this.resolution_ctx*/); + } + + _skip_deletion || this.reg.delete(id); + } + + if (this.db) { + this.commit_task.push({ del: id }); + this.commit_auto && autoCommit(this); + //return this.db.remove(id); + } + + // the cache could be used outside the InMemory store + if (this.cache) { + this.cache.remove(id); + } + + return this; +}; + +/** + * @param map + * @param id + * @return {number} + */ + +function remove_index(map, id) { + + // a check counter of filled resolution slots + // to prevent removing the field + let count = 0; + + if (is_array(map)) { + for (let x = 0, arr, index; x < map.length; x++) { + if ((arr = map[x]) && arr.length) { + index = arr.indexOf(id); + if (0 <= index) { + if (1 < arr.length) { + arr.splice(index, 1); + count++; + } else { + // remove resolution slot + delete map[x]; + } + // the index key:[res, id] is unique + break; + } else { + count++; + } + } + } + } else for (let item of map) { + const key = item[0], + value = item[1], + tmp = remove_index(value, id); + + tmp ? count += tmp : map.delete(key); + } + + return count; +} \ No newline at end of file diff --git a/dist/module-debug/index/search.js b/dist/module-debug/index/search.js new file mode 100644 index 0000000..44b4cb1 --- /dev/null +++ b/dist/module-debug/index/search.js @@ -0,0 +1,432 @@ + + +import { SearchOptions } from "../type.js"; +import { create_object, is_object, sort_by_length_down } from "../common.js"; +import Index from "../index.js"; +import default_compress from "../compress.js"; +import Resolver from "../resolver.js"; +import { intersect } from "../intersect.js"; +import resolve_default from "../resolve/default.js"; + +let global_resolve = 1; +export function set_resolve(resolve) { + global_resolve = resolve; +} + +/** + * @param {string|SearchOptions} query + * @param {number|SearchOptions=} limit + * @param {SearchOptions=} options + * @returns {Array|Resolver|Promise} + */ + +Index.prototype.search = function (query, limit, options) { + + if (!options) { + if (!limit && is_object(query)) { + options = /** @type {!SearchOptions} */query; + query = ""; + } else if (is_object(limit)) { + options = /** @type {!SearchOptions} */limit; + limit = 0; + } + } + + let result = [], + length, + context, + suggest, + offset = 0, + resolve, + enrich, + tag; + + + if (options) { + query = options.query || query; + limit = options.limit || limit; + offset = options.offset || 0; + context = options.context; + suggest = options.suggest; + resolve = global_resolve && + /* suggest */ /* append: */ /* enrich */!1 !== options.resolve; + resolve || (global_resolve = 0); + enrich = resolve && options.enrich; + tag = this.db && options.tag; + } else { + resolve = this.resolve || global_resolve; + } + + // todo: term deduplication during encoding when context is disabled + + // do not force a string as input + // https://github.com/nextapps-de/flexsearch/issues/432 + query = /** @type {Array} */this.encoder.encode(query); + length = query.length; + limit || !resolve || (limit = 100); + + // fast path single term + if (1 === length) { + return single_term_query.call(this, query[0], // term + "", // ctx + limit, offset, resolve, enrich, tag); + } + + // TODO: dedupe terms within encoder? + // TODO: deduplication will break the context chain + + context = this.depth && !1 !== context; + + // fast path single context + if (2 === length && context && !suggest) { + return single_term_query.call(this, query[0], // term + query[1], // ctx + limit, offset, resolve, enrich, tag); + } + + let maxlength = 0, + minlength = 0; + + + if (1 < length) { + + // term deduplication will break the context chain + // todo add context to dupe check + const dupes = create_object(), + query_new = []; + + + // if(context){ + // keyword = query[0]; + // dupes[keyword] = 1; + // query_new.push(keyword); + // maxlength = minlength = keyword.length; + // i = 1; + // } + + for (let i = 0, term; i < length; i++) { + + term = query[i]; + + if (term && !dupes[term]) { + + // todo add keyword check + // this fast path can't apply to persistent indexes + if (!suggest && !this.db && !this.get_array(term /*, keyword*/)) { + + // fast path "not found" + return resolve ? result : new Resolver(result); + } else { + + query_new.push(term); + dupes[term] = 1; + } + + const term_length = term.length; + maxlength = Math.max(maxlength, term_length); + minlength = minlength ? Math.min(minlength, term_length) : term_length; + } + // else if(term && (!this.depth || context === false)){ + // query_new.push(term); + // } + } + + query = query_new; + length = query.length; + } + + // the term length could be changed after deduplication + + if (!length) { + return resolve ? result : new Resolver(result); + } + + let index = 0, + keyword; + + // fast path single term + if (1 === length) { + return single_term_query.call(this, query[0], // term + "", // ctx + limit, offset, resolve, enrich, tag); + } + + // fast path single context + if (2 === length && context && !suggest) { + return single_term_query.call(this, query[0], // term + query[1], // ctx + limit, offset, resolve, enrich, tag); + } + + if (1 < length) { + if (context) { + // start with context right away + keyword = query[0]; + index = 1; + } + // todo + else if (9 < maxlength && 3 < maxlength / minlength) { + // sorting terms will break the context chain + // bigger terms has less occurrence + // this might also reduce the intersection task + // todo check intersection order + query.sort(sort_by_length_down); + } + } + + if (this.db) { + + if (this.db.search) { + // when the configuration is not supported it returns false + const result = this.db.search(this, query, limit, offset, suggest, resolve, enrich, tag); + if (!1 !== result) return result; + } + + const self = this; + return async function () { + + for (let arr, term; index < length; index++) { + + term = query[index]; + + if (keyword) { + + arr = await self.get_array(term, keyword); + arr = add_result(arr, result, suggest, self.resolution_ctx, + /** @type {!number} */limit, offset, 2 === length + /*, term, keyword*/ + ); + + // the context is a moving window where the keyword is going forward like a cursor + // 1. when suggestion enabled just forward keyword if term was found + // 2. as long as the result is empty forward the pointer also + if (!suggest || !1 !== arr || !result.length) { + keyword = term; + } + } else { + + arr = await self.get_array(term); + arr = add_result(arr, result, suggest, self.resolution, + /** @type {!number} */limit, offset, 1 === length + /*, term*/ + ); + } + + // limit reached + if (arr) { + return arr; + } + + // apply suggestions on last loop + if (suggest && index == length - 1) { + let length = result.length; + if (!length) { + // fallback to non-contextual search when no result was found + if (keyword) { + keyword = ""; + index = -1; + continue; + } + return result; + } else if (1 === length) { + return resolve ? resolve_default(result[0], /** @type {number} */limit, offset) : new Resolver(result[0]); + } + } + } + + return resolve ? intersect(result, /** @type {number} */limit, offset, suggest) : new Resolver(result[0]); + }(); + } + + for (let arr, term; index < length; index++) { + + term = query[index]; + + if (keyword) { + + arr = this.get_array(term, keyword); + arr = /*this.*/add_result(arr, result, suggest, this.resolution_ctx, + /** @type {!number} */limit, offset, 2 === length + /*, term, keyword*/ + ); + + // 1. when suggestion enabled just forward keyword if term was found + // 2. as long as the result is empty forward the pointer also + if (!suggest || !1 !== arr || !result.length) { + keyword = term; + } + } else { + + arr = this.get_array(term); + arr = /*this.*/add_result(arr, result, suggest, this.resolution, + /** @type {!number} */limit, offset, 1 === length + /*, term*/ + ); + } + + // limit reached + if (arr) { + return (/** @type {Array} */arr + ); + } + + // apply suggestions on last loop + if (suggest && index == length - 1) { + const length = result.length; + if (!length) { + // fallback to non-contextual search when no result was found + if (keyword) { + keyword = ""; + index = -1; + continue; + } + return result; + } else if (1 === length) { + return resolve ? resolve_default(result[0], limit, offset) : new Resolver(result[0]); + } + } + } + + return resolve ? intersect(result, limit, offset, suggest) : new Resolver(result[0]); +}; + +/** + * @param term + * @param keyword + * @param limit + * @param offset + * @param resolve + * @param enrich + * @param tag + * @this Index + * @return {Array|Resolver} + */ + +function single_term_query(term, keyword, limit, offset, resolve, enrich, tag) { + + const result = this.get_array(term, keyword, limit, offset, resolve, enrich, tag); + + if (this.db) { + return result.then(function (result) { + if (resolve) return result; + return result && result.length ? resolve ? resolve_default(result, limit, offset) : new Resolver(result) : resolve ? [] : new Resolver([]); + }); + } + + return result && result.length ? resolve ? resolve_default(result, limit, offset) : new Resolver(result) : resolve ? [] : new Resolver([]); +} + +/** + * Returns a 1-dimensional finalized array when the result is done (fast path return), + * returns false when suggestions is enabled and no result was found, + * or returns nothing when a set was pushed successfully to the results + * + * @private + * @param {Array} arr + * @param {Array} result + * @param {Array} suggest + * @param {number} resolution + * @param {number} limit + * @param {number} offset + * @param {boolean} single_term + * @return {Array|boolean|undefined} + */ + +function add_result(arr, result, suggest, resolution, limit, offset, single_term /*, term, keyword*/) { + + let word_arr = []; + //let arr;// = keyword ? this.ctx : this.map; + //arr = this.get_array(term, keyword); + + if (arr) { + + //const resolution = Math.min(arr.length, keyword ? this.resolution_ctx : this.resolution); + // apply reduced resolution for queries + resolution = Math.min(arr.length, resolution); + + for (let x = 0, size = 0, tmp; x < resolution; x++) { + if (tmp = arr[x]) { + + if (offset) { + // apply offset right here on single terms + if (tmp && single_term) { + if (tmp.length <= offset) { + offset -= tmp.length; + tmp = null; + } else { + tmp = tmp.slice(offset); + offset = 0; + } + } + } + + if (tmp) { + + // keep score (sparse array): + word_arr[x] = tmp; + // simplified score order: + //word_arr.push(tmp); + + if (single_term) { + size += tmp.length; + if (size >= limit) { + // fast path: + // a single term does not need to pre-collect results + break; + } + } + } + } + } + + if (word_arr.length) { + if (single_term) { + // fast path optimization + // offset was already applied at this point + // return an array will stop the query process immediately + return resolve_default(word_arr, limit, 0); + } + + result.push(word_arr); + // return nothing will continue the query + return; + } + } + + // 1. return an empty array will stop the loop + // 2. return a false value to prevent stop when using suggestions + return !suggest && word_arr; +} + +Index.prototype.get_array = function (term, keyword, limit, offset, resolve, enrich, tag) { + + let arr, swap; + + if (keyword) { + swap = this.bidirectional && term > keyword; + } + + if (this.compress) { + term = default_compress(term); + keyword && (keyword = default_compress(keyword)); + } + + if (this.db) { + return keyword ? this.db.get(swap ? keyword : term, // term + swap ? term : keyword, // ctx + limit, offset, resolve, enrich, tag) : this.db.get(term, "", // ctx + limit, offset, resolve, enrich, tag); + } + + if (keyword) { + // the frequency of the starting letter is slightly less + // on the last half of the alphabet (m-z) in almost every latin language, + // so we sort downwards (https://en.wikipedia.org/wiki/Letter_frequency) + arr = this.ctx.get(swap ? term : keyword); + arr = arr && arr.get(swap ? keyword : term); + } else { + arr = this.map.get(term); + } + + return arr; +}; \ No newline at end of file diff --git a/dist/module-debug/intersect.js b/dist/module-debug/intersect.js index 2935747..8b5575a 100644 --- a/dist/module-debug/intersect.js +++ b/dist/module-debug/intersect.js @@ -1,4 +1,231 @@ -import { create_object, concat } from "./common.js"; +import { create_object, concat, sort_by_length_up, get_max_len } from "./common.js"; + +/* + + from -> result[ + res[score][id], + res[score][id], + ] + + to -> [id] + + */ + +/** + * Implementation based on Object[key] provides better suggestions + * capabilities and has less performance scaling issues on large indexes. + * + * @param arrays + * @param limit + * @param offset + * @param {boolean|Array=} suggest + * @param {boolean=} resolve + * @returns {Array} + */ + +export function intersect(arrays, limit, offset, suggest) { + + const length = arrays.length; + + // todo remove + // if(length < 2){ + // throw new Error("Not optimized intersect"); + // } + + let result = [], + size = 0, + check, + check_suggest, + check_new, + res_arr; + + + if (suggest) { + suggest = []; + } + + // 1. a reversed order prioritize the order of words from a query + // because it ends with the first term. + // 2. process terms in reversed order often has advantage for + // the fast path "end reached". + + // alternatively the results could be sorted by length up + //arrays.sort(sort_by_length_up); + + // todo the outer loop should be the res array instead of term array, + // this isn't just simple because the intersection calculation needs to reflect this + //const maxlen = get_max_len(arrays); + + for (let x = length - 1, found; 0 <= x; x--) { + //for(let x = 0, found; x < length; x++){ + + res_arr = arrays[x]; + check_new = create_object(); + found = !check; + + // process relevance in forward order (direction is + // important for adding IDs during the last round) + + for (let y = 0, ids; y < res_arr.length; y++) { + + ids = res_arr[y]; + if (!ids || !ids.length) continue; + + for (let z = 0, id; z < ids.length; z++) { + + id = ids[z]; + + // check exists starting from the 2nd slot + if (check) { + if (check[id]) { + + // check if in last round + if (!x) { + //if(x === length - 1){ + + if (offset) { + offset--; + } else { + + result[size++] = id; + + if (size === limit) { + // fast path "end reached" + return result; /*resolve === false + ? { result, suggest } + :*/ + } + } + } + + if (x || suggest) { + //if((x < length - 1) || suggest){ + check_new[id] = 1; + } + + found = /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + } + + if (suggest) { + + if (!check_suggest[id]) { + check_suggest[id] = 1; + const arr = suggest[y] || (suggest[y] = []); + arr.push(id); + } + + // OLD: + // + // check_idx = (check_suggest[id] || 0) + 1; + // check_suggest[id] = check_idx; + // + // // do not adding IDs which are already included in the result (saves one loop) + // // the first intersection match has the check index 2, so shift by -2 + // + // if(check_idx < length){ + // + // const tmp = suggest[check_idx - 2] || (suggest[check_idx - 2] = []); + // tmp[tmp.length] = id; + // } + } + } else { + + // pre-fill in first round + check_new[id] = 1; + } + } + } + + if (suggest) { + + // re-use the first pre-filled check for suggestions + check || (check_suggest = check_new); + } else if (!found) { + + return []; + } + + check = check_new; + } + + // return intermediate result + // if(resolve === false){ + // return { result, suggest }; + // } + + if (suggest) { + + // needs to iterate in reverse direction + for (let x = suggest.length - 1, ids, len; 0 <= x; x--) { + + ids = suggest[x]; + len = ids.length; + + for (let y = 0, id; y < len; y++) { + + id = ids[y]; + + if (!check[id]) { + + if (offset) { + offset--; + } else { + + result[size++] = id; + + if (size === limit) { + // fast path "end reached" + return result; + } + } + + check[id] = 1; + } + } + } + } + + return result; +} + +/** + * @param mandatory + * @param arrays + * @returns {Array} + */ + +export function intersect_union(mandatory, arrays) { + const check = create_object(), + union = create_object(), + result = []; + + + for (let x = 0; x < mandatory.length; x++) { + + check[mandatory[x]] = 1; + } + + for (let x = 0, arr; x < arrays.length; x++) { + + arr = arrays[x]; + + for (let y = 0, id; y < arr.length; y++) { + + id = arr[y]; + + if (check[id]) { + + if (!union[id]) { + + union[id] = 1; + result.push(id); + } + } + } + } + + return result; +} /** * Implementation based on Array.includes() provides better performance, @@ -192,203 +419,4 @@ import { create_object, concat } from "./common.js"; // } // // return result; -// } - -/** - * Implementation based on Object[key] provides better suggestions - * capabilities and has less performance scaling issues on large indexes. - * - * @param arrays - * @param limit - * @param offset - * @param {boolean|Array=} suggest - * @returns {Array} - */ - -export function intersect(arrays, limit, offset, suggest) { - - const length = arrays.length; - let result = [], - check, - check_suggest, - size = 0; - - - if (suggest) { - - suggest = []; - } - - // process terms in reversed order often has advantage for the fast path "end reached". - // also a reversed order prioritize the order of words from a query. - - for (let x = length - 1; 0 <= x; x--) { - const word_arr = arrays[x], - word_arr_len = word_arr.length, - check_new = create_object(); - - - let found = !check; - - // process relevance in forward order (direction is - // important for adding IDs during the last round) - - for (let y = 0; y < word_arr_len; y++) { - const arr = word_arr[y], - arr_len = arr.length; - - - if (arr_len) { - - // loop through IDs - - for (let z = 0, check_idx, id; z < arr_len; z++) { - - id = arr[z]; - - if (check) { - - if (check[id]) { - - // check if in last round - - if (!x) { - - if (offset) { - - offset--; - } else { - - result[size++] = id; - - if (size === limit) { - - // fast path "end reached" - - return result; - } - } - } - - if (x || suggest) { - - check_new[id] = 1; - } - - found = /* append: */ /* skip update: */ /* skip_update: */!0; - } - - if (suggest) { - - check_idx = (check_suggest[id] || 0) + 1; - check_suggest[id] = check_idx; - - // do not adding IDs which are already included in the result (saves one loop) - // the first intersection match has the check index 2, so shift by -2 - - if (check_idx < length) { - - const tmp = suggest[check_idx - 2] || (suggest[check_idx - 2] = []); - tmp[tmp.length] = id; - } - } - } else { - - // pre-fill in first round - - check_new[id] = 1; - } - } - } - } - - if (suggest) { - - // re-use the first pre-filled check for suggestions - - check || (check_suggest = check_new); - } else if (!found) { - - return []; - } - - check = check_new; - } - - if (suggest) { - - // needs to iterate in reverse direction - - for (let x = suggest.length - 1, arr, len; 0 <= x; x--) { - - arr = suggest[x]; - len = arr.length; - - for (let y = 0, id; y < len; y++) { - - id = arr[y]; - - if (!check[id]) { - - if (offset) { - - offset--; - } else { - - result[size++] = id; - - if (size === limit) { - - // fast path "end reached" - - return result; - } - } - - check[id] = 1; - } - } - } - } - - return result; -} - -/** - * @param mandatory - * @param arrays - * @returns {Array} - */ - -export function intersect_union(mandatory, arrays) { - const check = create_object(), - union = create_object(), - result = []; - - - for (let x = 0; x < mandatory.length; x++) { - - check[mandatory[x]] = 1; - } - - for (let x = 0, arr; x < arrays.length; x++) { - - arr = arrays[x]; - - for (let y = 0, id; y < arr.length; y++) { - - id = arr[y]; - - if (check[id]) { - - if (!union[id]) { - - union[id] = 1; - result[result.length] = id; - } - } - } - } - - return result; -} \ No newline at end of file +// } \ No newline at end of file diff --git a/dist/module-debug/keystore.js b/dist/module-debug/keystore.js new file mode 100644 index 0000000..3f7b6d7 --- /dev/null +++ b/dist/module-debug/keystore.js @@ -0,0 +1,388 @@ +import { create_object } from "./common.js"; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreObj(bitlength = 8) { + + if (!(this instanceof KeystoreObj)) { + return new KeystoreObj(bitlength); + } + + this.index = create_object(); + this.keys = []; + + if (32 < bitlength) { + this.crc = lcg64; + this.bit = BigInt(bitlength); + } else { + this.crc = lcg; + this.bit = bitlength; + } + + return (/*this.proxy =*/new Proxy(this, { + get(target, key) { + const address = target.crc(key), + obj = target.index[address]; + + return obj && obj[key]; + }, + set(target, key, value) { + const address = target.crc(key); + let obj = target.index[address]; + if (!obj) { + target.index[address] = obj = create_object(); + target.keys.push(address); + } + obj[key] = value; + return (/* tag? */ /* stringify */ + /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ + // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ + ); + }, + delete(target, key) { + const address = target.crc(key), + obj = target.index[address]; + + obj && delete obj[key]; + return !0; + } + }) + ); +} + +KeystoreObj.prototype.clear = function () { + this.index = create_object(); + this.keys = []; +}; + +// KeystoreObj.prototype.destroy = function(){ +// this.index = null; +// this.keys = null; +// this.proxy = null; +// }; + +function _slice(self, start, end, splice) { + let arr = []; + for (let i = 0, index; i < self.index.length; i++) { + index = self.index[i]; + if (start >= index.length) { + start -= index.length; + } else { + const tmp = index[splice ? "splice" : "slice"](start, end), + length = tmp.length; + + if (length) { + arr = arr.length ? arr.concat(tmp) : tmp; + end -= length; + if (splice) self.length -= length; + if (!end) break; + } + start = 0; + } + } + return arr; +} + +/** + * @param arr + * @constructor + */ + +export function KeystoreArray(arr) { + + if (!(this instanceof KeystoreArray)) { + return new KeystoreArray(arr); + } + + this.index = arr ? [arr] : []; + this.length = arr ? arr.length : 0; + const self = this; + + return (/*this.proxy =*/new Proxy([], { + get(target, key) { + if ("length" === key) { + return self.length; + } + if ("push" === key) { + return function (value) { + self.index[self.index.length - 1].push(value); + self.length++; + }; + } + if ("pop" === key) { + return function () { + if (self.length) { + self.length--; + return self.index[self.index.length - 1].pop(); + } + }; + } + if ("indexOf" === key) { + return function (key) { + let index = 0; + for (let i = 0, arr, tmp; i < self.index.length; i++) { + arr = self.index[i]; + //if(!arr.includes(key)) continue; + tmp = arr.indexOf(key); + if (0 <= tmp) return index + tmp; + index += arr.length; + } + return -1; + }; + } + if ("includes" === key) { + return function (key) { + for (let i = 0; i < self.index.length; i++) { + if (self.index[i].includes(key)) { + return !0; + } + } + return (/* suggest */ /* append: */ /* enrich */!1 + ); + }; + } + if ("slice" === key) { + return function (start, end) { + return _slice(self, start || 0, end || self.length, !1); + }; + } + if ("splice" === key) { + return function (start, end) { + return _slice(self, start || 0, end || self.length, !0); + }; + } + if ("constructor" === key) { + return Array; + } + if ("symbol" == typeof key /*|| isNaN(key)*/) { + // not supported + return; + } + const arr = self.index[0 | key / 2147483648]; + + return arr && arr[key]; + }, + set(target, key, value) { + const index = 0 | key / 2147483648, + arr = self.index[index] || (self.index[index] = []); + + arr[key] = value; + self.length++; + return !0; + } + }) + ); +} + +KeystoreArray.prototype.clear = function () { + this.index.length = 0; +}; + +KeystoreArray.prototype.destroy = function () { + this.index = null; + this.proxy = null; +}; + +KeystoreArray.prototype.push = function () {}; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreMap(bitlength = 8) { + + if (!(this instanceof KeystoreMap)) { + return new KeystoreMap(bitlength); + } + + this.index = create_object(); + this.refs = []; + this.size = 0; + + if (32 < bitlength) { + this.crc = lcg64; + this.bit = BigInt(bitlength); + } else { + this.crc = lcg; + this.bit = bitlength; + } +} + +KeystoreMap.prototype.get = function (key) { + const address = this.crc(key), + map = this.index[address]; + + return map && map.get(key); +}; + +KeystoreMap.prototype.set = function (key, value) { + const address = this.crc(key); + let map = this.index[address]; + if (map) { + let size = map.size; + map.set(key, value); + size -= map.size; + size && this.size++; + } else { + this.index[address] = map = new Map([[key, value]]); + this.refs.push(map); + } +}; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreSet(bitlength = 8) { + + if (!(this instanceof KeystoreSet)) { + return new KeystoreSet(bitlength); + } + + // using plain Object with numeric key access + // just for max performance + this.index = create_object(); + this.refs = []; + + if (32 < bitlength) { + this.crc = lcg64; + this.bit = BigInt(bitlength); + } else { + this.crc = lcg; + this.bit = bitlength; + } +} + +KeystoreSet.prototype.add = function (key) { + const address = this.crc(key); + let set = this.index[address]; + if (set) { + let size = set.size; + set.add(key); + size -= set.size; + size && this.size++; + } else { + this.index[address] = set = new Set([key]); + this.refs.push(set); + } +}; + +KeystoreMap.prototype.has = KeystoreSet.prototype.has = function (key) { + const address = this.crc(key), + map_or_set = this.index[address]; + + return map_or_set && map_or_set.has(key); +}; + +/* +KeystoreMap.prototype.size = +KeystoreSet.prototype.size = function(){ + let size = 0; + const values = Object.values(this.index); + for(let i = 0; i < values.length; i++){ + size += values[i].size; + } + return size; +}; +*/ + +KeystoreMap.prototype.delete = KeystoreSet.prototype.delete = function (key) { + const address = this.crc(key), + map_or_set = this.index[address]; + + // set && (set.size === 1 + // ? this.index.delete(address) + // : set.delete(key)); + map_or_set && map_or_set.delete(key) && this.size--; +}; + +KeystoreMap.prototype.clear = KeystoreSet.prototype.clear = function () { + this.index = create_object(); + this.refs = []; + this.size = 0; +}; + +// KeystoreMap.prototype.destroy = +// KeystoreSet.prototype.destroy = function(){ +// this.index = null; +// this.refs = null; +// this.proxy = null; +// }; + +KeystoreMap.prototype.values = KeystoreSet.prototype.values = function* () { + // alternatively iterate through this.keys[] + //const refs = Object.values(this.index); + for (let i = 0; i < this.refs.length; i++) { + for (let value of this.refs[i].values()) { + yield value; + } + } +}; + +KeystoreMap.prototype.keys = KeystoreSet.prototype.keys = function* () { + //const values = Object.values(this.index); + for (let i = 0; i < this.refs.length; i++) { + for (let key of this.refs[i].keys()) { + yield key; + } + } +}; + +KeystoreMap.prototype.entries = KeystoreSet.prototype.entries = function* () { + //const values = Object.values(this.index); + for (let i = 0; i < this.refs.length; i++) { + for (let entry of this.refs[i].entries()) { + yield entry; + } + } +}; + +/** + * Linear Congruential Generator (LCG) + * @param str + * @this {KeystoreMap|KeystoreSet} + */ + +function lcg(str) { + let range = 2 ** this.bit - 1; + if ("number" == typeof str) { + return str & range; + } + let crc = 0, + bit = this.bit + 1; + for (let i = 0; i < str.length; i++) { + crc = (crc * bit ^ str.charCodeAt(i)) & range; + } + // shift Int32 to UInt32 because negative numbers + // extremely slows down key lookup + return 32 === this.bit ? crc + 2147483648 : crc; // & 0xFFFF; +} + +/** + * @param str + * @this {KeystoreMap|KeystoreSet} + */ + +function lcg64(str) { + let range = BigInt(2) ** /** @type {!BigInt} */this.bit - BigInt(1), + type = typeof str; + if ("bigint" == type) { + return (/** @type {!BigInt} */str & range + ); + } + if ("number" == type) { + return BigInt(str) & range; + } + let crc = BigInt(0), + bit = /** @type {!BigInt} */this.bit + BigInt(1); + for (let i = 0; i < str.length; i++) { + crc = (crc * bit ^ BigInt(str.charCodeAt(i))) & range; + } + return crc; // & 0xFFFFFFFFFFFFFFFF; +} \ No newline at end of file diff --git a/dist/module-debug/lang.js b/dist/module-debug/lang.js deleted file mode 100644 index 954ae86..0000000 --- a/dist/module-debug/lang.js +++ /dev/null @@ -1,321 +0,0 @@ -import { IndexInterface } from "./type.js"; -import { create_object, get_keys } from "./common.js"; - -/** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} _collapse - * @returns {string|Array} - * @this IndexInterface - */ - -export function pipeline(str, normalize, split, _collapse) { - - if (str) { - - if (normalize) { - - str = replace(str, /** @type {Array} */normalize); - } - - if (this.matcher) { - - str = replace(str, this.matcher); - } - - if (this.stemmer && 1 < str.length) { - - str = replace(str, this.stemmer); - } - - if (_collapse && 1 < str.length) { - - str = collapse(str); - } - - if (split || "" === split) { - - const words = str.split( /** @type {string|RegExp} */split); - - return this.filter ? filter(words, this.filter) : words; - } - } - - return str; -} - -// TODO improve normalize + remove non-delimited chars like in "I'm" + split on whitespace+ - -export const regex_whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; -// https://github.com/nextapps-de/flexsearch/pull/414 -//export const regex_whitespace = /[\s\xA0\u2000-\u200B\u2028\u2029\u3000\ufeff!"#$%&'()*+,\-./:;<=>?@[\\\]^_`{|}~]/ -const regex_normalize = /[\u0300-\u036f]/g; - -export function normalize(str) { - - if (str.normalize) { - - str = str.normalize("NFD").replace(regex_normalize, ""); - } - - return str; -} - -/** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} _collapse - * @returns {string|Array} - */ - -// FlexSearch.prototype.pipeline = function(str, normalize, split, _collapse){ -// -// if(str){ -// -// if(normalize && str){ -// -// str = replace(str, /** @type {Array} */ (normalize)); -// } -// -// if(str && this.matcher){ -// -// str = replace(str, this.matcher); -// } -// -// if(this.stemmer && str.length > 1){ -// -// str = replace(str, this.stemmer); -// } -// -// if(_collapse && str.length > 1){ -// -// str = collapse(str); -// } -// -// if(str){ -// -// if(split || (split === "")){ -// -// const words = str.split(/** @type {string|RegExp} */ (split)); -// -// return this.filter ? filter(words, this.filter) : words; -// } -// } -// } -// -// return str; -// }; - -// export function pipeline(str, normalize, matcher, stemmer, split, _filter, _collapse){ -// -// if(str){ -// -// if(normalize && str){ -// -// str = replace(str, normalize); -// } -// -// if(matcher && str){ -// -// str = replace(str, matcher); -// } -// -// if(stemmer && str.length > 1){ -// -// str = replace(str, stemmer); -// } -// -// if(_collapse && str.length > 1){ -// -// str = collapse(str); -// } -// -// if(str){ -// -// if(split !== false){ -// -// str = str.split(split); -// -// if(_filter){ -// -// str = filter(str, _filter); -// } -// } -// } -// } -// -// return str; -// } - - -/** - * @param {Array} words - * @returns {Object} - */ - -export function init_filter(words) { - - const filter = create_object(); - - for (let i = 0, length = words.length; i < length; i++) { - - filter[words[i]] = 1; - } - - return filter; -} - -/** - * @param {!Object} obj - * @param {boolean} is_stemmer - * @returns {Array} - */ - -export function init_stemmer_or_matcher(obj, is_stemmer) { - const keys = get_keys(obj), - length = keys.length, - final = []; - - - let removal = "", - count = 0; - - for (let i = 0, key, tmp; i < length; i++) { - - key = keys[i]; - tmp = obj[key]; - - if (tmp) { - - final[count++] = regex(is_stemmer ? "(?!\\b)" + key + "(\\b|_)" : key); - final[count++] = tmp; - } else { - - removal += (removal ? "|" : "") + key; - } - } - - if (removal) { - - final[count++] = regex(is_stemmer ? "(?!\\b)(" + removal + ")(\\b|_)" : "(" + removal + ")"); - final[count] = ""; - } - - return final; -} - -/** - * @param {!string} str - * @param {Array} regexp - * @returns {string} - */ - -export function replace(str, regexp) { - - for (let i = 0, len = regexp.length; i < len; i += 2) { - - str = str.replace(regexp[i], regexp[i + 1]); - - if (!str) { - - break; - } - } - - return str; -} - -/** - * @param {!string} str - * @returns {RegExp} - */ - -export function regex(str) { - - return new RegExp(str, "g"); -} - -/** - * Regex: replace(/(?:(\w)(?:\1)*)/g, "$1") - * @param {!string} string - * @returns {string} - */ - -export function collapse(string) { - - let final = "", - prev = ""; - - for (let i = 0, len = string.length, char; i < len; i++) { - - if ((char = string[i]) !== prev) { - - final += prev = char; - } - } - - return final; -} - -// TODO using fast-swap -export function filter(words, map) { - const length = words.length, - filtered = []; - - - for (let i = 0, count = 0; i < length; i++) { - - const word = words[i]; - - if (word && !map[word]) { - - filtered[count++] = word; - } - } - - return filtered; -} - -// const chars = {a:1, e:1, i:1, o:1, u:1, y:1}; -// -// function collapse_repeating_chars(string){ -// -// let collapsed_string = "", -// char_prev = "", -// char_next = ""; -// -// for(let i = 0; i < string.length; i++){ -// -// const char = string[i]; -// -// if(char !== char_prev){ -// -// if(i && (char === "h")){ -// -// if((chars[char_prev] && chars[char_next]) || (char_prev === " ")){ -// -// collapsed_string += char; -// } -// } -// else{ -// -// collapsed_string += char; -// } -// } -// -// char_next = ( -// -// (i === (string.length - 1)) ? -// -// "" -// : -// string[i + 1] -// ); -// -// char_prev = char; -// } -// -// return collapsed_string; -// } \ No newline at end of file diff --git a/dist/module-debug/lang/arabic/default.js b/dist/module-debug/lang/arabic/default.js deleted file mode 100644 index 89f65a6..0000000 --- a/dist/module-debug/lang/arabic/default.js +++ /dev/null @@ -1,27 +0,0 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline } from "../../lang.js"; - -export const rtl = /* append: */ /* skip update: */ /* skip_update: */!0; -export const tokenize = ""; -export default { - encode: encode, - rtl: !0 -}; - -const regex = /[\x00-\x7F]+/g, - split = /\s+/; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).replace(regex, " "), - /* normalize: */ - /* collapse: */!1, - /* split: */split, !1); -} \ No newline at end of file diff --git a/dist/module-debug/lang/at.js b/dist/module-debug/lang/at.js deleted file mode 100644 index af404df..0000000 --- a/dist/module-debug/lang/at.js +++ /dev/null @@ -1,41 +0,0 @@ -/** - * http://www.ranks.nl/stopwords - * @type {Array} - */ - -export const filter = ["aber", "als", "am", "an", "auch", "auf", "aus", "bei", "bin", "bis", "bist", "da", "dadurch", "daher", "darum", "das", "daß", "dass", "dein", "deine", "dem", "den", "der", "des", "dessen", "deshalb", "die", "dies", "dieser", "dieses", "doch", "dort", "du", "durch", "ein", "eine", "einem", "einen", "einer", "eines", "er", "es", "euer", "eure", "für", "hatte", "hatten", "hattest", "hattet", "hier", "hinter", "ich", "ihr", "ihre", "im", "in", "ist", "ja", "jede", "jedem", "jeden", "jeder", "jedes", "jener", "jenes", "jetzt", "kann", "kannst", "können", "könnt", "machen", "mein", "meine", "mit", "muß", "mußt", "musst", "müssen", "müßt", "nach", "nachdem", "nein", "nicht", "nun", "oder", "seid", "sein", "seine", "sich", "sie", "sind", "soll", "sollen", "sollst", "sollt", "sonst", "soweit", "sowie", "und", "unser", "unsere", "unter", "vom", "von", "vor", "wann", "warum", "was", "weiter", "weitere", "wenn", "wer", "werde", "werden", "werdet", "weshalb", "wie", "wieder", "wieso", "wir", "wird", "wirst", "wo", "woher", "wohin", "zu", "zum", "zur", "über"]; - -/** - * @type {Object} - */ - -export const stemmer = { - - niss: "", - isch: "", - lich: "", - heit: "", - keit: "", - end: "", - ung: "", - est: "", - ern: "", - em: "", - er: "", - en: "", - es: "", - st: "", - ig: "", - ik: "", - e: "", - s: "" -}; - -export const matcher = {}; - -export default { - - filter: filter, - stemmer: stemmer, - matcher: matcher -}; \ No newline at end of file diff --git a/dist/module-debug/lang/cjk/default.js b/dist/module-debug/lang/cjk/default.js deleted file mode 100644 index a0e94f2..0000000 --- a/dist/module-debug/lang/cjk/default.js +++ /dev/null @@ -1,26 +0,0 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline } from "../../lang.js"; - -export const rtl = /* normalize: */ /* collapse: */ -/* normalize: */ -/* collapse: */!1; -export const tokenize = "strict"; -export default { - encode: encode, - rtl: !1, - tokenize: "strict" -}; - -const regex = /[\x00-\x7F]+/g; - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).replace(regex, ""), !1, - /* split: */"", !1); -} \ No newline at end of file diff --git a/dist/module-debug/lang/cyrillic/default.js b/dist/module-debug/lang/cyrillic/default.js deleted file mode 100644 index de45263..0000000 --- a/dist/module-debug/lang/cyrillic/default.js +++ /dev/null @@ -1,27 +0,0 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline } from "../../lang.js"; - -export const rtl = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ -/* normalize: */ -/* collapse: */!1; -export const tokenize = ""; -export default { - encode: encode, - rtl: !1 -}; - -const regex = /[\x00-\x7F]+/g, - split = /\s+/; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).replace(regex, " "), !1, - /* split: */split, !1); -} \ No newline at end of file diff --git a/dist/module-debug/lang/de.js b/dist/module-debug/lang/de.js index 8196777..717c431 100644 --- a/dist/module-debug/lang/de.js +++ b/dist/module-debug/lang/de.js @@ -2,53 +2,38 @@ * Filter are also known as "stopwords", they completely filter out words from being indexed. * Source: http://www.ranks.nl/stopwords * Object Definition: Just provide an array of words. - * @type {Array} + * @type {Set} */ -export const filter = ["aber", "als", "am", "an", "auch", "auf", "aus", "bei", "bin", "bis", "bist", "da", "dadurch", "daher", "darum", "das", "daß", "dass", "dein", "deine", "dem", "den", "der", "des", "dessen", "deshalb", "die", "dies", "dieser", "dieses", "doch", "dort", "du", "durch", "ein", "eine", "einem", "einen", "einer", "eines", "er", "es", "euer", "eure", "für", "hatte", "hatten", "hattest", "hattet", "hier", "hinter", "ich", "ihr", "ihre", "im", "in", "ist", "ja", "jede", "jedem", "jeden", "jeder", "jedes", "jener", "jenes", "jetzt", "kann", "kannst", "können", "könnt", "machen", "mein", "meine", "mit", "muß", "mußt", "musst", "müssen", "müßt", "nach", "nachdem", "nein", "nicht", "nun", "oder", "seid", "sein", "seine", "sich", "sie", "sind", "soll", "sollen", "sollst", "sollt", "sonst", "soweit", "sowie", "und", "unser", "unsere", "unter", "vom", "von", "vor", "wann", "warum", "was", "weiter", "weitere", "wenn", "wer", "werde", "werden", "werdet", "weshalb", "wie", "wieder", "wieso", "wir", "wird", "wirst", "wo", "woher", "wohin", "zu", "zum", "zur", "über"]; +export const filter = new Set(["aber", "als", "am", "an", "auch", "auf", "aus", "bei", "bin", "bis", "bist", "da", "dadurch", "daher", "darum", "das", "dass", "dass", "dein", "deine", "dem", "den", "der", "des", "dessen", "deshalb", "die", "dies", "dieser", "dieses", "doch", "dort", "du", "durch", "ein", "eine", "einem", "einen", "einer", "eines", "er", "es", "euer", "eure", "fuer", "hatte", "hatten", "hattest", "hattet", "hier", "hinter", "ich", "ihr", "ihre", "im", "in", "ist", "ja", "jede", "jedem", "jeden", "jeder", "jedes", "jener", "jenes", "jetzt", "ggf", "kann", "kannst", "koennen", "koennt", "machen", "mein", "meine", "mit", "muss", "musst", "musst", "muessen", "muesst", "nach", "nachdem", "nein", "nicht", "noch", "nun", "oder", "seid", "sein", "seine", "sich", "sie", "sind", "soll", "sollen", "sollst", "sollt", "sonst", "soweit", "sowie", "und", "unser", "unsere", "unter", "usw", "uvm", "vom", "von", "vor", "wann", "warum", "was", "weiter", "weitere", "wenn", "wer", "werde", "werden", "werdet", "weshalb", "wie", "wieder", "wieso", "wir", "wird", "wirst", "wo", "woher", "wohin", "zu", "zum", "zur", "ueber"]); /** * Stemmer removes word endings and is a kind of "partial normalization". A word ending just matched when the word length is bigger than the matched partial. * Example: The word "correct" and "correctness" could be the same word, so you can define {"ness": ""} to normalize the ending. * Object Definition: the key represents the word ending, the value contains the replacement (or empty string for removal). - * @type {Object} + * http://snowball.tartarus.org/algorithms/german/stemmer.html + * @type {Map} */ -export const stemmer = { - - niss: "", - isch: "", - lich: "", - heit: "", - keit: "", - ell: "", - bar: "", - end: "", - ung: "", - est: "", - ern: "", - em: "", - er: "", - en: "", - es: "", - st: "", - ig: "", - ik: "", - e: "", - s: "" -}; +export const stemmer = new Map([["niss", ""], ["isch", ""], ["lich", ""], ["heit", ""], ["keit", ""], ["ell", ""], ["bar", ""], ["end", ""], ["ung", ""], ["est", ""], ["ern", ""], ["em", ""], ["er", ""], ["en", ""], ["es", ""], ["st", ""], ["ig", ""], ["ik", ""], ["e", ""], ["s", ""]]); /** * Matcher replaces all occurrences of a given string regardless of its position and is also a kind of "partial normalization". * Object Definition: the key represents the target term, the value contains the search string which should be replaced (could also be an array of multiple terms). - * @type {Object|string>} + * @type {Map} */ - -export const matcher = {}; +const map = new Map([["_", " "], ["ä", "ae"], ["ö", "oe"], ["ü", "ue"], ["ß", "ss"], ["&", " und "], ["€", " EUR "]]); export default { - + normalize: function (str) { + return str.toLowerCase(); + }, + prepare: function (str) { + // normalization + if (/[_äöüß&€]/.test(str)) str = str.replace(/[_äöüß&€]/g, match => map.get(match)); + // street names + return str.replace(/str\b/g, "strasse").replace(/(?!\b)strasse\b/g, " strasse").replace(/\bst\b/g, "sankt"); + }, filter: filter, - stemmer: stemmer, - matcher: matcher + stemmer: stemmer }; \ No newline at end of file diff --git a/dist/module-debug/lang/en.js b/dist/module-debug/lang/en.js index 7b1ba3a..65b6220 100644 --- a/dist/module-debug/lang/en.js +++ b/dist/module-debug/lang/en.js @@ -1,100 +1,128 @@ /** * http://www.ranks.nl/stopwords - * @type {Array} + * @type {Set} */ -export const filter = ["a", "about", "above", "after", "again", "against", "all", "also", "am", "an", "and", "any", "are", "aren't", "as", "at", -//"back", -"be", "because", "been", "before", "being", "below", -//"between", -"both", "but", "by", "can", "cannot", "can't", "come", "could", "couldn't", +// todo filter out minlength +export const filter = new Set(["a", "about", "above", "after", "again", "against", "all", "also", "am", "an", "and", "any", "are", "arent", "as", "at", "back", "be", "because", "been", "before", "being", "below", "between", "both", "but", "by", "can", "cannot", "cant", "come", "could", "couldnt", //"day", -"did", "didn't", "do", "does", "doesn't", "doing", "dont", "down", "during", "each", "even", "few", "first", "for", "from", "further", "get", +"did", "didnt", "do", "does", "doesnt", "doing", "dont", "down", "during", "each", "even", "few", +//"first", +"for", "from", "further", "get", //"give", -"go", -//"good", -"had", "hadn't", "has", "hasn't", "have", "haven't", "having", "he", "hed", +"go", "good", "had", "hadnt", "has", "hasnt", "have", "havent", "having", "he", "hed", //"hell", -"her", "here", "here's", "hers", "herself", "hes", "him", "himself", "his", "how", "how's", "i", "id", "if", "ill", "im", "in", "into", "is", "isn't", "it", "it's", "itself", "i've", "just", "know", "let's", "like", +"her", "here", "heres", "hers", "herself", "hes", "him", "himself", "his", "how", "hows", "i", "id", "if", "ill", "im", "in", "into", "is", "isnt", "it", "its", "itself", "ive", "just", "know", "lets", "like", //"look", -"make", "me", "more", "most", "mustn't", "my", "myself", "new", "no", "nor", "not", "now", "of", "off", "on", "once", -//"one", -"only", "or", "other", "ought", "our", "our's", "ourselves", "out", "over", "own", -//"people", -"same", "say", "see", "shan't", "she", "she'd", "shell", "shes", "should", "shouldn't", "so", "some", "such", -//"take", -"than", "that", "that's", "the", "their", "theirs", "them", "themselves", "then", "there", "there's", "these", "they", "they'd", "they'll", "they're", "they've", -//"think", -"this", "those", "through", "time", "to", "too", +"lot", "make", "made", "me", "more", "most", "mustnt", "my", "myself", "new", "no", "nor", "not", "now", "of", "off", "on", "once", "one", "only", "or", "other", "ought", "our", "ours", "ourselves", "out", "over", "own", "people", "same", "say", "see", "shant", "she", "shed", "shell", "shes", "should", "shouldnt", "so", "some", "such", "take", "than", "that", "thats", "the", "their", "theirs", "them", "themselves", "then", "there", "theres", "these", "they", "theyd", "theyll", "theyre", "theyve", "think", "this", "those", "through", "time", "times", "to", "too", //"two", -//"under", -"until", "up", "us", -//"use", -"very", "want", "was", "wasn't", "way", "we", "wed", "well", "were", "weren't", "we've", "what", "what's", "when", "when's", "where", "where's", "which", "while", "who", "whom", "who's", "why", "why's", "will", "with", "won't", -//"work", -"would", "wouldn't", +"under", "until", "up", "us", "use", "very", "want", "was", "wasnt", "way", "we", "wed", "well", "were", "werent", "weve", "what", "whats", "when", "whens", "where", "wheres", "which", "while", "who", "whom", "whos", "why", "whys", "will", "with", "wont", "work", "would", "wouldnt", //"year", -"you", "you'd", "you'll", "your", "you're", "your's", "yourself", "yourselves", "you've"]; +"ya", "you", "youd", "youll", "your", "youre", "yours", "yourself", "yourselves", "youve"]); /** * @type {Object} */ -export const stemmer = { +export const stemmer = new Map([["ational", ""], ["iveness", ""], ["fulness", ""], ["ousness", ""], ["ization", ""], ["tional", ""], ["biliti", ""], ["icate", ""], ["ative", ""], ["alize", ""], ["iciti", ""], ["entli", ""], ["ousli", ""], ["alism", ""], ["ation", ""], ["aliti", ""], ["iviti", ""], ["ement", ""], ["izer", ""], ["able", ""], ["alli", ""], ["ator", ""], ["logi", ""], ["ical", ""], ["ance", ""], ["ence", ""], ["ness", ""], ["ble", ""], ["ment", ""], ["eli", ""], ["bli", ""], ["ful", ""], ["ant", ""], ["ent", ""], ["ism", ""], ["ate", ""], ["iti", ""], ["ous", ""], ["ive", ""], ["ize", ""], ["ing", ""], ["ion", ""], ["al", ""], ["ou", ""], ["er", ""], ["ic", ""], ["ly", ""]]); - ational: "ate", - iveness: "ive", - fulness: "ful", - ousness: "ous", - ization: "ize", - tional: "tion", - biliti: "ble", - icate: "ic", - ative: "", - alize: "al", - iciti: "ic", - entli: "ent", - ousli: "ous", - alism: "al", - ation: "ate", - aliti: "al", - iviti: "ive", - ement: "", - enci: "ence", - anci: "ance", - izer: "ize", - alli: "al", - ator: "ate", - logi: "log", - ical: "ic", - ance: "", - ence: "", - ness: "", - able: "", - ible: "", - ment: "", - eli: "e", - bli: "ble", - ful: "", - ant: "", - ent: "", - ism: "", - ate: "", - iti: "", - ous: "", - ive: "", - ize: "", - al: "", - ou: "", - er: "", - ic: "" -}; +// export const replacer = new Map([ +// ["&", " and "], +// ]); -export const matcher = {}; +/* + he’s (= he is / he has) + she’s (= she is / she has) + I’ll (= I will) + I’ve (= I have) + I’d (= I would / I had) + don’t (= do not) + doesn’t (= does not) + didn’t (= did not) + isn’t (= is not) + hasn’t (= has not) + can’t (= cannot) + won’t (= will not) +*/ + +// const explode = new Map([ +// ["^i'm$", "i am"], +// ["^can't$", "can not"], +// ["^cannot$", "can not"], +// ["^won't$", "will not"], +// ["'s$", " is has"], +// ["n't$", " not"], +// ["'ll$", " will"], +// ["'re$", " are"], +// ["'ve$", " have"], +// ["'d$", " would had"], +// ]); + +// const pairs = [ +// /´`’ʼ/, /´`’ʼ/g, "'", +// /_/, /_+/g, " ", +// /&/, /&/g, " and ", +// /\bi'm\b/, /\bi'm\b/g, "i am", +// /\b(can't|cannot)\b/, /\b(can't|cannot)\b/g, "can not", +// /\bwon't\b/, /\bwon't\b/g, "will not", +// /[a-z]n't\b/, /[a-z]n't\b/g, "$1 not", +// /[a-z]'s\b/, /([a-z])'s\b/g, "$1 is has", +// /[a-z]'ll\b/, /[a-z]'ll\b/g, "$1 will", +// /[a-z]'re\b/, /[a-z]'re\b/g, "$1 are", +// /[a-z]'ve\b/, /[a-z]'ve\b/g, "$1 have", +// /[a-z]'d\b/, /[a-z]'d\b/g, "$1 is has" +// ]; + +// const map = new Map([ +// ["´", "'"], +// ["`", "'"], +// ["’", "'"], +// ["ʼ", "'"], +// ["_", " "], +// ["&", " and "] +// ]); export default { + prepare: function (str) { + // if(/[´`’ʼ_&]/.test(str)) + // str = str.replace(/[´`’ʼ_&]/g, match => map.get(match)); + // if(/´`’ʼ/.test(str)) + // str = str.replace(/´`’ʼ/g, "'"); + // if(/_/.test(str)) + // str = str.replace(/_+/g, " "); + // if(/&/.test(str)) + // str = str.replace(/&/g, " and "); + + // if(/\bi'm\b/.test(str)) + // str = str.replace(/\bi'm\b/g, "i am"); + // if(/\b(can't|cannot)\b/.test(str)) + // str = str.replace(/\b(can't|cannot)\b/g, "can not"); + // if(/\bwon't\b/.test(str)) + // str = str.replace(/\bwon't\b/g, "will not"); + // if(/[a-z]n't\b/.test(str)) + // str = str.replace(/([a-z])n't\b/g, "$1 not"); + // if(/[a-z]'s\b/.test(str)) + // str = str.replace(/([a-z])'s\b/g, "$1 is has"); + // if(/[a-z]'ll\b/.test(str)) + // str = str.replace(/([a-z])'ll\b/g, "$1 will"); + // if(/[a-z]'re\b/.test(str)) + // str = str.replace(/([a-z])'re\b/g, "$1 are"); + // if(/[a-z]'ve\b/.test(str)) + // str = str.replace(/([a-z])'ve\b/g, "$1 have"); + // if(/[a-z]'d\b/.test(str)) + // str = str.replace(/([a-z])'d\b/g, "$1 would had"); + // return str; + + return str //.replace(/[´`’ʼ_&]/g, match => map.get(match)) + // normalization + .replace(/´`’ʼ/g, "'").replace(/_+/g, " ").replace(/&/g, " and ") + //.replace(/([0-9 ]|^)\$([0-9 ]|$)/g, "$1 USD $2") + //.replace(/([0-9 ]|^)£([0-9 ]|$)/g, "$1 GBP $2") + .replace(/\$/g, " USD ").replace(/£/g, " GBP ") + // explode short forms + .replace(/([a-z])'s\b/g, "$1 is").replace(/\bi'm\b/g, "i am").replace(/\b(can't|cannot)\b/g, "can not").replace(/\bwon't\b/g, "will not").replace(/([a-z])n't\b/g, "$1 not").replace(/([a-z])'ll\b/g, "$1 will").replace(/([a-z])'re\b/g, "$1 are").replace(/([a-z])'ve\b/g, "$1 have").replace(/([a-z])'d\b/g, "$1 would"); + }, filter: filter, - stemmer: stemmer, - matcher: matcher + stemmer: stemmer }; \ No newline at end of file diff --git a/dist/module-debug/lang/fr.js b/dist/module-debug/lang/fr.js new file mode 100644 index 0000000..573652f --- /dev/null +++ b/dist/module-debug/lang/fr.js @@ -0,0 +1,21 @@ +/** + * http://www.ranks.nl/stopwords + * http://snowball.tartarus.org/algorithms/french/stop.txt + * @type {Set} + */ + +export const filter = new Set(["au", "aux", "avec", "ce", "ces", "dans", "de", "des", "du", "elle", "en", "et", "eux", "il", "je", "la", "le", "leur", "lui", "ma", "mais", "me", "meme", "mes", "moi", "mon", "ne", "nos", "notre", "nous", "on", "ou", "par", "pas", "pour", "qu", "que", "qui", "sa", "se", "ses", "son", "sur", "ta", "te", "tes", "toi", "ton", "tu", "un", "une", "vos", "votre", "vous", "c", "d", "j", "l", "m", "n", "s", "t", "a", "y", "ete", "etee", "etees", "etes", "etant", "suis", "es", "est", "sommes", "etes", "sont", "serai", "seras", "sera", "serons", "serez", "seront", "serais", "serait", "serions", "seriez", "seraient", "etais", "etait", "etions", "etiez", "etaient", "fus", "fut", "fumes", "futes", "furent", "sois", "soit", "soyons", "soyez", "soient", "fusse", "fusses", "fut", "fussions", "fussiez", "fussent", "ayant", "eu", "eue", "eues", "eus", "ai", "as", "avons", "avez", "ont", "aurai", "auras", "aura", "aurons", "aurez", "auront", "aurais", "aurait", "aurions", "auriez", "auraient", "avais", "avait", "avions", "aviez", "avaient", "eut", "eumes", "eutes", "eurent", "aie", "aies", "ait", "ayons", "ayez", "aient", "eusse", "eusses", "eut", "eussions", "eussiez", "eussent", "ceci", "cela", "cela", "cet", "cette", "ici", "ils", "les", "leurs", "quel", "quels", "quelle", "quelles", "sans", "soi"]); + +/** + * @type {Object} + */ + +export const stemmer = new Map([["lement", ""], ["ient", ""], ["nera", ""], ["ment", ""], ["ais", ""], ["ait", ""], ["ant", ""], ["ent", ""], ["iez", ""], ["ion", ""], ["nez", ""], ["ai", ""], ["es", ""], ["er", ""], ["ez", ""], ["le", ""], ["na", ""], ["ne", ""], ["a", ""], ["e", ""]]); + +export default { + prepare: function (str) { + return str.replace(/´`’ʼ/g, "'").replace(/_+/g, " ").replace(/&/g, " et ").replace(/€/g, " EUR ").replace(/\bl'([^\b])/g, "la le $1").replace(/\bt'([^\b])/g, "ta te $1").replace(/\bc'([^\b])/g, "ca ce $1").replace(/\bd'([^\b])/g, "da de $1").replace(/\bj'([^\b])/g, "ja je $1").replace(/\bn'([^\b])/g, "na ne $1").replace(/\bm'([^\b])/g, "ma me $1").replace(/\bs'([^\b])/g, "sa se $1").replace(/\bau\b/g, "a le").replace(/\baux\b/g, "a les").replace(/\bdu\b/g, "de le").replace(/\bdes\b/g, "de les"); + }, + filter: filter, + stemmer: stemmer +}; \ No newline at end of file diff --git a/dist/module-debug/lang/latin/advanced.js b/dist/module-debug/lang/latin/advanced.js index 94f5cb1..fa5ea16 100644 --- a/dist/module-debug/lang/latin/advanced.js +++ b/dist/module-debug/lang/latin/advanced.js @@ -1,89 +1,144 @@ -import { IndexInterface } from "../../type.js"; -import { regex, replace, collapse } from "../../lang.js"; -import { encode as encode_balance } from "./balance.js"; +import Encoder from "../../encoder.js"; +import { soundex } from "./balance.js"; + +// const soundex = new Map([ +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// +// //["e", "e"], +// ["i", "e"], +// ["y", "e"], +// +// //["o", "o"], +// ["u", "o"] +// ]); + +export const matcher = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]); + +export const replacer = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"]; -export const rtl = /* normalize: */ -/* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0, + mapper: soundex, + replacer: replacer, + matcher: matcher +}; - // Phonetic Normalization - -};const regex_ae = regex("ae"), - -//regex_ai = regex("ai"), -//regex_ay = regex("ay"), -//regex_ey = regex("ey"), -regex_oe = regex("oe"), - //regex_ue = regex("ue"), -//regex_ie = regex("ie"), -//regex_sz = regex("sz"), -//regex_zs = regex("zs"), -//regex_ck = regex("ck"), -//regex_cc = regex("cc"), -regex_sh = regex("sh"), - regex_th = regex("th"), - -//regex_dt = regex("dt"), -regex_ph = regex("ph"), - regex_pf = regex("pf"), - pairs = [regex_ae, "a", -// regex_ai, "ei", -// regex_ay, "ei", -// regex_ey, "ei", -regex_oe, "o", -// regex_ue, "u", -// regex_ie, "i", -// regex_sz, "s", -// regex_zs, "s", -regex_sh, "s", -// regex_ck, "k", -// regex_cc, "k", -regex_th, "t", -// regex_dt, "t", -regex_ph, "f", regex_pf, "f", -// regex_ou, "o", -// regex_uo, "u" - -// regex("(?![aeiouy])h(?![aeiouy])"), "", -// regex("(?!^[aeiouy])h(?!^[aeiouy])"), "" -regex("(?![aeo])h(?![aeo])"), "", regex("(?!^[aeo])h(?!^[aeo])"), ""]; -//regex_ou = regex("ou"), -//regex_uo = regex("uo"); - -/** - * @param {string|number} str - * @param {boolean=} _skip_postprocessing - * @this IndexInterface - */ - -export function encode(str, _skip_postprocessing) { - - if (str) { - - str = encode_balance.call(this, str).join(" "); - - if (2 < str.length) { - - str = replace(str, pairs); - } - - if (!_skip_postprocessing) { - - if (1 < str.length) { - - str = collapse(str); - } - - if (str) { - - str = str.split(" "); - } - } - } - - return str || []; -} \ No newline at end of file +// import { regex, replace, collapse } from "../../lang.js"; +// import { encode as encode_balance } from "./balance.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // Phonetic Normalization +// +// const regex_ae = regex("ae"), +// //regex_ai = regex("ai"), +// //regex_ay = regex("ay"), +// //regex_ey = regex("ey"), +// regex_oe = regex("oe"), +// //regex_ue = regex("ue"), +// //regex_ie = regex("ie"), +// //regex_sz = regex("sz"), +// //regex_zs = regex("zs"), +// //regex_ck = regex("ck"), +// //regex_cc = regex("cc"), +// regex_sh = regex("sh"), +// regex_th = regex("th"), +// //regex_dt = regex("dt"), +// regex_ph = regex("ph"), +// regex_pf = regex("pf"); +// //regex_ou = regex("ou"), +// //regex_uo = regex("uo"); +// +// const pairs = [ +// regex_ae, "a", +// // regex_ai, "ei", +// // regex_ay, "ei", +// // regex_ey, "ei", +// regex_oe, "o", +// // regex_ue, "u", +// // regex_ie, "i", +// // regex_sz, "s", +// // regex_zs, "s", +// regex_sh, "s", +// // regex_ck, "k", +// // regex_cc, "k", +// regex_th, "t", +// // regex_dt, "t", +// regex_ph, "f", +// regex_pf, "f", +// // regex_ou, "o", +// // regex_uo, "u" +// +// // regex("(?![aeiouy])h(?![aeiouy])"), "", +// // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "" +// regex("(?![aeo])h(?![aeo])"), "", +// regex("(?!^[aeo])h(?!^[aeo])"), "" +// ]; +// +// /** +// * @param {string|number} str +// * @param {boolean=} _skip_postprocessing +// */ +// +// export function encode(str, _skip_postprocessing){ +// +// if(str){ +// +// str = encode_balance.call(this, str).join(" "); +// +// if(str.length > 2){ +// +// str = replace(str, pairs); +// } +// +// if(!_skip_postprocessing){ +// +// if(str.length > 1){ +// +// str = collapse(str); +// } +// +// if(str){ +// +// str = str.split(" "); +// } +// } +// } +// +// return str || []; +// } \ No newline at end of file diff --git a/dist/module-debug/lang/latin/balance.js b/dist/module-debug/lang/latin/balance.js index 10de393..bd8fd20 100644 --- a/dist/module-debug/lang/latin/balance.js +++ b/dist/module-debug/lang/latin/balance.js @@ -1,119 +1,279 @@ -import { IndexInterface } from "../../type.js"; -import { encode as encode_simple } from "./simple.js"; +import Encoder from "../../encoder.js"; -// custom soundex implementation +export const soundex = new Map([["b", "p"], +//["p", "p"], + +//["f", "f"], +["v", "f"], ["w", "f"], + +//["s", "s"], +["z", "s"], ["x", "s"], ["d", "t"], +//["t", "t"], + +//["m", "m"], +["n", "m"], + +//["k", "k"], +["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], + +//["r", "r"], +//["h", "h"], +//["l", "l"], + +//["a", "a"], + +//["e", "e"], +["i", "e"], ["y", "e"], + +//["o", "o"], +["u", "o"]]); -export const rtl = /* normalize: */ /* collapse: */ -/* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */!1; -export const tokenize = "strict"; export default { - encode: encode, - rtl: !1, - tokenize: "strict" - - //const regex_whitespace = /[\W_]+/g; -};const regex_strip = /[^a-z0-9]+/, - soundex = { - - b: "p", - //"p": "p", - - //"f": "f", - v: "f", w: "f", - - //"s": "s", - z: "s", - x: "s", - ß: "s", - - d: "t", - //"t": "t", - - //"l": "l", - - //"m": "m", - n: "m", - - c: "k", - g: "k", - j: "k", - //"k": "k", - q: "k", - - //"r": "r", - //"h": "h", - //"a": "a", - - //"e": "e", - i: "e", - y: "e", - - //"o": "o", - u: "o" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0, + mapper: soundex }; -// const pairs = [ -// regex_whitespace, " ", -// regex_strip, "" -// ]; - -// modified - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - str = encode_simple.call(this, str).join(" "); - - // str = this.pipeline( - // - // /* string: */ normalize("" + str).toLowerCase(), - // /* normalize: */ false, - // /* split: */ false, - // /* collapse: */ false - // ); - - const result = []; - - if (str) { - const words = str.split(regex_strip), - length = words.length; - - - for (let x = 0, tmp, count = 0; x < length; x++) { - - if ((str = words[x]) && ( /*&& (str.length > 2)*/!this.filter || !this.filter[str])) { - - tmp = str[0]; - let code = soundex[tmp] || tmp, - previous = code; //str[0]; - - //soundex[code] || code; - - for (let i = 1; i < str.length; i++) { - - tmp = str[i]; - const current = soundex[tmp] || tmp; - - if (current && current !== previous) { - - code += current; - previous = current; - - // if(code.length === 7){ - // - // break; - // } - } - } - - result[count++] = code; //(code + "0000").substring(0, 4); - } - } - } - - return result; -} \ No newline at end of file +// //import { encode as encode_simple } from "./simple.js"; +// import { pipeline } from "../../lang.js"; +// +// // custom soundex implementation +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// //const regex_whitespace = /[\W_]+/g; +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// const normalize = "".normalize && /[\u0300-\u036f]/g; +// //const regex_strip = /[^a-z0-9]+/; +// +// // const pairs = [ +// // regex_whitespace, " ", +// // regex_strip, "" +// // ]; +// +// // modified +// const pairs = new Map([ +// +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["ß", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// ["ñ", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["ç", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// ["à", "a"], +// ["á", "a"], +// ["â", "a"], +// ["ã", "a"], +// ["ä", "a"], +// ["å", "a"], +// +// //["e", "e"], +// ["è", "e"], +// ["é", "e"], +// ["ê", "e"], +// ["ë", "e"], +// ["i", "e"], +// ["ì", "e"], +// ["í", "e"], +// ["î", "e"], +// ["ï", "e"], +// ["y", "e"], +// ["ý", "e"], +// ["ŷ", "e"], +// ["ÿ", "e"], +// +// //["o", "o"], +// ["ò", "o"], +// ["ó", "o"], +// ["ô", "o"], +// ["õ", "o"], +// ["ö", "o"], +// ["ő", "o"], +// ["u", "o"], +// ["ù", "o"], +// ["ú", "o"], +// ["û", "o"], +// ["ü", "o"], +// ["ű", "o"], +// ]); +// +// const map_soundex = new Map([ +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// +// //["e", "e"], +// ["i", "e"], +// ["y", "e"], +// +// //["o", "o"], +// ["u", "o"] +// ]); +// // const soundex = { +// // +// // "b": "p", +// // //"p": "p", +// // +// // //"f": "f", +// // "v": "f", +// // "w": "f", +// // +// // //"s": "s", +// // "z": "s", +// // "x": "s", +// // "ß": "s", +// // +// // "d": "t", +// // //"t": "t", +// // +// // //"l": "l", +// // +// // //"m": "m", +// // "n": "m", +// // +// // "c": "k", +// // "g": "k", +// // "j": "k", +// // //"k": "k", +// // "q": "k", +// // +// // //"r": "r", +// // //"h": "h", +// // //"a": "a", +// // +// // //"e": "e", +// // "i": "e", +// // "y": "e", +// // +// // //"o": "o", +// // "u": "o" +// // }; +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// return pipeline.call( +// this, +// /* string: */ ("" + str).normalize("NFD").replace(normalize, "").toLowerCase(), +// /* normalize: */ map_soundex, +// /* split: */ whitespace, +// ///* collapse: */ false +// ); +// +// // return pipeline.call( +// // this, +// // /* string: */ ("" + str).toLowerCase(), +// // /* normalize: */ /*pairs*/ new Map(), +// // /* split: */ whitespace, +// // ///* collapse: */ false +// // ); +// +// // str = encode_simple.call(this, str).join(" "); +// // +// // // str = this.pipeline( +// // // +// // // /* string: */ normalize("" + str).toLowerCase(), +// // // /* normalize: */ false, +// // // /* split: */ false, +// // // /* collapse: */ false +// // // ); +// // +// // const result = []; +// // +// // if(str){ +// // +// // const words = str.split(regex_strip); +// // const length = words.length; +// // +// // for(let x = 0, tmp, count = 0; x < length; x++){ +// // +// // if((str = words[x]) /*&& (str.length > 2)*/ && (!this.filter || !this.filter.has(str))){ +// // +// // tmp = str[0]; +// // let code = soundex[tmp] || tmp; //str[0]; +// // let previous = code; //soundex[code] || code; +// // +// // for(let i = 1; i < str.length; i++){ +// // +// // tmp = str[i]; +// // const current = soundex[tmp] || tmp; +// // +// // if(current && (current !== previous)){ +// // +// // code += current; +// // previous = current; +// // +// // // if(code.length === 7){ +// // // +// // // break; +// // // } +// // } +// // } +// // +// // result[count++] = code; //(code + "0000").substring(0, 4); +// // } +// // } +// // } +// // +// // return result; +// } \ No newline at end of file diff --git a/dist/module-debug/lang/latin/default.js b/dist/module-debug/lang/latin/default.js index dac5a9c..a7e126e 100644 --- a/dist/module-debug/lang/latin/default.js +++ b/dist/module-debug/lang/latin/default.js @@ -1,23 +1,36 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline, normalize, regex_whitespace } from "../../lang.js"; +import Encoder from "../../encoder.js"; -export const rtl = /* normalize: */ -/* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ -/* normalize: */ -/* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: function (str) { + return str.toLowerCase(); + }, + dedupe: /* suggest */ /* append: */ /* enrich */!1 +}; - /** - * @param {string|number} str - * @this IndexInterface - */ - -};export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).toLowerCase(), !1, /* split: */regex_whitespace, !1); -} \ No newline at end of file +// import { pipeline } from "../../lang.js"; +// +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// return pipeline.call( +// +// this, +// /* string: */ ("" + str).toLowerCase(), +// /* normalize: */ false, +// /* split: */ whitespace, +// /* collapse: */ false +// ); +// } \ No newline at end of file diff --git a/dist/module-debug/lang/latin/exact.js b/dist/module-debug/lang/latin/exact.js new file mode 100644 index 0000000..1ef1dae --- /dev/null +++ b/dist/module-debug/lang/latin/exact.js @@ -0,0 +1,4 @@ +export default { + normalize: /* suggest */ /* append: */ /* enrich */!1, + dedupe: !1 +}; \ No newline at end of file diff --git a/dist/module-debug/lang/latin/extra.js b/dist/module-debug/lang/latin/extra.js index c55ea6a..ec53420 100644 --- a/dist/module-debug/lang/latin/extra.js +++ b/dist/module-debug/lang/latin/extra.js @@ -1,65 +1,82 @@ -import { IndexInterface } from "../../type.js"; -import { regex, replace, collapse } from "../../lang.js"; -import { encode as encode_advanced } from "./advanced.js"; +import Encoder from "../../encoder.js"; +import { soundex } from "./balance.js"; +import { matcher, replacer } from "./advanced.js"; + +export const compact = [/(?!^)[aeoy]/g, "" // old: aioy +]; -export const rtl = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0, + mapper: soundex, + replacer: replacer.concat(compact), + matcher: matcher +}; - // Soundex Normalization - -};const prefix = "(?!\\b)", - //soundex_b = regex(prefix + "p"), -// soundex_s = regex(prefix + "z"), -// soundex_k = regex(prefix + "[cgq]"), -// soundex_m = regex(prefix + "n"), -// soundex_t = regex(prefix + "d"), -// soundex_f = regex(prefix + "[vw]"), -//regex_vowel = regex(prefix + "[aeiouy]"); -regex_vowel = regex("(?!\\b)[aeo]"), - pairs = [ - -// soundex_b, "b", -// soundex_s, "s", -// soundex_k, "k", -// soundex_m, "m", -// soundex_t, "t", -// soundex_f, "f", -// regex("(?![aeiouy])h(?![aeiouy])"), "", -// regex("(?!^[aeiouy])h(?!^[aeiouy])"), "", -regex_vowel, ""]; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - if (str) { - - str = encode_advanced.call(this, str, /* append: */ /* skip update: */ /* skip_update: */ /* skip post-processing: */!0); - - if (1 < str.length) { - - //str = replace(str, pairs); - str = str.replace(regex_vowel, ""); - } - - if (1 < str.length) { - - str = collapse(str); - } - - if (str) { - - str = str.split(" "); - } - } - - return str || []; -} \ No newline at end of file +// import { regex, replace, collapse } from "../../lang.js"; +// import { encode as encode_advanced } from "./advanced.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // Soundex Normalization +// +// const prefix = "(?!\\b)"; +// const //soundex_b = regex(prefix + "p"), +// // soundex_s = regex(prefix + "z"), +// // soundex_k = regex(prefix + "[cgq]"), +// // soundex_m = regex(prefix + "n"), +// // soundex_t = regex(prefix + "d"), +// // soundex_f = regex(prefix + "[vw]"), +// //regex_vowel = regex(prefix + "[aeiouy]"); +// regex_vowel = regex(prefix + "[aeo]"); +// +// const pairs = [ +// +// // soundex_b, "b", +// // soundex_s, "s", +// // soundex_k, "k", +// // soundex_m, "m", +// // soundex_t, "t", +// // soundex_f, "f", +// // regex("(?![aeiouy])h(?![aeiouy])"), "", +// // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "", +// regex_vowel, "" +// ]; +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// if(str){ +// +// str = encode_advanced.call(this, str, /* skip post-processing: */ true); +// +// if(str.length > 1){ +// +// //str = replace(str, pairs); +// //str = str.replace(regex_vowel, ""); +// str = str.charAt(0) + str.substring(1).replace(regex_vowel, ""); +// } +// +// if(str.length > 1){ +// +// str = collapse(str); +// } +// +// if(str){ +// +// str = str.split(" "); +// } +// } +// +// return str || []; +// } \ No newline at end of file diff --git a/dist/module-debug/lang/latin/simple.js b/dist/module-debug/lang/latin/simple.js index 1b9ca40..b01236e 100644 --- a/dist/module-debug/lang/latin/simple.js +++ b/dist/module-debug/lang/latin/simple.js @@ -1,45 +1,343 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline, normalize, regex_whitespace, regex } from "../../lang.js"; +import Encoder from "../../encoder.js"; -export const rtl = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ -/* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0 +}; - // Charset Normalization - -};const //regex_whitespace = /\W+/, -//regex_strip = regex("[^a-z0-9 ]"), -regex_a = regex("[àáâãäå]"), - regex_e = regex("[èéêë]"), - regex_i = regex("[ìíîï]"), - regex_o = regex("[òóôõöő]"), - regex_u = regex("[ùúûüű]"), - regex_y = regex("[ýŷÿ]"), - regex_n = regex("ñ"), - regex_c = regex("[çc]"), - regex_s = regex("ß"), - regex_and = regex(" & "), - pairs = [regex_a, "a", regex_e, "e", regex_i, "i", regex_o, "o", regex_u, "u", regex_y, "y", regex_n, "n", regex_c, "k", regex_s, "s", regex_and, " and " -//regex_whitespace, " " -//regex_strip, "" -]; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - str = "" + str; - - return pipeline.call(this, - /* string: */normalize(str).toLowerCase(), - /* normalize: */!str.normalize && pairs, - /* split: */regex_whitespace, !1); -} \ No newline at end of file +// import { pipeline, regex } from "../../lang.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // p{Z} = whitespaces +// // p{S} = symbols (emotes) +// // p{P} = special chars +// // p{C} = controls (linebreak, tabulator) +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// const normalize = "".normalize && /[\u0300-\u036f]/g; +// const pairs = !normalize && new Map([ +// +// // Charset Normalization +// // String.normalize("NFKD").replace(/[\u0300-\u036f]/g, "") +// +// ["ª","a"], +// ["²","2"], +// ["³","3"], +// ["¹","1"], +// ["º","o"], +// ["¼","1⁄4"], +// ["½","1⁄2"], +// ["¾","3⁄4"], +// ["à","a"], +// ["á","a"], +// ["â","a"], +// ["ã","a"], +// ["ä","a"], +// ["å","a"], +// ["ç","c"], +// ["è","e"], +// ["é","e"], +// ["ê","e"], +// ["ë","e"], +// ["ì","i"], +// ["í","i"], +// ["î","i"], +// ["ï","i"], +// ["ñ","n"], +// ["ò","o"], +// ["ó","o"], +// ["ô","o"], +// ["õ","o"], +// ["ö","o"], +// ["ù","u"], +// ["ú","u"], +// ["û","u"], +// ["ü","u"], +// ["ý","y"], +// ["ÿ","y"], +// ["ā","a"], +// ["ă","a"], +// ["ą","a"], +// ["ć","c"], +// ["ĉ","c"], +// ["ċ","c"], +// ["č","c"], +// ["ď","d"], +// ["ē","e"], +// ["ĕ","e"], +// ["ė","e"], +// ["ę","e"], +// ["ě","e"], +// ["ĝ","g"], +// ["ğ","g"], +// ["ġ","g"], +// ["ģ","g"], +// ["ĥ","h"], +// ["ĩ","i"], +// ["ī","i"], +// ["ĭ","i"], +// ["į","i"], +// ["ij","ij"], +// ["ĵ","j"], +// ["ķ","k"], +// ["ĺ","l"], +// ["ļ","l"], +// ["ľ","l"], +// ["ŀ","l"], +// ["ń","n"], +// ["ņ","n"], +// ["ň","n"], +// ["ʼn","n"], +// ["ō","o"], +// ["ŏ","o"], +// ["ő","o"], +// ["ŕ","r"], +// ["ŗ","r"], +// ["ř","r"], +// ["ś","s"], +// ["ŝ","s"], +// ["ş","s"], +// ["š","s"], +// ["ţ","t"], +// ["ť","t"], +// ["ũ","u"], +// ["ū","u"], +// ["ŭ","u"], +// ["ů","u"], +// ["ű","u"], +// ["ų","u"], +// ["ŵ","w"], +// ["ŷ","y"], +// ["ź","z"], +// ["ż","z"], +// ["ž","z"], +// ["ſ","s"], +// ["ơ","o"], +// ["ư","u"], +// ["dž","dz"], +// ["lj","lj"], +// ["nj","nj"], +// ["ǎ","a"], +// ["ǐ","i"], +// ["ǒ","o"], +// ["ǔ","u"], +// ["ǖ","u"], +// ["ǘ","u"], +// ["ǚ","u"], +// ["ǜ","u"], +// ["ǟ","a"], +// ["ǡ","a"], +// ["ǣ","ae"], +// ["æ","ae"], +// ["ǽ","ae"], +// ["ǧ","g"], +// ["ǩ","k"], +// ["ǫ","o"], +// ["ǭ","o"], +// ["ǯ","ʒ"], +// ["ǰ","j"], +// ["dz","dz"], +// ["ǵ","g"], +// ["ǹ","n"], +// ["ǻ","a"], +// ["ǿ","ø"], +// ["ȁ","a"], +// ["ȃ","a"], +// ["ȅ","e"], +// ["ȇ","e"], +// ["ȉ","i"], +// ["ȋ","i"], +// ["ȍ","o"], +// ["ȏ","o"], +// ["ȑ","r"], +// ["ȓ","r"], +// ["ȕ","u"], +// ["ȗ","u"], +// ["ș","s"], +// ["ț","t"], +// ["ȟ","h"], +// ["ȧ","a"], +// ["ȩ","e"], +// ["ȫ","o"], +// ["ȭ","o"], +// ["ȯ","o"], +// ["ȱ","o"], +// ["ȳ","y"], +// ["ʰ","h"], +// ["ʱ","h"], +// ["ɦ","h"], +// ["ʲ","j"], +// ["ʳ","r"], +// ["ʴ","ɹ"], +// ["ʵ","ɻ"], +// ["ʶ","ʁ"], +// ["ʷ","w"], +// ["ʸ","y"], +// ["ˠ","ɣ"], +// ["ˡ","l"], +// ["ˢ","s"], +// ["ˣ","x"], +// ["ˤ","ʕ"], +// ["ΐ","ι"], +// ["ά","α"], +// ["έ","ε"], +// ["ή","η"], +// ["ί","ι"], +// ["ΰ","υ"], +// ["ϊ","ι"], +// ["ϋ","υ"], +// ["ό","ο"], +// ["ύ","υ"], +// ["ώ","ω"], +// ["ϐ","β"], +// ["ϑ","θ"], +// ["ϒ","Υ"], +// ["ϓ","Υ"], +// ["ϔ","Υ"], +// ["ϕ","φ"], +// ["ϖ","π"], +// ["ϰ","κ"], +// ["ϱ","ρ"], +// ["ϲ","ς"], +// ["ϵ","ε"], +// ["й","и"], +// ["ѐ","е"], +// ["ё","е"], +// ["ѓ","г"], +// ["ї","і"], +// ["ќ","к"], +// ["ѝ","и"], +// ["ў","у"], +// ["ѷ","ѵ"], +// ["ӂ","ж"], +// ["ӑ","а"], +// ["ӓ","а"], +// ["ӗ","е"], +// ["ӛ","ә"], +// ["ӝ","ж"], +// ["ӟ","з"], +// ["ӣ","и"], +// ["ӥ","и"], +// ["ӧ","о"], +// ["ӫ","ө"], +// ["ӭ","э"], +// ["ӯ","у"], +// ["ӱ","у"], +// ["ӳ","у"], +// ["ӵ","ч"] +// +// // Charset Normalization +// +// // ["à", "a"], +// // ["á", "a"], +// // ["â", "a"], +// // ["ã", "a"], +// // ["ä", "a"], +// // ["å", "a"], +// // +// // ["è", "e"], +// // ["é", "e"], +// // ["ê", "e"], +// // ["ë", "e"], +// // +// // ["ì", "i"], +// // ["í", "i"], +// // ["î", "i"], +// // ["ï", "i"], +// // +// // ["ò", "o"], +// // ["ó", "o"], +// // ["ô", "o"], +// // ["õ", "o"], +// // ["ö", "o"], +// // ["ő", "o"], +// // +// // ["ù", "u"], +// // ["ú", "u"], +// // ["û", "u"], +// // ["ü", "u"], +// // ["ű", "u"], +// // +// // ["ý", "y"], +// // ["ŷ", "y"], +// // ["ÿ", "y"], +// // +// // ["ñ", "n"], +// // ["ç", "c"], +// // ["ß", "s"] +// +// // Special Chars Removal +// +// // [",", ""], +// // [".", ""], +// // ["'", ""] +// +// // Non-Whitespace Separators +// +// // split by default p{P} +// // ["-", " "], +// // [":", " "], +// // ["_", " "], +// // ["|", " "], +// // ["/", " "], +// // ["\\", " "] +// ]); +// +// // let pairs; +// // +// // if(!normalize){ +// // +// // // Charset Normalization +// // +// // const regex_a = regex("[àáâãäå]"), +// // regex_e = regex("[èéêë]"), +// // regex_i = regex("[ìíîï]"), +// // regex_o = regex("[òóôõöő]"), +// // regex_u = regex("[ùúûüű]"), +// // regex_y = regex("[ýŷÿ]"), +// // regex_n = regex("ñ"), +// // //regex_c = regex("[çc]"), +// // regex_c = regex("ç"), +// // regex_s = regex("ß"), +// // //regex_and = regex(" & "), +// // regex_and = regex("&"); +// // +// // pairs = [ +// // regex_a, "a", +// // regex_e, "e", +// // regex_i, "i", +// // regex_o, "o", +// // regex_u, "u", +// // regex_y, "y", +// // regex_n, "n", +// // //regex_c, "k", +// // regex_c, "c", +// // regex_s, "s", +// // regex_and, " and " +// // ]; +// // } +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// //str = "" + str; +// +// return pipeline.call( +// +// this, +// /* string: */ (/*normalize ? str.normalize("NFD").replace(normalize, "") :*/ "" + str).toLowerCase(), +// /* normalize: */ pairs, +// /* split: */ whitespace, +// ///* collapse: */ false +// ); +// } \ No newline at end of file diff --git a/dist/module-debug/lang/latin/soundex.js b/dist/module-debug/lang/latin/soundex.js new file mode 100644 index 0000000..0a38309 --- /dev/null +++ b/dist/module-debug/lang/latin/soundex.js @@ -0,0 +1,52 @@ +export default { + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: /* suggest */ /* append: */ /* enrich */!1, + include: { + letter: !0 + }, + finalize: function (arr) { + for (let i = 0; i < arr.length; i++) { + arr[i] = soundex(arr[i]); + } + } +}; + +const codes = { + a: "", e: "", i: "", o: "", u: "", y: "", + b: 1, f: 1, p: 1, v: 1, + c: 2, g: 2, j: 2, k: 2, q: 2, s: 2, x: 2, z: 2, ß: 2, + d: 3, t: 3, + l: 4, + m: 5, n: 5, + r: 6 +}; + +function soundex(stringToEncode) { + let encodedString = stringToEncode.charAt(0), + last = codes[encodedString]; + + for (let i = 1, char; i < stringToEncode.length; i++) { + char = stringToEncode.charAt(i); + // Remove all occurrences of "h" and "w" + if ("h" !== char && "w" !== char) { + // Replace all consonants with digits + char = codes[char]; + // Remove all occurrences of a,e,i,o,u,y except first letter + if (char) { + // Replace all adjacent same digits with one digit + if (char !== last) { + encodedString += char; + last = char; + if (4 === encodedString.length) { + break; + } + } + } + } + } + // while(encodedString.length < 4){ + // encodedString += "0"; + // } + return encodedString; +} \ No newline at end of file diff --git a/dist/module-debug/polyfill.js b/dist/module-debug/polyfill.js deleted file mode 100644 index e8c7408..0000000 --- a/dist/module-debug/polyfill.js +++ /dev/null @@ -1,74 +0,0 @@ - -export let promise = Promise; - -Object.assign || (Object.assign = function () { - const args = arguments, - size = args.length, - obj = args[0]; - - - for (let x = 1, current, keys, length; x < size; x++) { - - current = args[x]; - keys = Object.keys(current); - length = keys.length; - - for (let i = 0, key; i < length; i++) { - - key = keys[i]; - obj[key] = current[key]; - } - } - - return obj; -}); - -// Object.values || (Object.values = function(obj){ -// -// const keys = Object.keys(obj); -// const length = keys.length; -// const values = new Array(length); -// -// for(let x = 0; x < length; x++){ -// -// values[x] = obj[keys[x]]; -// } -// -// return values; -// }); - -if (!promise) { - - /** - * @param {Function} fn - * @constructor - */ - - function SimplePromise(fn) { - - this.callback = null; - - const self = this; - - fn(function (val) { - - if (self.callback) { - - self.callback(val); - // self.callback = null; - // self = null; - } - }); - } - - /** - * @param {Function} callback - */ - - SimplePromise.prototype.then = function (callback) { - - this.callback = callback; - }; - - promise = SimplePromise; -} \ No newline at end of file diff --git a/dist/module-debug/preset.js b/dist/module-debug/preset.js index 0ea59f9..1d68bb7 100644 --- a/dist/module-debug/preset.js +++ b/dist/module-debug/preset.js @@ -1,96 +1,56 @@ import { is_string } from "./common.js"; +import { IndexOptions } from "./type.js"; /** * @enum {Object} * @const */ -const preset = { +const presets = { memory: { - charset: "latin:extra", - //tokenize: "strict", - resolution: 3, - //threshold: 0, - minlength: 4, - fastupdate: /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ - /* collapse: */ - /* collapse: */!1 + resolution: 1 }, performance: { - //charset: "latin", - //tokenize: "strict", - resolution: 3, - minlength: 3, - //fastupdate: true, - optimize: !1, //fastupdate: true, + resolution: 6, + fastupdate: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, context: { - depth: 2, resolution: 1 - //bidirectional: false + depth: 1, + resolution: 3 } }, match: { - charset: "latin:extra", - tokenize: "reverse" - //resolution: 9, - //threshold: 0 + tokenize: "forward" }, score: { - charset: "latin:advanced", - //tokenize: "strict", - resolution: 20, - minlength: 3, + resolution: 9, context: { - depth: 3, + depth: 2, resolution: 9 - //bidirectional: true } - }, - - default: { - // charset: "latin:default", - // tokenize: "strict", - // resolution: 3, - // threshold: 0, - // depth: 3 } - - // "fast": { - // //charset: "latin", - // //tokenize: "strict", - // threshold: 8, - // resolution: 9, - // depth: 1 - // } }; +/** + * + * @param {!IndexOptions|string} options + * @return {IndexOptions} + */ + export default function apply_preset(options) { - if (is_string(options)) { + const preset = is_string(options) ? options : options.preset; - if (!preset[options]) { - - console.warn("Preset not found: " + options); - } - - options = preset[options]; - } else { - - const preset = options.preset; - - if (preset) { - - if (!preset[preset]) { - - console.warn("Preset not found: " + preset); - } - - options = Object.assign({}, preset[preset], /** @type {Object} */options); + if (preset) { + if (!presets[preset]) { + console.warn("Preset not found: " + preset); } + options = Object.assign({}, presets[preset], /** @type {Object} */options); } return options; diff --git a/dist/module-debug/resolve/and.js b/dist/module-debug/resolve/and.js new file mode 100644 index 0000000..9e7994f --- /dev/null +++ b/dist/module-debug/resolve/and.js @@ -0,0 +1,242 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object, get_max_len } from "../common.js"; +// import xor from "./xor.js"; +// import or from "./or.js"; +// import not from "./not.js"; + +Resolver.prototype.and = function () { + if (this.result.length) { + + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.and.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.and.apply(this, first_argument); + } + } + + // for(let i = 0; i < args.length; i++){ + // if(args[i].result instanceof Promise){ + // return; + // } + // } + + // if(args.length < 2){ + // if(first_argument.index){ + // first_argument.resolve = false; + // return first_argument.index.search(first_argument); + // } + // } + + let final = [], + promises = [], + limit = 0, + offset = 0, + enrich, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.or) { + result = this.or(query.or); + } else if (query.xor) { + result = this.xor(query.xor); + } else if (query.not) { + result = this.not(query.not); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + final = [self.result].concat(final); + self.result = intersect(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + final = [this.result].concat(final); + this.result = intersect(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? this.result : this; + } + return this; +}; + +/** + * Aggregate the intersection of N raw results + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function intersect(result, limit, offset, enrich, resolve, boost) { + + // if(!result.length){ + // // todo remove + // console.log("Empty Result") + // return result; + // } + + if (2 > result.length) { + // todo remove + //console.log("Single Result") + return []; + // if(resolve){ + // return default_resolver(result[0], limit, offset, enrich); + // } + // else{ + // return result[0]; + // } + } + + let final = [], + count = 0, + contain = create_object(), + maxres = get_max_len(result); + + // fast path single slot + // if(result.length < 2){ + // if(limit || offset){ + // let res = result[0]; + // for(let j = 0, ids; j < res.length; j++){ + // ids = res[j]; + // if(!ids) continue; + // for(let k = 0, id; k < ids.length; k++){ + // id = ids[k]; + // if(offset){ + // offset--; + // continue; + // } + // if(resolve){ + // final.push(id); + // } + // else{ + // final[j + this.boost] || (final[j + this.boost] = []); + // final[j + this.boost].push(id); + // } + // if(limit && ++count === limit){ + // this.boost = 0; + // return final; + // } + // } + // } + // } + // this.boost = 0; + // return result[0]; + // } + + if (!maxres) return final; + + // for(let j = 0, ids, res = result[0]; j < res.length; j++){ + // ids = res[j]; + // for(let k = 0; k < ids.length; k++){ + // contain[ids[k]] = 1; + // } + // } + + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res || !res.length) return []; + let contain_new = create_object(), + match = 0, + last_round = i === result.length - 1; + + + for (let j = 0, ids; j < maxres; j++) { + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id, min; k < ids.length; k++) { + id = ids[k]; + // fill in first round + if (!i) { + // shift resolution +1 + // shift resolution by boost (inverse) + contain_new[id] = j + 1 + (i ? boost : 0); + match = 1; + } + // result in last round + else if (last_round) { + if (min = contain[id]) { + match = 1; + //if(!contain_new[id]){ + if (offset) { + offset--; + continue; + } + if (resolve) { + final.push(id); + } else { + // reduce resolution -1 + min--; + if (j < min) min = j; + final[min] || (final[min] = []); + final[min].push(id); + } + if (limit && ++count === limit) { + //this.boost = 0; + return final; + } + // shift resolution +1 + //contain_new[id] = min + 1; + //} + } + } + // check for intersection + else if (min = contain[id]) { + // shift resolution +1 + if (j + 1 < min) min = j + 1; + contain_new[id] = min; + match = 1; + } + } + } + + if (!match) { + //this.boost = 0; + return []; + } + + contain = contain_new; + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/dist/module-debug/resolve/default.js b/dist/module-debug/resolve/default.js new file mode 100644 index 0000000..2edeb6d --- /dev/null +++ b/dist/module-debug/resolve/default.js @@ -0,0 +1,98 @@ +import { concat } from "../common.js"; + +/* + from -> res[score][id] + to -> [id] +*/ + +/** + * Aggregate the union of a single raw result + * @param {!Array} result + * @param {!number} limit + * @param {number=} offset + * @param {boolean=} enrich + * @return Array + */ + +export default function (result, limit, offset, enrich) { + + // fast path: when there is just one slot in the result + if (1 === result.length) { + result = result[0]; + result = offset || result.length > limit ? limit ? result.slice(offset, offset + limit) : result.slice(offset) : result; + return enrich ? enrich_result(result) : result; + } + + // this is an optimized workaround instead of + // just doing result = concat(result) + + let final = []; + + for (let i = 0, arr, len; i < result.length; i++) { + if (!(arr = result[i]) || !(len = arr.length)) continue; + + if (offset) { + // forward offset pointer + if (offset >= len) { + offset -= len; + continue; + } + // apply offset pointer when length differs + if (offset < len) { + arr = limit ? arr.slice(offset, offset + limit) : arr.slice(offset); + len = arr.length; + offset = 0; + } + } + + if (!final.length) { + // fast path: when limit was reached in first slot + if (len >= limit) { + if (len > limit) { + arr = arr.slice(0, limit); + } + return enrich ? enrich_result(arr) : arr; + } + final = [arr]; + } else { + if (len > limit) { + arr = arr.slice(0, limit); + len = arr.length; + } + final.push(arr); + } + + // reduce limit + limit -= len; + + // todo remove + // if(limit < 0){ + // throw new Error("Impl.Error"); + // } + + // break if limit was reached + if (!limit) { + break; + } + } + + // todo remove + if (!final.length) { + //throw new Error("No results found"); + return final; + } + + final = 1 < final.length ? concat(final) : final[0]; + + return enrich ? enrich_result(final) : final; +} + +function enrich_result(ids) { + for (let i = 0; i < ids.length; i++) { + ids[i] = { + score: i, + id: ids[i] + }; + } + return ids; +} \ No newline at end of file diff --git a/dist/module-debug/resolve/not.js b/dist/module-debug/resolve/not.js new file mode 100644 index 0000000..13364fb --- /dev/null +++ b/dist/module-debug/resolve/not.js @@ -0,0 +1,115 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object } from "../common.js"; +// import or from "./or.js"; +// import and from "./and.js"; +// import xor from "./xor.js"; + +Resolver.prototype.not = function () { + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.not.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.not.apply(this, first_argument); + } + } + + let final = [], + promises = [], + limit = 0, + offset = 0, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.or) { + result = this.or(query.or); + } else if (query.and) { + result = this.and(query.and); + } else if (query.xor) { + result = this.xor(query.xor); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + self.result = exclusion.call(self, final, limit, offset, resolve); + return resolve ? self.result : self; + }); + } + + this.result = exclusion.call(this, final, limit, offset, resolve); + return resolve ? this.result : this; +}; + +/** + * @param result + * @param limit + * @param offset + * @param resolve + * @this Resolver + * @return {Array} + */ + +function exclusion(result, limit, offset, resolve) { + + if (!result.length) { + return this.result; + } + + const final = [], + exclude = new Set(result.flat().flat()); + + + for (let j = 0, ids; j < this.result.length; j++) { + ids = this.result[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + if (!exclude.has(id)) { + if (resolve) { + final.push(id); + } else { + final[j] || (final[j] = []); + final[j].push(id); + } + } + } + } + + return final; +} \ No newline at end of file diff --git a/dist/module-debug/resolve/or.js b/dist/module-debug/resolve/or.js new file mode 100644 index 0000000..d14981c --- /dev/null +++ b/dist/module-debug/resolve/or.js @@ -0,0 +1,182 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object, get_max_len } from "../common.js"; +// import xor from "./xor.js"; +// import and from "./and.js"; +// import not from "./not.js"; + +Resolver.prototype.or = function () { + + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.or.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.or.apply(this, first_argument); + } + } + + // for(let i = 0; i < args.length; i++){ + // if(args[i].result instanceof Promise){ + // return; + // } + // } + + // if(args.length < 2){ + // if(first_argument.index){ + // first_argument.resolve = false; + // const result = first_argument.index.search(first_argument); + // if(result instanceof Promise){ + // result.then(function(result){ + // final = self.result.concat(result); + // self.result = resolver(final, limit, offset, enrich, !resolve); + // return resolve ? self.result : self; + // }); + // } + // else{ + // final = this.result.concat(result); + // this.result = resolver(final, limit, offset, enrich, !resolve); + // return resolve ? this.result : this; + // } + // } + // } + + let final = [], + promises = [], + limit = 0, + offset = 0, + enrich, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.and) { + result = this.and(query.and); + } else if (query.xor) { + result = this.xor(query.xor); + } else if (query.not) { + result = this.not(query.not); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + self.result.length && (final = [self.result].concat(final)); + self.result = resolver(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + this.result.length && (final = [this.result].concat(final)); + this.result = resolver(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? this.result : this; +}; + +/** + * Aggregate the union of N raw results + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function resolver(result, limit, offset, enrich, resolve, boost) { + + if (!result.length) { + // todo remove + //console.log("Empty Result") + return result; + } + + if ("object" == typeof limit) { + offset = limit.offset || 0; + enrich = limit.enrich || !1; + limit = limit.limit || 0; + } + + if (2 > result.length) { + // todo remove + //console.log("Single Result") + if (resolve) { + return default_resolver(result[0], limit, offset, enrich); + } else { + return result[0]; + } + } + + let final = [], + count = 0, + dupe = create_object(), + maxres = get_max_len(result); + + + for (let j = 0, ids; j < maxres; j++) { + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res) continue; + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + if (!dupe[id]) { + dupe[id] = 1; + if (offset) { + offset--; + continue; + } + if (resolve) { + final.push(id); + } else { + // shift resolution by boost (inverse) + const index = j + (i ? boost : 0); + final[index] || (final[index] = []); + final[index].push(id); + } + if (limit && ++count === limit) { + //this.boost = 0; + return final; + } + } + } + } + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/dist/module-debug/resolve/xor.js b/dist/module-debug/resolve/xor.js new file mode 100644 index 0000000..1eed9e8 --- /dev/null +++ b/dist/module-debug/resolve/xor.js @@ -0,0 +1,154 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object } from "../common.js"; +// import or from "./or.js"; +// import and from "./and.js"; +// import not from "./not.js"; + +Resolver.prototype.xor = function () { + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.xor.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.xor.apply(this, first_argument); + } + } + + let final = [], + promises = [], + limit = 0, + offset = 0, + enrich, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.or) { + result = this.or(query.or); + } else if (query.and) { + result = this.and(query.and); + } else if (query.not) { + result = this.not(query.not); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + self.result.length && (final = [self.result].concat(final)); + self.result = exclusive(final, limit, offset, enrich, !resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + this.result.length && (final = [this.result].concat(final)); + this.result = exclusive(final, limit, offset, enrich, !resolve, self.boostval); + return resolve ? this.result : this; +}; + +/** + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function exclusive(result, limit, offset, enrich, resolve, boost) { + + if (!result.length) { + // todo remove + //console.log("Empty Result") + return result; + } + + if (2 > result.length) { + // todo remove + //console.log("Single Result") + if (resolve) { + return default_resolver(result[0], limit, offset, enrich); + } else { + return result[0]; + } + } + + const final = [], + check = create_object(); + + + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res) continue; + + for (let j = 0, ids; j < res.length; j++) { + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + check[id] ? check[id]++ : check[id] = 1; + } + } + } + + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res) continue; + + for (let j = 0, ids; j < res.length; j++) { + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + if (1 === check[id]) { + if (resolve) { + final.push(id); + } else { + // shift resolution by boost (inverse) + const index = j + (i ? boost : 0); + final[index] || (final[index] = []); + final[index].push(id); + } + } + } + } + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/dist/module-debug/resolver.js b/dist/module-debug/resolver.js new file mode 100644 index 0000000..fbd1bf7 --- /dev/null +++ b/dist/module-debug/resolver.js @@ -0,0 +1,99 @@ +import default_resolver from "./resolve/default.js"; +import { set_resolve } from "./index/search.js"; +// import or from "./resolve/or.js"; +// import and from "./resolve/and.js"; +// import xor from "./resolve/xor.js"; +// import not from "./resolve/not.js"; +import "./resolve/or.js"; +import "./resolve/and.js"; +import "./resolve/xor.js"; +import "./resolve/not.js"; + +/** + * @param result + * @constructor + */ + +export default function Resolver(result) { + if (result && result.index) { + result.resolve = /* suggest */ /* append: */ /* enrich */!1; + this.index = result.index; + return result.index.search(result); + } + if (!(this instanceof Resolver)) { + return new Resolver(result); + } + if (result instanceof Resolver) { + // todo remove + //console.log("Resolver Loopback") + return result; + } + this.index = null; + this.result = result || []; + this.boostval = 0; +} + +// Resolver.prototype.or = or; +// Resolver.prototype.and = and; +// Resolver.prototype.not = not; +// Resolver.prototype.xor = xor; + +Resolver.prototype.limit = function (limit) { + if (this.result.length) { + const final = []; + let count = 0; + for (let j = 0, ids; j < this.result.length; j++) { + ids = this.result[j]; + if (ids.length + count < limit) { + final[j] = ids; + count += ids.length; + } else { + final[j] = ids.slice(0, limit - count); + this.result = final; + break; + } + } + } + return this; +}; + +Resolver.prototype.offset = function (offset) { + if (this.result.length) { + const final = []; + let count = 0; + for (let j = 0, ids; j < this.result.length; j++) { + ids = this.result[j]; + if (ids.length + count < offset) { + count += ids.length; + } else { + final[j] = ids.slice(offset - count); + count = offset; + } + } + this.result = final; + } + return this; +}; + +Resolver.prototype.boost = function (boost) { + this.boostval += boost; + return this; +}; + +Resolver.prototype.resolve = function (limit, offset, enrich) { + set_resolve(1); + const result = this.result; + this.index = null; + this.result = null; + + if (result.length) { + if ("object" == typeof limit) { + enrich = limit.enrich; + offset = limit.offset; + limit = limit.limit; + } + return default_resolver(result, limit || 100, offset, enrich); + } + + return result; +}; \ No newline at end of file diff --git a/dist/module-debug/serialize.js b/dist/module-debug/serialize.js index 0baab85..66ad30b 100644 --- a/dist/module-debug/serialize.js +++ b/dist/module-debug/serialize.js @@ -1,36 +1,44 @@ // TODO return promises instead of inner await -import { IndexInterface, DocumentInterface } from "./type.js"; +import Index from "./index.js"; +import Document from "./document.js"; import { create_object, is_string } from "./common.js"; function async(callback, self, field, key, index_doc, index, data, on_done) { - setTimeout(function () { + //setTimeout(function(){ - const res = callback(field ? field + "." + key : key, JSON.stringify(data)); + const res = callback(field ? field + "." + key : key, JSON.stringify(data)); - // await isn't supported by ES5 + // await isn't supported by ES5 - if (res && res.then) { + if (res && res.then) { - res.then(function () { - - self.export(callback, self, field, index_doc, index + 1, on_done); - }); - } else { + res.then(function () { self.export(callback, self, field, index_doc, index + 1, on_done); - } - }); + }); + } else { + + self.export(callback, self, field, index_doc, index + 1, on_done); + } + //}); } /** - * @this IndexInterface + * @param callback + * @param self + * @param field + * @param index_doc + * @param index + * @param on_done + * @this {Index|Document} */ export function exportIndex(callback, self, field, index_doc, index, on_done) { - let return_value = /* append: */ /* skip update: */ /* skip_update: */ /* skip post-processing: */!0; + let return_value = /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; if ('undefined' == typeof on_done) { return_value = new Promise(resolve => { on_done = resolve; @@ -51,13 +59,13 @@ export function exportIndex(callback, self, field, index_doc, index, on_done) { data = create_object(); - for (let key in this.register) { + for (let key of this.reg.keys()) { data[key] = 1; } } else { - data = this.register; + data = this.reg; } break; @@ -100,7 +108,7 @@ export function exportIndex(callback, self, field, index_doc, index, on_done) { } /** - * @this IndexInterface + * @this Index */ export function importIndex(key, data) { @@ -126,8 +134,8 @@ export function importIndex(key, data) { // fastupdate isn't supported by import - this.fastupdate = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* collapse: */!1; - this.register = data; + this.fastupdate = /* suggest */ /* append: */ /* enrich */!1; + this.reg = data; break; case "map": @@ -143,7 +151,7 @@ export function importIndex(key, data) { } /** - * @this DocumentInterface + * @this Document */ export function exportDocument(callback, self, field, index_doc, index, on_done) { @@ -165,16 +173,16 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) self = this; - setTimeout(function () { + //setTimeout(function(){ - if (!idx.export(callback, self, index ? field /*.replace(":", "-")*/ : "", index_doc, index++, on_done)) { + if (!idx.export(callback, self, index ? field /*.replace(":", "-")*/ : "", index_doc, index++, on_done)) { - index_doc++; - index = 1; + index_doc++; + index = 1; - self.export(callback, self, field, index_doc, index, on_done); - } - }); + self.export(callback, self, field, index_doc, index, on_done); + } + //}); } else { let key, data; @@ -214,7 +222,7 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) } /** - * @this DocumentInterface + * @this Document */ export function importDocument(key, data) { @@ -241,12 +249,12 @@ export function importDocument(key, data) { // fastupdate isn't supported by import this.fastupdate = !1; - this.register = data; + this.reg = data; for (let i = 0, index; i < this.field.length; i++) { index = this.index[this.field[i]]; - index.register = data; + index.reg = data; index.fastupdate = !1; } diff --git a/dist/module-debug/type.js b/dist/module-debug/type.js index 1a784b3..0c377c0 100644 --- a/dist/module-debug/type.js +++ b/dist/module-debug/type.js @@ -1,69 +1,144 @@ -/** - * @interface - */ - -export function IndexInterface() { - - this.cache = null; - this.matcher = null; - this.stemmer = null; - this.filter = null; -} +// When you are looking for type definitions which fully describes the usage take a look into the index.d.ts file. +// Some of the types here aren't supposed to be used as public, they might be defined just for internal state. +import Encoder from "./encoder.js"; +import StorageInterface from "./db/interface.js"; /** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} collapse - * @returns {string|Array} + * @typedef IndexOptions {{ + * preset: string|undefined, + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * resolve: [boolean=true], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -//IndexInterface.prototype.pipeline; +export let IndexOptions; /** - * @param {!number|string} id - * @param {!string} content + * @typedef DocumentOptions {{ + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * db: StorageInterface|undefined, + * doc: DocumentDescriptor|Array|undefined, + * document: DocumentDescriptor|Array|undefined, + * worker: boolean|string|undefined + * }} */ - -IndexInterface.prototype.add; +export let DocumentOptions; /** - * @param {!number|string} id - * @param {!string} content + * @typedef ContextOptions {{ + * depth: number, + * bidirectional: boolean|undefined, + * resolution: number|undefined + * }} */ - -IndexInterface.prototype.append; +export let ContextOptions; /** - * @param {!string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @returns {Array} + * @typedef DocumentDescriptor {{ + * field: FieldOptions|Array|undefined, + * index: FieldOptions|Array|undefined, + * tag: TagOptions|Array|undefined, + * store: StoreOptions|Array|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.search; +export let DocumentDescriptor; /** - * @param {!number|string} id - * @param {!string} content + * @typedef FieldOptions {{ + * field: string, + * filter: Function|undefined, + * custom: Function|undefined, + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.update; +export let FieldOptions; /** - * @param {!number|string} id + * @typedef TagOptions {{ + * field: string, + * tag: Object>|Array|string, + * filter: Function|undefined, + * custom: Function|undefined, + * keystore: [number=0], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.remove; +export let TagOptions; /** - * @interface + * @typedef StoreOptions {{ + * field: string, + * filter: Function|undefined, + * custom: Function|undefined, + * keystore: [number=0], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ +export let StoreOptions; -export function DocumentInterface() { +/** + * @typedef SearchOptions {{ + * query: string=, + * limit: [number=100], + * offset: [number=0], + * context: boolean|undefined, + * suggest: [boolean=false], + * resolve: [boolean=true], + * enrich: [boolean=false], + * tag: Array|undefined + * }} + */ +export let SearchOptions; - this.field = null; +/** + * @typedef DocumentSearchOptions {{ + * query: string=, + * limit: [number=100], + * offset: [number=0], + * context: boolean|undefined, + * suggest: [boolean=false], + * enrich: [boolean=false], + * tag: Array|undefined, + * field: FieldOptions|Array|undefined, + * index: FieldOptions|Array|undefined, + * pluck: boolean|undefined, + * merge: [boolean=false] + * }} + */ +export let DocumentSearchOptions; - /** @type IndexInterface */ - this.index = null; -} \ No newline at end of file +export let EncoderOptions; +export let ResolverOptions; \ No newline at end of file diff --git a/dist/module-debug/webpack.js b/dist/module-debug/webpack.js new file mode 100644 index 0000000..723eaee --- /dev/null +++ b/dist/module-debug/webpack.js @@ -0,0 +1,173 @@ + +import { SearchOptions, ContextOptions, DocumentDescriptor, DocumentSearchOptions, FieldOptions, IndexOptions, DocumentOptions } from "./type.js"; +import Document from "./document.js"; +import Index from "./index.js"; +import WorkerIndex from "./worker/index.js"; +import Resolver from "./resolver.js"; +import Encoder from "./encoder.js"; +import IdxDB from "./db/indexeddb/index.js"; +import { global_charset, global_lang } from "./global.js"; +import charset_exact from "./lang/latin/exact.js"; +import charset_default from "./lang/latin/default.js"; +import charset_simple from "./lang/latin/simple.js"; +import charset_balance from "./lang/latin/balance.js"; +import charset_advanced from "./lang/latin/advanced.js"; +import charset_extra from "./lang/latin/extra.js"; +import charset_soundex from "./lang/latin/soundex.js"; + +/** @export */Index.prototype.add; +/** @export */Index.prototype.append; +/** @export */Index.prototype.search; +/** @export */Index.prototype.update; +/** @export */Index.prototype.remove; +/** @export */Index.prototype.contain; +/** @export */Index.prototype.clear; +/** @export */Index.prototype.cleanup; + +/** @export */Document.prototype.add; +/** @export */Document.prototype.append; +/** @export */Document.prototype.search; +/** @export */Document.prototype.update; +/** @export */Document.prototype.remove; +/** @export */Document.prototype.contain; +/** @export */Document.prototype.clear; +/** @export */Document.prototype.cleanup; + +/** @export */Document.prototype.get; +/** @export */Document.prototype.set; + +/** @export */Index.prototype.searchCache; + +/** @export */Document.prototype.searchCache; + +/** @export */Index.prototype.addAsync; +/** @export */Index.prototype.appendAsync; +/** @export */Index.prototype.searchAsync; +/** @export */Index.prototype.updateAsync; +/** @export */Index.prototype.removeAsync; + +/** @export */Document.prototype.addAsync; +/** @export */Document.prototype.appendAsync; +/** @export */Document.prototype.searchAsync; +/** @export */Document.prototype.updateAsync; +/** @export */Document.prototype.removeAsync; + +/** @export */Index.prototype.export; +/** @export */Index.prototype.import; + +/** @export */Document.prototype.export; +/** @export */Document.prototype.import; + +/** @export */Index.prototype.mount; +/** @export */Index.prototype.commit; +/** @export */Index.db; + +/** @export */Document.prototype.mount; +/** @export */Document.prototype.commit; +/** @export */Document.db; + +/** @export */IndexOptions.preset; +/** @export */IndexOptions.context; +/** @export */IndexOptions.encoder; +/** @export */IndexOptions.encode; +/** @export */IndexOptions.resolution; +/** @export */IndexOptions.tokenize; +/** @export */IndexOptions.fastupdate; +/** @export */IndexOptions.score; +/** @export */IndexOptions.keystore; +/** @export */IndexOptions.rtl; +/** @export */IndexOptions.cache; +/** @export */IndexOptions.resolve; +/** @export */IndexOptions.db; + +/** @export */DocumentOptions.context; +/** @export */DocumentOptions.encoder; +/** @export */DocumentOptions.encode; +/** @export */DocumentOptions.resolution; +/** @export */DocumentOptions.tokenize; +/** @export */DocumentOptions.fastupdate; +/** @export */DocumentOptions.score; +/** @export */DocumentOptions.keystore; +/** @export */DocumentOptions.rtl; +/** @export */DocumentOptions.cache; +/** @export */DocumentOptions.db; +/** @export */DocumentOptions.doc; +/** @export */DocumentOptions.document; +/** @export */DocumentOptions.worker; + +/** @export */DocumentDescriptor.field; +/** @export */DocumentDescriptor.index; +/** @export */DocumentDescriptor.tag; +/** @export */DocumentDescriptor.store; + +/** @export */ContextOptions.depth; +/** @export */ContextOptions.bidirectional; +/** @export */ContextOptions.resolution; + +/** @export */SearchOptions.query; +/** @export */SearchOptions.limit; +/** @export */SearchOptions.offset; +/** @export */SearchOptions.context; +/** @export */SearchOptions.suggest; +/** @export */SearchOptions.resolve; +/** @export */SearchOptions.enrich; +/** @export */SearchOptions.tag; + +/** @export */DocumentSearchOptions.query; +/** @export */DocumentSearchOptions.limit; +/** @export */DocumentSearchOptions.offset; +/** @export */DocumentSearchOptions.context; +/** @export */DocumentSearchOptions.suggest; +/** @export */DocumentSearchOptions.enrich; +/** @export */DocumentSearchOptions.tag; +/** @export */DocumentSearchOptions.field; +/** @export */DocumentSearchOptions.index; +/** @export */DocumentSearchOptions.pluck; +/** @export */DocumentSearchOptions.merge; + +global_charset["latin:exact"] = charset_exact; +global_charset["latin:default"] = charset_default; +global_charset["latin:simple"] = charset_simple; +global_charset["latin:balance"] = charset_balance; +global_charset["latin:advanced"] = charset_advanced; +global_charset["latin:extra"] = charset_extra; +global_charset["latin:soundex"] = charset_soundex; + + +const FlexSearch = { + Index: Index, + Encoder: Encoder, + Charset: global_charset, + Language: global_lang, + Document: Document, + Worker: WorkerIndex, + Resolver: Resolver, + IndexedDB: IdxDB + //"registerCharset": registerCharset, + //"registerLanguage": registerLanguage +}; + +// Export as library (Bundle) +// -------------------------------- + +{ + + const root = self; + let prop; + + // AMD (RequireJS) + if ((prop = root.define) && prop.amd) { + prop([], function () { + return FlexSearch; + }); + } + // CommonJS + else if ("object" == typeof root.exports) { + root.exports = FlexSearch; + } + // Global (window) + else { + /** @export */ + root.FlexSearch = FlexSearch; + } +} \ No newline at end of file diff --git a/dist/module-debug/worker/handler.js b/dist/module-debug/worker/handler.js index 914439c..0ada932 100644 --- a/dist/module-debug/worker/handler.js +++ b/dist/module-debug/worker/handler.js @@ -1,6 +1,7 @@ import Index from "../index.js"; +import { IndexOptions } from "../type.js"; -export default function (data) { +export default (async function (data) { data = data.data; @@ -13,19 +14,26 @@ export default function (data) { switch (task) { case "init": - const options = data.options || {}, - factory = data.factory, - encode = options.encode; + /** @type IndexOptions */ + let options = data.options || {}, + filepath = options.config; - options.cache = /* normalize: */ /* collapse: */ /* normalize: */ - - /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* collapse: */!1; - - if (encode && 0 === encode.indexOf("function")) { - options.encode = Function("return " + encode)(); + if (filepath) { + options = filepath; + // will be replaced after build with the line below because + // there is an issue with closure compiler dynamic import + //options = await import(filepath); } + // deprecated: + // const encode = options.encode; + // if(encode && (encode.indexOf("function") === 0)){ + // options.encode = Function("return " + encode)(); + // } + + const factory = data.factory; + if (factory) { // export the FlexSearch global payload to "self" @@ -41,6 +49,7 @@ export default function (data) { self._index = new Index(options); } + postMessage({ id: data.id }); break; default: @@ -49,4 +58,4 @@ export default function (data) { postMessage("search" === task ? { id: id, msg: message } : { id: id }); } -} \ No newline at end of file +}); \ No newline at end of file diff --git a/dist/module-debug/worker/index.js b/dist/module-debug/worker/index.js index 5f42c56..8d7df5e 100644 --- a/dist/module-debug/worker/index.js +++ b/dist/module-debug/worker/index.js @@ -1,31 +1,22 @@ //import { promise as Promise } from "../polyfill.js"; +import { IndexOptions } from "../type.js"; import { create_object, is_function, is_object, is_string } from "../common.js"; import handler from "./handler.js"; let pid = 0; /** - * @param {Object=} options + * @param {IndexOptions=} options * @constructor */ function WorkerIndex(options) { if (!(this instanceof WorkerIndex)) { - return new WorkerIndex(options); } - let opt; - - if (options) { - - if (is_function(opt = options.encode)) { - - options.encode = opt.toString(); - } - } else { - + if (!options) { options = {}; } @@ -33,9 +24,7 @@ function WorkerIndex(options) { // we use "self" as a trap for node.js let factory = (self || window)._factory; - if (factory) { - factory = factory.toString(); } @@ -50,25 +39,37 @@ function WorkerIndex(options) { return; } - if (is_node_js) { + function onmessage(msg) { + msg = msg.data || msg; + const id = msg.id, + res = id && _self.resolver[id]; - this.worker.on("message", function (msg) { + if (res) { + res(msg.msg); + delete _self.resolver[id]; + } + } - _self.resolver[msg.id](msg.msg); - delete _self.resolver[msg.id]; + is_node_js ? this.worker.on("message", onmessage) : this.worker.onmessage = onmessage; + + if (options.config) { + + // when extern configuration needs to be loaded + // it needs to return a promise to await for + return new Promise(function (resolve) { + _self.resolver[++pid] = function () { + resolve(_self); + }; + _self.worker.postMessage({ + id: pid, + task: "init", + factory: factory, + options: options + }); }); - } else { - - this.worker.onmessage = function (msg) { - - msg = msg.data; - _self.resolver[msg.id](msg.msg); - delete _self.resolver[msg.id]; - }; } this.worker.postMessage({ - task: "init", factory: factory, options: options @@ -93,27 +94,22 @@ function register(key) { let callback; if (is_function(arg)) { - callback = arg; args.splice(args.length - 1, 1); } const promise = new Promise(function (resolve) { - - setTimeout(function () { - - self.resolver[++pid] = resolve; - self.worker.postMessage({ - - task: key, - id: pid, - args: args - }); + //setTimeout(function(){ + self.resolver[++pid] = resolve; + self.worker.postMessage({ + task: key, + id: pid, + args: args }); + //}); }); if (callback) { - promise.then(callback); return this; } else { @@ -125,12 +121,11 @@ function register(key) { function create(factory, is_node_js, worker_path) { - let worker; - - try { - - worker = is_node_js ? eval('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') : factory ? new Worker(URL.createObjectURL(new Blob(["onmessage=" + handler.toString()], { type: "text/javascript" }))) : new Worker(is_string(worker_path) ? worker_path : "worker/worker.js", { type: "module" }); - } catch (e) {} + let worker = is_node_js ? + // This eval will be removed when compiling, it isn't there in final build + (0, eval)('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') + //eval('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') + : factory ? new window.Worker(URL.createObjectURL(new Blob(["onmessage=" + handler.toString()], { type: "text/javascript" }))) : new window.Worker(is_string(worker_path) ? worker_path : "worker/worker.js", { type: "module" }); return worker; } \ No newline at end of file diff --git a/dist/module-debug/worker/node.js b/dist/module-debug/worker/node.js index 5326018..ba05f24 100644 --- a/dist/module-debug/worker/node.js +++ b/dist/module-debug/worker/node.js @@ -1,5 +1,10 @@ const { parentPort } = require("worker_threads"), - { Index } = require("../flexsearch.bundle.min.js"); + { join } = require("path"), + { Index } = require("../../dist/flexsearch.bundle.min.js"); +// TODO EXCHANGE + + +//const { Index } = require("../flexsearch.bundle.min.js"); let index; @@ -14,16 +19,25 @@ parentPort.on("message", function (data) { switch (task) { case "init": - const options = data.options || {}, - encode = options.encode; + let options = data.options || {}, + filepath = options.config; + // load extern field configuration - options.cache = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* collapse: */!1; - - if (encode && 0 === encode.indexOf("function")) { - - options.encode = new Function("return " + encode)(); + if ("/" !== filepath[0] && "\\" !== filepath[0]) { + // current working directory + const dir = process.cwd(); + filepath = join(dir, filepath); } + if (filepath) { + options = require(filepath); + } + + // deprecated: + // const encode = options["encode"]; + // if(encode && (encode.indexOf("function") === 0)){ + // options["encode"] = new Function("return " + encode)(); + // } index = new Index(options); break; diff --git a/dist/module-min/async.js b/dist/module-min/async.js deleted file mode 100644 index 9813300..0000000 --- a/dist/module-min/async.js +++ /dev/null @@ -1 +0,0 @@ -import{IndexInterface,DocumentInterface}from"./type.js";import{is_function,is_object,is_string}from"./common.js";export default function(a){register(a,"add"),register(a,"append"),register(a,"search"),register(a,"update"),register(a,"remove")}function register(a,b){a[b+"Async"]=function(){const a=this,c=arguments,d=c[c.length-1];let e;is_function(d)&&(e=d,delete c[c.length-1]);const f=new Promise(function(d){setTimeout(function(){a.async=!0;const e=a[b].apply(a,c);a.async=!1,d(e)})});return e?(f.then(e),this):f}} \ No newline at end of file diff --git a/dist/module-min/cache.js b/dist/module-min/cache.js deleted file mode 100644 index 07f4553..0000000 --- a/dist/module-min/cache.js +++ /dev/null @@ -1 +0,0 @@ -import{IndexInterface,DocumentInterface}from"./type.js";import{create_object,is_object}from"./common.js";function CacheClass(a){this.limit=!0!==a&&a,this.cache=create_object(),this.queue=[]}export default CacheClass;export function searchCache(a,b,c){is_object(a)&&(a=a.query);let d=this.cache.get(a);return d||(d=this.search(a,b,c),this.cache.set(a,d)),d}CacheClass.prototype.set=function(a,b){if(!this.cache[a]){let b=this.queue.length;b===this.limit?delete this.cache[this.queue[b-1]]:b++;for(let a=b-1;0=this.minlength&&(g||!f[i])){let l=get_score(h,e,j),m="";switch(this.tokenize){case"full":if(2b;d--)if(d-b>=this.minlength){const g=get_score(h,e,j,k,b);m=i.substring(b,d),this.push_index(f,m,g,a,c)}break}case"reverse":if(1=this.minlength){const d=get_score(h,e,j,k,b);this.push_index(f,m,d,a,c)}m=""}case"forward":if(1=this.minlength&&this.push_index(f,m,l,a,c);break}default:if(this.boost&&(l=Math.min(0|l/this.boost(b,i,j),h-1)),this.push_index(f,i,l,a,c),g&&1=this.minlength&&!f[i]){f[i]=1;const b=get_score(h+(e/2>h?0:1),e,j,l-1,g-1),m=this.bidirectional&&i>k;this.push_index(d,m?k:i,b,a,c,m?i:k)}}}}}this.fastupdate||(this.register[a]=1)}}return this};function get_score(a,b,c,d,e){return c&&1=this.minlength&&!b[e]){if(!this.optimize&&!f&&!this.map[e])return g;c[i++]=e,b[e]=1}a=c,d=a.length}if(!d)return g;b||(b=100);let i,j=this.depth&&1=c)))));l++);if(b)return e?single_result(h,c,0):void(a[a.length]=h)}return!b&&h};function single_result(a,b,c){return a=1===a.length?a[0]:concat(a),c||a.length>b?a.slice(c,c+b):a}function get_array(a,b,c,d){if(c){const e=d&&b>c;a=a[e?b:c],a=a&&a[e?c:b]}else a=a[b];return a}Index.prototype.contain=function(a){return!!this.register[a]},Index.prototype.update=function(a,b){return this.remove(a).add(a,b)},Index.prototype.remove=function(a,b){const c=this.register[a];if(c){if(this.fastupdate)for(let b,d=0;d{f=a}));let h,i;switch(e||(e=0)){case 0:if(h="reg",this.fastupdate)for(let a in i=create_object(),this.register)i[a]=1;else i=this.register;break;case 1:h="cfg",i={doc:0,opt:this.optimize?1:0};break;case 2:h="map",i=this.map;break;case 3:h="ctx",i=this.ctx;break;default:return void("undefined"==typeof c&&f&&f());}return async(a,b||this,c,h,d,e,i,f),g}export function importIndex(a,b){b&&(is_string(b)&&(b=JSON.parse(b)),"cfg"===a?this.optimize=!!b.opt:"reg"===a?(this.fastupdate=!1,this.register=b):"map"===a?this.map=b:"ctx"===a?this.ctx=b:void 0)}export function exportDocument(a,b,c,d,e,f){let g;if("undefined"==typeof f&&(g=new Promise(a=>{f=a})),e||(e=0),d||(d=0),d setTimeout(tick, 0, resolve)) + // ); + // + // // apply different performance budgets + // if(key === "update" || key === "remove" && this.fastupdate === false){ + // budget += 1000 * this.resolution; + // if(this.depth) + // budget += 1000 * this.resolution_ctx; + // } + // else if(key === "search"){ + // budget++; + // } + // else{ + // budget += 20 * this.resolution; + // if(this.depth) + // budget += 20 * this.resolution_ctx; + // } + // + // // wait for the event loop cycle + // if(budget >= 1e6){ + // await cycle; + // } + + const args = /*[].slice.call*/arguments, arg = args[args.length - 1]; let callback; - if (is_function(arg)) { - + if ("function" == typeof arg) { callback = arg; delete args[args.length - 1]; } - const promise = new Promise(function (resolve) { - - setTimeout(function () { - - self.async = !0; - const res = self[key].apply(self, args); - self.async = !1; - resolve(res); - }); - }); - - if (callback) { - - promise.then(callback); - return this; - } else { - - return promise; - } + this.async = !0; + const res = this[key].apply(this, args); + this.async = !1; + res.then ? res.then(callback) : callback(res); + return res; }; } \ No newline at end of file diff --git a/dist/module/cache.js b/dist/module/cache.js index 43262f7..a21e83a 100644 --- a/dist/module/cache.js +++ b/dist/module/cache.js @@ -1,168 +1,77 @@ -import { IndexInterface, DocumentInterface } from "./type.js"; -import { create_object, is_object } from "./common.js"; +import Index from "./index.js"; +import Document from "./document.js"; + +/** + * @param {string|Object} query + * @param {number|Object=} limit + * @param {Object=} options + * @this {Index|Document} + * @returns {Array|Promise} + */ + +export function searchCache(query, limit, options) { + + query = ("object" == typeof query ? "" + query.query : "" + query).toLowerCase(); + + //let encoded = this.encoder.encode(query).join(" "); + let cache = this.cache.get(query); + if (!cache) { + cache = this.search(query, limit, options); + if (cache instanceof Promise) { + const self = this; + cache.then(function (cache) { + self.cache.set(query, cache); + }); + } + this.cache.set(query, cache); + } + return cache; +} /** * @param {boolean|number=} limit * @constructor */ -function CacheClass(limit) { - +export default function CacheClass(limit) { /** @private */ - this.limit = !0 !== limit && limit; - + this.limit = !limit || !0 === limit ? 1000 : limit; /** @private */ - this.cache = create_object(); - + this.cache = new Map(); /** @private */ - this.queue = []; - - //this.clear(); + this.last = ""; } -export default CacheClass; - -/** - * @param {string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @this {IndexInterface} - * @returns {Array} - */ - -export function searchCache(query, limit, options) { - - if (is_object(query)) { - - query = query.query; - } - - let cache = this.cache.get(query); - - if (!cache) { - - cache = this.search(query, limit, options); - this.cache.set(query, cache); - } - - return cache; -} - -// CacheClass.prototype.clear = function(){ -// -// /** @private */ -// this.cache = create_object(); -// -// /** @private */ -// this.queue = []; -// }; - CacheClass.prototype.set = function (key, value) { - - if (!this.cache[key]) { - - // it is just a shame that native function array.shift() performs so bad - - // const length = this.queue.length; - // - // this.queue[length] = key; - // - // if(length === this.limit){ - // - // delete this.cache[this.queue.shift()]; - // } - - // the same bad performance - - // this.queue.unshift(key); - // - // if(this.queue.length === this.limit){ - // - // this.queue.pop(); - // } - - // fast implementation variant - - // let length = this.queue.length; - // - // if(length === this.limit){ - // - // length--; - // - // delete this.cache[this.queue[0]]; - // - // for(let x = 0; x < length; x++){ - // - // this.queue[x] = this.queue[x + 1]; - // } - // } - // - // this.queue[length] = key; - - // current fastest implementation variant - // theoretically that should not perform better compared to the example above - - let length = this.queue.length; - - if (length === this.limit) { - - delete this.cache[this.queue[length - 1]]; - } else { - - length++; + if (!this.cache.has(key)) { + this.cache.set(this.last = key, value); + if (this.limit && this.cache.size > this.limit) { + this.cache.delete(this.cache.keys().next().value); } - - for (let x = length - 1; 0 < x; x--) { - - this.queue[x] = this.queue[x - 1]; - } - - this.queue[0] = key; } - - this.cache[key] = value; }; CacheClass.prototype.get = function (key) { - - const cache = this.cache[key]; - - if (this.limit && cache) { - - // probably the indexOf() method performs faster when matched content is on front (left-to-right) - // using lastIndexOf() does not help, it performs almost slower - - const pos = this.queue.indexOf(key); - - // if(pos < this.queue.length - 1){ - // - // const tmp = this.queue[pos]; - // this.queue[pos] = this.queue[pos + 1]; - // this.queue[pos + 1] = tmp; - // } - - if (pos) { - - const tmp = this.queue[pos - 1]; - this.queue[pos - 1] = this.queue[pos]; - this.queue[pos] = tmp; - } + const cache = this.cache.get(key); + if (cache && this.limit && this.last !== key) { + this.cache.delete(key); + this.cache.set(this.last = key, cache); } - return cache; }; -CacheClass.prototype.del = function (id) { +CacheClass.prototype.remove = function (id) { + for (const item of this.cache) { + const key = item[0], + value = item[1]; - for (let i = 0, item, key; i < this.queue.length; i++) { - - key = this.queue[i]; - item = this.cache[key]; - - if (item.includes(id)) { - - this.queue.splice(i--, 1); - delete this.cache[key]; + if (value.includes(id)) { + this.cache.delete(key); } } +}; + +CacheClass.prototype.clear = function () { + this.cache.clear(); + this.last = ""; }; \ No newline at end of file diff --git a/dist/module/common.js b/dist/module/common.js index 95e4ace..b733012 100644 --- a/dist/module/common.js +++ b/dist/module/common.js @@ -1,6 +1,65 @@ -export function parse_option(value, default_value) { +/** + * @param {*} value + * @param {*} default_value + * @param {*=} merge_value + * @return {*} + */ - return "undefined" != typeof value ? value : default_value; +export function parse_option(value, default_value, merge_value) { + const type_merge = typeof merge_value, + type_value = typeof value; + + + if ("undefined" != type_merge) { + if ("undefined" != type_value) { + + if (merge_value) { + if ("function" == type_value && type_merge == type_value) { + return function (str) { + return (/** @type Function */value( + /** @type Function */merge_value(str)) + ); + }; + } + + const constructor_value = value.constructor, + constructor_merge = merge_value.constructor; + + + if (constructor_value === constructor_merge) { + + if (constructor_value === Array) { + return merge_value.concat(value); + } + + if (constructor_value === Map) { + const map = new Map( /** @type !Map */merge_value); + for (const item of /** @type !Map */value) { + const key = item[0], + val = item[1]; + + map.set(key, val); + } + return map; + } + + if (constructor_value === Set) { + const set = new Set( /** @type !Set */merge_value); + for (const val of /** @type !Set */value.values()) { + set.add(val); + } + return set; + } + } + } + + return value; + } else { + return merge_value; + } + } + + return "undefined" == type_value ? default_value : value; } /** @@ -13,19 +72,33 @@ export function create_object_array(count) { const array = Array(count); for (let i = 0; i < count; i++) { - array[i] = create_object(); } return array; } +/** + * @param {!number} count + * @returns {Array} + */ + +export function create_map_array(count) { + + const array = Array(count); + + for (let i = 0; i < count; i++) { + array[i] = new Map(); + } + + return array; +} + export function create_arrays(count) { const array = Array(count); for (let i = 0; i < count; i++) { - array[i] = []; } @@ -38,41 +111,75 @@ export function create_arrays(count) { */ export function get_keys(obj) { - return Object.keys(obj); } export function create_object() { - return Object.create(null); } export function concat(arrays) { - return [].concat.apply([], arrays); } export function sort_by_length_down(a, b) { - return b.length - a.length; } -export function is_array(val) { +export function sort_by_length_up(a, b) { + return a.length - b.length; +} +export function is_array(val) { return val.constructor === Array; } export function is_string(val) { - return "string" == typeof val; } export function is_object(val) { - return "object" == typeof val; } export function is_function(val) { - return "function" == typeof val; +} + +/** + * @param {Map|Set} val + * @param {boolean=} stringify + * @return {Array} + */ + +export function toArray(val, stringify) { + const result = []; + for (const key of val.keys()) { + result.push(stringify ? "" + key : key); + } + return result; +} + +// TODO support generic function created from string when tree depth > 1 +export function parse_simple(obj, tree) { + + if (is_string(tree)) { + obj = obj[tree]; + } else for (let i = 0; obj && i < tree.length; i++) { + obj = obj[tree[i]]; + } + + return obj; +} + +export function get_max_len(arr) { + let len = 0; + for (let i = 0, res; i < arr.length; i++) { + if (res = arr[i]) { + if (len < res.length) { + len = res.length; + } + } + } + return len; } \ No newline at end of file diff --git a/dist/module/compress.js b/dist/module/compress.js new file mode 100644 index 0000000..8efa91a --- /dev/null +++ b/dist/module/compress.js @@ -0,0 +1,65 @@ +let table, timer; // = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/"; + +const cache = new Map(); + +function toRadix(number, radix = 255) { + + if (!table) { + table = Array(radix); + // the char code 0 could be a special marker + for (let i = 0; i < radix; i++) table[i] = i + 1; + table = String.fromCharCode.apply(null, table); + } + + let rixit, + residual = number, + result = ""; + + + while (!0) { + rixit = residual % radix; + result = table.charAt(rixit) + result; + residual = 0 | residual / radix; + if (!residual) break; + } + + return result; +} + +export default function (str) { + if (timer) { + if (cache.has(str)) { + return cache.get(str); + } + } else { + timer = setTimeout(clear); + } + + /* 2 ** ((level + 1.5) * 1.6 | 0) */ + + const result = toRadix("number" == typeof str ? str : lcg(str)); + + 2e5 < cache.size && cache.clear(); + cache.set(str, result); + + + return result; +} + +function lcg(str) { + let range = 4294967295; + if ("number" == typeof str) { + return str & range; + } + let crc = 0; + for (let i = 0; i < str.length; i++) { + crc = (crc * 33 ^ str.charCodeAt(i)) & range; + } + // shift up from Int32 to UInt32 range 0xFFFFFFFF + return crc + 2147483648; +} + +function clear() { + timer = null; + cache.clear(); +} \ No newline at end of file diff --git a/dist/module/db/clickhouse/index.js b/dist/module/db/clickhouse/index.js new file mode 100644 index 0000000..51bfa95 --- /dev/null +++ b/dist/module/db/clickhouse/index.js @@ -0,0 +1,648 @@ + +import { ClickHouse } from "clickhouse"; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { concat, toArray } from "../../common.js"; +const defaults = { + host: "http://localhost", + port: "8123", + debug: !1, + basicAuth: null, + isUseGzip: !1, + trimQuery: !1, + usePost: !1, + format: "json", + raw: !1, + config: { + output_format_json_quote_64bit_integers: 0, + enable_http_compression: 0, + database: "default" + } +}, + VERSION = 1, + fields = ["map", "ctx", "tag", "reg", "cfg"], + types = { + text: "String", + char: "String", + varchar: "String", + string: "String", + number: "Int32", + numeric: "Int32", + integer: "Int32", + smallint: "Int16", + tinyint: "Int8", + mediumint: "Int32", + int: "Int32", + int8: "Int8", + uint8: "UInt8", + int16: "Int16", + uint16: "UInt16", + int32: "Int32", + uint32: "UInt32", + int64: "Int64", + uint64: "UInt64", + bigint: "Int64" +}; + + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +let DB; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function ClickhouseDB(name, config = {}) { + if (!(this instanceof ClickhouseDB)) { + return new ClickhouseDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + //field = "Test-456"; + this.id = "flexsearch" + (name ? "_" + sanitize(name) : ""); + this.field = config.field ? "_" + sanitize(config.field) : ""; + // Clickhouse does not support ALTER TABLE to upgrade + // the type of the ID when it is a part of the merge key + this.type = config.type ? types[config.type.toLowerCase()] : "String"; + if (!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); + //this.trx = false; + this.support_tag_search = !0; + this.db = DB || (DB = config.db || null); + Object.assign(defaults, config); + config.database && (defaults.config.database = config.database); + this.db && delete defaults.db; +} + +ClickhouseDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + defaults.resolution = Math.max(flexsearch.resolution, flexsearch.resolution_ctx); + flexsearch.db = this; + return this.open(); +}; + +ClickhouseDB.prototype.open = async function () { + + if (!this.db) { + this.db = DB || (DB = new ClickHouse(defaults)); + } + + const exists = await this.db.query(` + SELECT 1 FROM system.databases WHERE name = '${this.id}'; + `).toPromise(); + + if (!exists || !exists.length) { + await this.db.query(` + CREATE DATABASE IF NOT EXISTS ${this.id}; + `).toPromise(); + } + + for (let i = 0; i < fields.length; i++) { + switch (fields[i]) { + case "map": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.map${this.field}( + key String, + res ${255 >= defaults.resolution ? "UInt8" : "UInt16"}, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (key)*/ + ORDER BY (key, id); + `, { params: { name: this.id + ".map" + this.field } }).toPromise(); + break; + + case "ctx": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.ctx${this.field}( + ctx String, + key String, + res ${255 >= defaults.resolution ? "UInt8" : "UInt16"}, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (ctx, key)*/ + ORDER BY (ctx, key, id); + `).toPromise(); + break; + + case "tag": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.tag${this.field}( + tag String, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (ctx, key)*/ + ORDER BY (tag, id); + `).toPromise(); + break; + + case "reg": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.reg( + id ${this.type}, + doc Nullable(String) + ) + ENGINE = MergeTree + ORDER BY (id); + `).toPromise(); + break; + + case "cfg": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.cfg${this.field}( + cfg String + ) + ENGINE = TinyLog; + `).toPromise(); + break; + } + } + + return this.db; +}; + +ClickhouseDB.prototype.close = function () { + this.db.close(); + this.db = null; + return this; +}; + +ClickhouseDB.prototype.destroy = async function () { + await Promise.all([this.db.query(`DROP TABLE ${this.id}.map${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.ctx${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.tag${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.cfg${this.field};`).toPromise(), this.db.query(`DROP TABLE ${this.id}.reg;`).toPromise()]); + this.close(); +}; + +ClickhouseDB.prototype.clear = function () { + return Promise.all([this.db.query(`TRUNCATE TABLE ${this.id}.map${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.ctx${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.tag${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.cfg${this.field};`).toPromise(), this.db.query(`TRUNCATE TABLE ${this.id}.reg;`).toPromise()]); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + for (let i = 0; i < rows.length; i++) { + if (enrich) { + if (rows[i].doc) { + rows[i].doc = JSON.parse(rows[i].doc); + } + } else { + rows[i] = rows[i].id; + } + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +ClickhouseDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = !1, tags) { + let rows, + params = ctx ? { ctx, key } : { key }, + table = this.id + (ctx ? ".ctx" : ".map") + this.field; + + if (tags) { + for (let i = 0, count = 1; i < tags.length; i += 2) { + ` AND ${table}.id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = {tag${count}:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + if (ctx) { + rows = this.db.query(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE ctx = {ctx:String} AND key = {key:String} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, { params }).toPromise(); + } else { + rows = this.db.query(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE key = {key:String} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, { params }).toPromise(); + } + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +ClickhouseDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const table = this.id + ".tag" + this.field, + promise = this.db.query(` + SELECT ${table}.id + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE tag = {tag:String} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, { params: { tag } }).toPromise(); + + enrich || promise.then(function (rows) { + return create_result(rows, !0, !1); + }); + return promise; +}; + +ClickhouseDB.prototype.enrich = async function (ids) { + let MAXIMUM_QUERY_VARS = 1e5, + result = []; + + if ("object" != typeof ids) { + ids = [ids]; + } + for (let count = 0; count < ids.length;) { + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + let params = {}, + stmt = ""; + + for (let i = 0; i < chunk.length; i++) { + stmt += (stmt ? "," : "") + "{id" + (i + 1) + ":String}"; + params["id" + (i + 1)] = chunk[i]; + } + const res = await this.db.query(` + SELECT id, doc + FROM ${this.id}.reg + WHERE id IN (${stmt})`, { params }).toPromise(); + if (res && res.length) { + for (let i = 0, doc; i < res.length; i++) { + if (doc = res[i].doc) { + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + return 1 === result.length ? result[0] : 1 < result.length ? concat(result) : result; +}; + +ClickhouseDB.prototype.has = function (id) { + return this.db.query(` + SELECT EXISTS( + SELECT 1 + FROM ${this.id}.reg + WHERE id = {id:${this.type /*=== "number" ? "Int32" : "String"*/}} + LIMIT 1 + )`, { params: { id } }).toPromise(); +}; + +ClickhouseDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !0, tags) { + let rows; + if (1 < query.length && flexsearch.depth) { + let where = "", + params = {}, + keyword = query[0], + term; + + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + where += (where ? " OR " : "") + `(ctx = {ctx${i}:String} AND key = {key${i}:String})`; + params["ctx" + i] = swap ? term : keyword; + params["key" + i] = swap ? keyword : term; + keyword = term; + } + + if (tags) { + where = "(" + where + ")"; + for (let i = 0, count = 1; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = {tag${count}:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + + rows = this.db.query(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.ctx${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + (query.length - 1)} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, { params }).toPromise(); + + // for(let i = 1; i < query.length; i++){ + // where += (where ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM ${this.id}.ctx${this.field} + // WHERE ctx = {ctx${i}:String} AND key = {key${i}:String} + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params["ctx" + i] = swap ? term : keyword; + // params["key" + i] = swap ? keyword : term; + // keyword = term; + // } + // + // rows = await this.db.query(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, { params }).toPromise(); + } else { + let where = "", + params = {}; + + + for (let i = 0; i < query.length; i++) { + where += (where ? "," : "") + `{key${i}:String}`; + params["key" + i] = query[i]; + } + where = "key " + (1 < query.length ? "IN (" + where + ")" : "= " + where); + + if (tags) { + where = "(" + where + ")"; + for (let i = 0, count = 1; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = {tag${count}:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + + rows = this.db.query(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.map${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT OUTER JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + query.length} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, { params }).toPromise(); + + // for(let i = 0; i < query.length; i++){ + // params["key" + i] = query[i]; + // where += (where ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM ${ this.id }.map${ this.field } + // WHERE key = {key${i}:String} + // `; + // } + // rows = await this.db.query(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, { params }).toPromise(); + } + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +ClickhouseDB.prototype.info = function () { + // todo +}; + +ClickhouseDB.prototype.transaction = function (task) { + + // not supported + return task.call(this); +}; + +ClickhouseDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + if (!flexsearch.reg.size) { + return; + } + + if (flexsearch.map.size) { + let data = []; + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + //this.type || (this.type = typeof ids[0]); + for (let j = 0; j < ids.length; j++) { + data.push({ + key: key, + res: i, + id: /*this.type === "number" + ? parseInt(ids[j], 10) + :*/ids[j] + }); + } + } + } + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.map${this.field} (key, res, id)`, data).toPromise(); + } + } + + if (flexsearch.ctx.size) { + let data = []; + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + for (let j = 0; j < ids.length; j++) { + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: /*this.type === "number" + ? parseInt(ids[j], 10) + :*/ids[j] + }); + } + } + } + } + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.ctx${this.field} (ctx, key, res, id)`, data).toPromise(); + } + } + + if (flexsearch.tag) { + let data = []; + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let j = 0; j < ids.length; j++) { + data.push({ tag, id: ids[j] }); + } + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.tag${this.field} (tag, id)`, data).toPromise(); + } + } + + if (flexsearch.store) { + let data = []; + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + data.push({ id, doc: doc && JSON.stringify(doc) }); + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.reg (id, doc)`, data).toPromise(); + } + } else if (!flexsearch.bypass) { + let data = toArray(flexsearch.reg); + for (let i = 0; i < data.length; i++) { + data[i] = { id: data[i] }; + } + if (data.length) { + await this.db.insert(`INSERT INTO ${this.id}.reg (id)`, data).toPromise(); + } + } + + // TODO + // await this.db.insert(`INSERT INTO ${this.id}.cfg${this.field} (cfg)`, [{ + // cfg: JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }) + // }]).toPromise(); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); + + await Promise.all([this.db.query(`OPTIMIZE TABLE ${this.id}.map${this.field} FINAL`).toPromise(), this.db.query(`OPTIMIZE TABLE ${this.id}.ctx${this.field} FINAL`).toPromise(), this.db.query(`OPTIMIZE TABLE ${this.id}.tag${this.field} FINAL`).toPromise(), this.db.query(`OPTIMIZE TABLE ${this.id}.reg FINAL`).toPromise()]); +}; + +ClickhouseDB.prototype.remove = async function (ids) { + + if ("object" != typeof ids) { + ids = [ids]; + } + + while (ids.length) { + + let chunk = ids.slice(0, 1e5); + ids = ids.slice(1e5); + chunk = "String" === this.type ? "'" + chunk.join("','") + "'" : chunk.join(","); + + await Promise.all([this.db.query(` + ALTER TABLE ${this.id}.map${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise(), this.db.query(` + ALTER TABLE ${this.id}.ctx${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise(), this.db.query(` + ALTER TABLE ${this.id}.tag${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise(), this.db.query(` + ALTER TABLE ${this.id}.reg + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;`).toPromise()]); + } +}; \ No newline at end of file diff --git a/dist/module/db/indexeddb/index.js b/dist/module/db/indexeddb/index.js new file mode 100644 index 0000000..e7c6fee --- /dev/null +++ b/dist/module/db/indexeddb/index.js @@ -0,0 +1,589 @@ + +import Document from "../../document.js"; + +const VERSION = 1, + IndexedDB = "undefined" != typeof window && (window.indexedDB || window.mozIndexedDB || window.webkitIndexedDB || window.msIndexedDB), + IDBTransaction = "undefined" != typeof window && (window.IDBTransaction || window.webkitIDBTransaction || window.msIDBTransaction), + IDBKeyRange = "undefined" != typeof window && (window.IDBKeyRange || window.webkitIDBKeyRange || window.msIDBKeyRange), + fields = ["map", "ctx", "tag", "reg", "cfg"]; + +import StorageInterface from "../interface.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +/** + * @constructor + * @implements StorageInterface + */ + +export default function IdxDB(name, config = {}) { + if (!(this instanceof IdxDB)) { + return new IdxDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = "flexsearch" + (name ? ":" + sanitize(name) : ""); + this.field = config.field ? sanitize(config.field) : ""; + this.support_tag_search = !1; + this.db = null; + this.trx = {}; +} + +IdxDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +IdxDB.prototype.open = function () { + + let self = this; + + navigator.storage && navigator.storage.persist(); + + return this.db || new Promise(function (resolve, reject) { + + const req = IndexedDB.open(self.id + (self.field ? ":" + self.field : ""), VERSION); + + req.onupgradeneeded = function () { + + const db = self.db = this.result; + + // Using Indexes + IDBKeyRange on schema map => [key, res, id] performs + // too bad and blows up amazingly in size + // The schema map:key => [res][id] is currently used instead + // In fact that bypass the idea of a storage solution, + // IndexedDB is such a poor contribution :( + + fields.forEach(ref => { + db.objectStoreNames.contains(ref) || db.createObjectStore(ref); //{ autoIncrement: true /*keyPath: "id"*/ } + //.createIndex("idx", "ids", { multiEntry: true, unique: false }); + }); + + // switch(event.oldVersion){ // existing db version + // case 0: + // // version 0 means that the client had no database + // // perform initialization + // case 1: + // // client had version 1 + // // update + // } + }; + + req.onblocked = function (event) { + // this event shouldn't trigger if we handle onversionchange correctly + // it means that there's another open connection to the same database + // and it wasn't closed after db.onversionchange triggered for it + console.error("blocked", event); + reject(); + }; + + req.onerror = function (event) { + console.error(this.error, event); + reject(); + }; + + req.onsuccess = function () { + self.db = this.result; //event.target.result; + self.db.onversionchange = function () { + //database is outdated + self.close(); + }; + resolve(self); + }; + }); +}; + +IdxDB.prototype.close = function () { + this.db.close(); + this.db = null; +}; + +IdxDB.prototype.destroy = function () { + this.db && this.close(); + return IndexedDB.deleteDatabase(this.id + (this.field ? ":" + this.field : "")); +}; + +// IdxDB.prototype.set = function(ref, key, ctx, data){ +// const transaction = this.db.transaction(ref, "readwrite"); +// const map = transaction.objectStore(ref); +// const req = map.put(data, ctx ? ctx + ":" + key : key); +// return transaction;//promisfy(req, callback); +// }; + +// IdxDB.prototype.delete = function(ref, key, ctx){ +// const transaction = this.db.transaction(ref, "readwrite"); +// const map = transaction.objectStore(ref); +// const req = map.delete(ctx ? ctx + ":" + key : key); +// return transaction;//promisfy(req, callback); +// }; + +IdxDB.prototype.clear = function () { + const transaction = this.db.transaction(fields, "readwrite"); + for (let i = 0; i < fields.length; i++) { + transaction.objectStore(fields[i]).clear(); + } + return promisfy(transaction); +}; + +IdxDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = /* tag? */!0, enrich = !1) { + const transaction = this.db.transaction(ctx ? "ctx" : "map", "readonly"), + map = transaction.objectStore(ctx ? "ctx" : "map"), + req = map.get(ctx ? ctx + ":" + key : key), + self = this; + + return promisfy(req).then(function (res) { + let result = []; + if (!res || !res.length) return result; + if (resolve) { + if (!limit && !offset && 1 === res.length) { + return res[0]; + } + for (let i = 0, arr; i < res.length; i++) { + if ((arr = res[i]) && arr.length) { + if (offset >= arr.length) { + offset -= arr.length; + continue; + } + const end = limit ? offset + Math.min(arr.length - offset, limit) : arr.length; + for (let j = offset; j < end; j++) { + result.push(arr[j]); + } + offset = 0; + if (result.length === limit) { + break; + } + } + } + return enrich ? self.enrich(result) : result; + } else { + return res; + } + }); +}; + +IdxDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const transaction = this.db.transaction("tag", "readonly"), + map = transaction.objectStore("tag"), + req = map.get(tag), + self = this; + + return promisfy(req).then(function (ids) { + if (!ids || !ids.length || offset >= ids.length) return []; + if (!limit && !offset) return ids; + const result = ids.slice(offset, offset + limit); + return enrich ? self.enrich(result) : result; + }); +}; + + +IdxDB.prototype.enrich = function (ids) { + if ("object" != typeof ids) { + ids = [ids]; + } + const transaction = this.db.transaction("reg", "readonly"), + map = transaction.objectStore("reg"), + promises = []; + + for (let i = 0; i < ids.length; i++) { + promises[i] = promisfy(map.get(ids[i])); + } + return Promise.all(promises).then(function (docs) { + for (let i = 0; i < docs.length; i++) { + docs[i] = { + id: ids[i], + doc: docs[i] ? JSON.parse(docs[i]) : null + }; + } + return docs; + }); +}; + + +IdxDB.prototype.has = function (id) { + const transaction = this.db.transaction("reg", "readonly"), + map = transaction.objectStore("reg"), + req = map.getKey(id); + + return promisfy(req); +}; + +IdxDB.prototype.search = null; + +// IdxDB.prototype.has = function(ref, key, ctx){ +// const transaction = this.db.transaction(ref, "readonly"); +// const map = transaction.objectStore(ref); +// const req = map.getKey(ctx ? ctx + ":" + key : key); +// return promisfy(req); +// }; + +IdxDB.prototype.info = function () { + // todo +}; + +/** + * @param {!string} ref + * @param {!string} modifier + * @param {!Function} task + */ + +IdxDB.prototype.transaction = function (ref, modifier, task) { + + let store = this.trx[ref + ":" + modifier]; + if (store) return task.call(this, store); + + let transaction = this.db.transaction(ref, modifier); + this.trx[ref + ":" + modifier] = store = transaction.objectStore(ref); + + return new Promise((resolve, reject) => { + transaction.onerror = err => { + this.trx[ref + ":" + modifier] = null; + transaction.abort(); + transaction = store = null; + reject(err); + //db.close; + }; + transaction.oncomplete = res => { + this.trx[ref + ":" + modifier] = null; + transaction = store = null; + resolve(res || !0); + //db.close; + }; + return task.call(this, store); + }); +}; + +IdxDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction("map", "readwrite", function (store) { + + for (const item of flexsearch.map) { + const key = item[0], + value = item[1]; + + if (!value.length) continue; + + if (_replace) { + store.put(value, key); + continue; + } + + store.get(key).onsuccess = function () { + let result = this.result, + changed; + + if (result && result.length) { + const maxlen = Math.max(result.length, value.length); + for (let i = 0, res, val; i < maxlen; i++) { + val = value[i]; + if (val && val.length) { + res = result[i]; + if (res && res.length) { + for (let j = 0; j < val.length; j++) { + res.push(val[j]); + } + changed = 1; + //result[i] = res.concat(val); + //result[i] = new Set([...result[i], ...value[i]]); + //result[i] = result[i].union(new Set(value[i])); + } else { + result[i] = val; + changed = 1; + //result[i] = new Set(value[i]) + } + } + } + } else { + result = value; + changed = 1; + //result = []; + //for(let i = 0; i < value.length; i++){ + // if(value[i]) result[i] = new Set(value[i]); + //} + } + + changed && store.put(result, key); + }; + } + }); + + await this.transaction("ctx", "readwrite", function (store) { + + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + + for (const item of ctx_value) { + const key = item[0], + value = item[1]; + + if (!value.length) continue; + + if (_replace) { + store.put(value, ctx_key + ":" + key); + continue; + } + + store.get(ctx_key + ":" + key).onsuccess = function () { + let result = this.result, + changed; + + if (result && result.length) { + const maxlen = Math.max(result.length, value.length); + for (let i = 0, res, val; i < maxlen; i++) { + val = value[i]; + if (val && val.length) { + res = result[i]; + if (res && res.length) { + for (let j = 0; j < val.length; j++) { + res.push(val[j]); + } + //result[i] = res.concat(val); + changed = 1; + } else { + result[i] = val; + changed = 1; + } + } + } + } else { + result = value; + changed = 1; + } + + changed && store.put(result, ctx_key + ":" + key); + }; + } + } + }); + + if (flexsearch.store) { + await this.transaction("reg", "readwrite", function (store) { + for (const item of flexsearch.store) { + const id = item[0], + doc = item[1]; + + store.put("object" == typeof doc ? JSON.stringify(doc) : 1, id); + } + }); + } else if (!flexsearch.bypass) { + await this.transaction("reg", "readwrite", function (store) { + for (const id of flexsearch.reg.keys()) { + store.put(1, id); + } + }); + } + + if (flexsearch.tag) { + await this.transaction("tag", "readwrite", function (store) { + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + + store.get(tag).onsuccess = function () { + let result = this.result; + result = result && result.length ? result.concat(ids) : ids; + store.put(result, tag); + }; + } + }); + } + + // TODO + // await this.transaction("cfg", "readwrite", function(store){ + // store.put({ + // "charset": flexsearch.charset, + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "fastupdate": flexsearch.fastupdate, + // "compress": flexsearch.compress, + // "encoder": { + // "minlength": flexsearch.encoder.minlength + // }, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }, "current"); + // }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + + flexsearch.tag && flexsearch.tag.clear(); + + flexsearch.store && flexsearch.store.clear(); + + flexsearch.document || flexsearch.reg.clear(); +}; + +/** + * @param {IDBCursorWithValue} cursor + * @param {Array} ids + * @param {boolean=} _tag + */ + +function handle(cursor, ids, _tag) { + + const arr = cursor.value; + let changed, + parse, + count = 0; + + + for (let x = 0, result; x < arr.length; x++) { + // tags has no resolution layer + if (result = _tag ? arr : arr[x]) { + for (let i = 0, pos, id; i < ids.length; i++) { + id = ids[i]; + pos = result.indexOf(parse ? parseInt(id, 10) : id); + if (0 > pos && !parse && "string" == typeof id && !isNaN(id)) { + pos = result.indexOf(parseInt(id, 10)); + pos && (parse = 1); + } + if (0 <= pos) { + changed = 1; + if (1 < result.length) { + result.splice(pos, 1); + } else { + arr[x] = []; + break; + } + } + } + + count += result.length; + } + if (_tag) break; + } + + if (!count) { + + cursor.delete(); + //store.delete(cursor.key); + } else if (changed) { + + //await new Promise(resolve => { + cursor.update(arr); //.onsuccess = resolve; + //}); + } + + cursor.continue(); +} + +IdxDB.prototype.remove = function (ids) { + + if ("object" != typeof ids) { + ids = [ids]; + } + + return (/** @type {!Promise} */Promise.all([this.transaction("map", "readwrite", function (store) { + store.openCursor().onsuccess = function () { + const cursor = this.result; + cursor && handle(cursor, ids); + }; + }), this.transaction("ctx", "readwrite", function (store) { + store.openCursor().onsuccess = function () { + const cursor = this.result; + cursor && handle(cursor, ids); + }; + }), this.transaction("tag", "readwrite", function (store) { + store.openCursor().onsuccess = function () { + const cursor = this.result; + cursor && handle(cursor, ids, !0); + }; + }), + // let filtered = []; + this.transaction("reg", "readwrite", function (store) { + for (let i = 0; i < ids.length; i++) { + store.delete(ids[i]); + } + // return new Promise(resolve => { + // store.openCursor().onsuccess = function(){ + // const cursor = this.result; + // if(cursor){ + // const id = cursor.value; + // if(ids.includes(id)){ + // filtered.push(id); + // cursor.delete(); + // } + // cursor.continue(); + // } + // else{ + // resolve(); + // } + // }; + // }); + }) + // ids = filtered; + ]) + ); +}; + +/** + * @param {IDBRequest} req + * @param {Function=} callback + * @return {!Promise} + */ + +function promisfy(req, callback) { + return new Promise((resolve, reject) => { + /** @this IDBRequest */ + req.onsuccess = function () { + callback && callback(this.result); + resolve(this.result); + }; + /** @this IDBRequest */ + req.oncomplete = function () { + callback && callback(this.result); + resolve(this.result); + }; + req.onerror = reject; + req = null; + }); +} \ No newline at end of file diff --git a/dist/module/db/mongo/index.js b/dist/module/db/mongo/index.js new file mode 100644 index 0000000..0475df2 --- /dev/null +++ b/dist/module/db/mongo/index.js @@ -0,0 +1,584 @@ +import { MongoClient } from "mongodb"; +const defaults = { + host: "localhost", + port: "27017", + user: null, + pass: null +}, + VERSION = 1, + fields = ["map", "ctx", "tag", "reg", "cfg"]; + +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +let CLIENT, + DB = Object.create(null); + + +/** + * @constructor + * @implements StorageInterface + */ + +export default function MongoDB(name, config = {}) { + if (!(this instanceof MongoDB)) { + return new MongoDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = "flexsearch" + (name ? "-" + sanitize(name) : ""); + this.field = config.field ? "-" + sanitize(config.field) : ""; + this.type = config.type || ""; + this.db = config.db || DB[this.id] || CLIENT || null; + this.trx = !1; + this.support_tag_search = /* tag? */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + Object.assign(defaults, config); + this.db && delete defaults.db; +} + +// MongoDB.mount = function(flexsearch){ +// return new this().mount(flexsearch); +// }; + +MongoDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +async function createCollection(db, ref, field) { + switch (ref) { + case "map": + await db.createCollection("map" + field); + await db.collection("map" + field).createIndex({ key: 1 }); + await db.collection("map" + field).createIndex({ id: 1 }); + break; + case "ctx": + await db.createCollection("ctx" + field); + await db.collection("ctx" + field).createIndex({ ctx: 1, key: 1 }); + await db.collection("ctx" + field).createIndex({ id: 1 }); + break; + case "tag": + await db.createCollection("tag" + field); + await db.collection("tag" + field).createIndex({ tag: 1 }); + await db.collection("tag" + field).createIndex({ id: 1 }); + break; + case "reg": + await db.createCollection("reg"); + await db.collection("reg").createIndex({ id: 1 }); + break; + case "cfg": + await db.createCollection("cfg" + field); + } +} + +MongoDB.prototype.open = async function () { + + if (!this.db) { + if (!(this.db = DB[this.id])) { + if (!(this.db = CLIENT)) { + + let url = defaults.url; + if (!url) { + url = defaults.user ? `mongodb://${defaults.user}:${defaults.pass}@${defaults.host}:${defaults.port}` : `mongodb://${defaults.host}:${defaults.port}`; + } + this.db = CLIENT = new MongoClient(url); + await this.db.connect(); + } + } + } + + if (this.db.db) { + this.db = DB[this.id] = this.db.db(this.id); + } + + const collections = await this.db.listCollections().toArray(); + + for (let i = 0, found; i < fields.length; i++) { + found = !1; + + for (let j = 0; j < collections.length; j++) { + if (collections[j].name === fields[i] + ("reg" !== fields[i] ? this.field : "")) { + found = !0; + break; + } + } + if (!found) { + await createCollection(this.db, fields[i], this.field); + } + } + + return this.db; +}; + +MongoDB.prototype.close = function () { + this.db.close(); + this.db = null; + return this; +}; + +MongoDB.prototype.destroy = async function () { + await Promise.all([this.db.dropCollection("map" + this.field), this.db.dropCollection("ctx" + this.field), this.db.dropCollection("tag" + this.field), this.db.dropCollection("cfg" + this.field), this.db.dropCollection("reg")]); + this.close(); +}; + +async function clear(ref) { + await this.db.dropCollection(ref); + await createCollection(this.db, ref, this.field); +} + +MongoDB.prototype.clear = function () { + return Promise.all([clear.call(this, "map" + this.field), clear.call(this, "ctx" + this.field), clear.call(this, "tag" + this.field), clear.call(this, "cfg" + this.field), clear.call(this, "reg")]); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + if (!enrich) for (let i = 0; i < rows.length; i++) { + rows[i] = rows[i].id; + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +MongoDB.prototype.get = async function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = !1, tags) { + let rows, + params = ctx ? { ctx, key } : { key }; + + if (!enrich && !tags) { + rows = await this.db.collection((ctx ? "ctx" : "map") + this.field).find(params, { projection: { _id: 0, res: 1, id: 1 }, limit, skip: offset }).toArray(); + } else { + const project = { _id: 0, id: 1 }, + stmt = [{ $match: params }]; + + + if (!resolve) { + project.res = 1; + } + + if (enrich) { + project.doc = "$doc.doc"; + stmt.push({ $lookup: { + from: "reg", + localField: "id", + foreignField: "id", + as: "doc" + } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }); + } + + if (tags) { + + const match = {}; + for (let i = 0, count = 1; i < tags.length; i += 2) { + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push({ $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "id", + foreignField: "id", + as: "tag" + count + } }); + count++; + } + + stmt.push({ $project: project }, { $match: match }, { $project: { id: 1, doc: 1 } }); + } else { + stmt.push({ $project: project }); + } + + stmt.push({ $sort: { res: 1 } }, { $skip: offset }, { $limit: limit }); + + rows = []; + const result = await this.db.collection((ctx ? "ctx" : "map") + this.field).aggregate(stmt); + + while (!0) { + const row = await result.next(); + if (row) rows.push(row);else break; + } + } + + return create_result(rows, resolve, enrich); +}; + +MongoDB.prototype.tag = async function (tag, limit = 0, offset = 0, enrich = !1) { + + if (!enrich) { + + const rows = await this.db.collection("tag" + this.field).find({ tag }, { projection: { _id: 0, id: 1 }, limit, skip: offset }).toArray(); + return create_result(rows, !0, !1); + } else { + + let rows = []; + const result = await this.db.collection("tag" + this.field).aggregate([{ $match: { tag } }, { $skip: offset }, { $limit: limit }, { $lookup: { + from: "reg", + localField: "id", + foreignField: "id", + as: "doc" + } }, { $project: { _id: 0, id: 1, doc: "$doc.doc" } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }]); + + while (!0) { + const row = await result.next(); + if (row) rows.push(row);else break; + } + + return rows; + } +}; + +MongoDB.prototype.enrich = function (ids) { + if ("object" != typeof ids) { + ids = [ids]; + } + return this.db.collection("reg").find({ id: { $in: ids } }, { projection: { _id: 0, id: 1, doc: 1 } }).toArray(); +}; + +MongoDB.prototype.has = function (id) { + return this.db.collection("reg").countDocuments({ id }, { limit: 1 }); +}; + +MongoDB.prototype.search = async function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let result = [], + rows; + + if (1 < query.length && flexsearch.depth) { + let params = [], + keyword = query[0], + term; + + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + params.push({ + ctx: swap ? term : keyword, + key: swap ? keyword : term + }); + keyword = term; + } + + let project = resolve ? { _id: 1 } : { _id: 1, res: 1 }; + + const stmt = [{ $match: { $or: params } }, { $group: { + _id: "$id", + res: suggest ? { $sum: 1 } : { $min: 1 }, + count: { $sum: 1 } + } }]; + + suggest || stmt.push({ $match: { count: query.length - 1 } }); + + if (enrich) { + project.doc = "$doc.doc"; + stmt.push({ $lookup: { + from: "reg", + localField: "_id", + foreignField: "id", + as: "doc" + } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }); + } + + if (tags) { + + const match = {}; + for (let i = 0, count = 1; i < tags.length; i += 2) { + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push({ $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "_id", + foreignField: "id", + as: "tag" + count + } }); + count++; + } + + stmt.push({ $project: project }, { $match: match }); + } else { + stmt.push({ $project: project }); + } + + stmt.push({ $sort: suggest ? { count: -1, res: 1 } : { res: 1 } }, { $skip: offset }, { $limit: limit }); + + if (tags) { + project = { _id: 1 }; + if (!resolve) project.res = 1; + if (enrich) project.doc = 1; + + stmt.push({ $project: project }); + } + + rows = await this.db.collection("ctx" + this.field).aggregate(stmt); + } else { + + let project = resolve ? { _id: 1 } : { _id: 1, res: 1 }; + + const stmt = [{ $match: { + key: { $in: query } + } }, { $group: { + _id: "$id", + res: suggest ? { $sum: 1 } : { $min: 1 }, + count: { $sum: 1 } + } }]; + + suggest || stmt.push({ $match: { count: query.length } }); + + if (enrich) { + project.doc = "$doc.doc"; + stmt.push({ $lookup: { + from: "reg", + localField: "_id", + foreignField: "id", + as: "doc" + } }, { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: !0 + } }); + } + + if (tags) { + + const match = {}; + for (let i = 0, count = 1; i < tags.length; i += 2) { + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push({ $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "_id", + foreignField: "id", + as: "tag" + count + } }); + count++; + } + + stmt.push({ $project: project }, { $match: match }); + } else { + stmt.push({ $project: project }); + } + + stmt.push({ $sort: suggest ? { count: -1, res: 1 } : { res: 1 } }, { $skip: offset }, { $limit: limit }); + + if (tags) { + project = { _id: 1 }; + if (!resolve) project.res = 1; + if (enrich) project.doc = 1; + + stmt.push({ $project: project }); + } + + rows = await this.db.collection("map" + this.field).aggregate(stmt); + } + + while (!0) { + const row = await rows.next(); + if (row) { + if (resolve && !enrich) { + result.push(row._id); + } else { + row.id = row._id; + delete row._id; + result.push(row); + } + } else break; + } + + if (resolve && !enrich) { + return result; + } else { + return create_result(result, resolve, enrich); + } +}; + +MongoDB.prototype.info = function () { + // todo +}; + +MongoDB.prototype.transaction = function (task) { + // not supported + return task.call(this); +}; + +MongoDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + if (flexsearch.map.size) { + let data = []; + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + this.type || (this.type = typeof ids[0]); + for (let j = 0; j < ids.length; j++) { + data.push({ + key: key, + res: i, + id: ids[j] + }); + } + } + } + } + if (data.length) { + await this.db.collection("map" + this.field).insertMany(data); + flexsearch.map.clear(); + } + } + + if (flexsearch.ctx.size) { + let data = []; + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + for (let j = 0; j < ids.length; j++) { + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: ids[j] + }); + } + } + } + } + } + if (data.length) { + await this.db.collection("ctx" + this.field).insertMany(data); + flexsearch.ctx.clear(); + } + } + + if (flexsearch.tag) { + let data = []; + if (flexsearch.tag) { + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let j = 0; j < ids.length; j++) { + data.push({ tag, id: ids[j] }); + } + } + } + if (data.length) { + await this.db.collection("tag" + this.field).insertMany(data); + flexsearch.tag.clear(); + } + } + + let data = []; + if (flexsearch.store) { + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + data.push({ id, doc }); + } + } else if (!flexsearch.bypass) { + for (const id of flexsearch.reg.keys()) { + data.push({ id }); + } + } + if (data.length) { + await this.db.collection("reg").insertMany(data); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); + } + + // TODO + // await this.db.collection("cfg" + this.field).insertOne({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }); +}; + +MongoDB.prototype.remove = function (ids) { + + if (!ids && 0 !== ids) return; + if ("object" != typeof ids) { + ids = [ids]; + } + + // if(this.type !== "string" && typeof ids[0] !== "number"){ + // ids = ids.map(item => parseInt(item, 10)); + // } + + return Promise.all([this.db.collection("map" + this.field).deleteMany({ id: { $in: ids } }), this.db.collection("ctx" + this.field).deleteMany({ id: { $in: ids } }), this.db.collection("tag" + this.field).deleteMany({ id: { $in: ids } }), this.db.collection("reg").deleteMany({ id: { $in: ids } })]); +}; \ No newline at end of file diff --git a/dist/module/db/postgres/index.js b/dist/module/db/postgres/index.js new file mode 100644 index 0000000..33cb63a --- /dev/null +++ b/dist/module/db/postgres/index.js @@ -0,0 +1,971 @@ + + +import pg_promise from "pg-promise"; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { concat, toArray } from "../../common.js"; +const defaults = { + schema: "flexsearch", + user: "postgres", + pass: "postgres", + name: "postgres", + host: "localhost", + port: "5432" +}, + pgp = pg_promise(), + VERSION = 1, + MAXIMUM_QUERY_VARS = 16000, + fields = ["map", "ctx", "reg", "tag", "cfg"], + types = { + text: "text", + char: "text", + varchar: "text", + string: "text", + number: "int", + numeric: "int", + integer: "int", + smallint: "int", + tinyint: "int", + mediumint: "int", + int: "int", + int8: "int", + uint8: "int", + int16: "int", + uint16: "int", + int32: "int", + uint32: "bigint", + int64: "bigint", + bigint: "bigint" +}; + + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +let DB, TRX; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function PostgresDB(name, config = {}) { + if (!(this instanceof PostgresDB)) { + return new PostgresDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = (config.schema ? sanitize(config.schema) : defaults.schema) + (name ? "_" + sanitize(name) : ""); + this.field = config.field ? "_" + sanitize(config.field) : ""; + this.type = config.type ? types[config.type.toLowerCase()] : "text"; + this.support_tag_search = /* tag? */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + if (!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); + this.db = DB || (DB = config.db || null); + Object.assign(defaults, config); + this.db && delete defaults.db; +} + +PostgresDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +PostgresDB.prototype.open = async function () { + + if (!this.db) { + this.db = DB || (DB = pgp(`postgres://${defaults.user}:${encodeURIComponent(defaults.pass)}@${defaults.host}:${defaults.port}/${defaults.name}`)); + } + + const exist = await this.db.oneOrNone(` + SELECT EXISTS ( + SELECT 1 + FROM information_schema.schemata + WHERE schema_name = '${this.id}' + ); + `); + if (!exist || !exist.exists) { + await this.db.none(`CREATE SCHEMA IF NOT EXISTS ${this.id};`); + } + + for (let i = 0; i < fields.length; i++) { + const exist = await this.db.oneOrNone(` + SELECT EXISTS ( + SELECT 1 FROM pg_tables + WHERE schemaname = '${this.id}' AND tablename = '${fields[i] + ("reg" !== fields[i] ? this.field : "")}' + ); + `); + if (exist && exist.exists) continue; + + const type = "text" === this.type ? "varchar(128)" : this.type; + + switch (fields[i]) { + case "map": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.map${this.field}( + key varchar(128) NOT NULL, + res smallint NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_map_index${this.field} + ON ${this.id}.map${this.field} (key); + CREATE INDEX IF NOT EXISTS ${this.id}_map_id${this.field} + ON ${this.id}.map${this.field} (id); + `); + break; + + case "ctx": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.ctx${this.field}( + ctx varchar(128) NOT NULL, + key varchar(128) NOT NULL, + res smallint NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_ctx_index${this.field} + ON ${this.id}.ctx${this.field} (ctx, key); + CREATE INDEX IF NOT EXISTS ${this.id}_ctx_id${this.field} + ON ${this.id}.ctx${this.field} (id); + `); + break; + + case "tag": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.tag${this.field}( + tag varchar(128) NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_tag_index${this.field} + ON ${this.id}.tag${this.field} (tag); + CREATE INDEX IF NOT EXISTS ${this.id}_tag_id${this.field} + ON ${this.id}.tag${this.field} (id); + `); + break; + + case "reg": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.reg( + id ${type} NOT NULL + CONSTRAINT ${this.id}_reg_pk + PRIMARY KEY, + doc text DEFAULT NULL + ); + `).catch(() => { + // document indexes will try to create same registry table + // and the "IF NOT EXISTS" did not apply on parallel + }); + break; + + case "cfg": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.cfg${this.field}( + cfg text NOT NULL + ); + `); + break; + } + } + + return this.db; +}; + +PostgresDB.prototype.close = function () { + this.db.close && this.db.close(); + this.db = DB = null; + return this; +}; + +PostgresDB.prototype.destroy = async function () { + await this.db.none(` + DROP TABLE IF EXISTS ${this.id}.map${this.field}; + DROP TABLE IF EXISTS ${this.id}.ctx${this.field}; + DROP TABLE IF EXISTS ${this.id}.tag${this.field}; + DROP TABLE IF EXISTS ${this.id}.cfg${this.field}; + DROP TABLE IF EXISTS ${this.id}.reg; + `); + this.close(); +}; + +PostgresDB.prototype.clear = function () { + return this.db.none(` + TRUNCATE TABLE ${this.id}.map${this.field}; + TRUNCATE TABLE ${this.id}.ctx${this.field}; + TRUNCATE TABLE ${this.id}.tag${this.field}; + TRUNCATE TABLE ${this.id}.cfg${this.field}; + TRUNCATE TABLE ${this.id}.reg; + `); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + for (let i = 0; i < rows.length; i++) { + if (enrich) { + if (rows[i].doc) { + rows[i].doc = JSON.parse(rows[i].doc); + } + } else { + rows[i] = rows[i].id; + } + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +PostgresDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = !1, tags) { + let rows, + stmt = '', + params = ctx ? [ctx, key] : [key], + table = this.id + (ctx ? ".ctx" : ".map") + this.field; + + if (tags) { + for (let i = 0, count = params.length + 1; i < tags.length; i += 2) { + stmt += ` AND ${table}.id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = $${count++})`; + params.push(tags[i + 1]); + } + } + if (ctx) { + rows = this.db.any(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE ctx = $1 AND key = $2 ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, params); + } else { + rows = this.db.any(` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE key = $1 ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, params); + } + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +PostgresDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const table = this.id + ".tag" + this.field, + promise = this.db.any(` + SELECT ${table}.id + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = ${table}.id + ` : ""} + WHERE tag = $1 + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""}`, [tag]); + + enrich || promise.then(function (rows) { + return create_result(rows, !0, !1); + }); + return promise; +}; + +PostgresDB.prototype.enrich = async function (ids) { + let result = []; + if ("object" != typeof ids) { + ids = [ids]; + } + for (let count = 0; count < ids.length;) { + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + let stmt = ""; + for (let i = 1; i <= chunk.length; i++) { + stmt += (stmt ? "," : "") + "$" + i; + } + const res = await this.db.any(` + SELECT id, doc + FROM ${this.id}.reg + WHERE id IN (${stmt})`, ids); + if (res && res.length) { + for (let i = 0, doc; i < res.length; i++) { + if (doc = res[i].doc) { + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + return 1 === result.length ? result[0] : 1 < result.length ? concat(result) : result; +}; + +PostgresDB.prototype.has = function (id) { + return this.db.oneOrNone("SELECT EXISTS(SELECT 1 FROM " + this.id + ".reg WHERE id = $1)", [id]); +}; + +PostgresDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let rows; + + if (1 < query.length && flexsearch.depth) { + let where = "", + params = [], + keyword = query[0], + term, + count = 1; + + // variant new + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + where += (where ? " OR " : "") + `(ctx = $${count++} AND key = $${count++})`; + params.push(swap ? term : keyword, swap ? keyword : term); + keyword = term; + } + + if (tags) { + where = "(" + where + ")"; + for (let i = 0; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = $${count++})`; + params.push(tags[i + 1]); + } + } + + rows = this.db.any(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.ctx${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + (query.length - 1)} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, params); + + // variant 1 + + // for(let i = 1, count = 1; i < query.length; i++){ + // where += (where ? " UNION " : "") + ` + // SELECT id, res + // FROM ${this.id}.ctx${this.field} + // WHERE ctx = $${count++} AND key = $${count++} + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params.push( + // swap ? term : keyword, + // swap ? keyword : term + // ); + // keyword = term; + // } + // + // rows = await db.any(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, params); + } else { + let where = "", + count = 1, + query_length = query.length; + + for (let i = 0; i < query_length; i++) { + where += (where ? "," : "") + "$" + count++; + } + where = "key " + (1 < query_length ? "IN (" + where + ")" : "= " + where); + + if (tags) { + where = "(" + where + ")"; + for (let i = 0; i < tags.length; i += 2) { + where += ` AND id IN (SELECT id FROM ${this.id}.tag_${sanitize(tags[i])} WHERE tag = $${count++})`; + query.push(tags[i + 1]); + } + } + + rows = this.db.any(` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM ${this.id}.map${this.field} + WHERE ${where} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN ${this.id}.reg ON ${this.id}.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + query_length} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, query); + + // variant 1 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " UNION " : "") + ` + // SELECT id, res + // FROM ${ this.id }.map${ this.field } + // WHERE key = $${i} + // `; + // } + // rows = await db.any(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, query); + + // variant 2 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " AND EXISTS " : "") + `(SELECT FROM ${this.id}.map${this.field} as d WHERE d.id = t.id AND d.key = $` + i + ")"; + // } + // rows = await db.any(` + // SELECT t.id, min(t.res) + // FROM ${this.id}.map${this.field} AS t + // WHERE EXISTS ${where} + // GROUP BY t.id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + + // variant 3 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " INTERSECT " : "") + `SELECT id FROM ${this.id}.map${this.field} WHERE key = $` + i; + // } + // rows = await db.any(` + // WITH filtering (id) AS(${where}) + // SELECT t.id, min(t.res) + // FROM ${this.id}.map${this.field} AS t + // JOIN filtering USING (id) + // GROUP BY t.id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + + // variant 4 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " INTERSECT " : "") + `SELECT id FROM ${this.id}.map${this.field} WHERE key = $` + i; + // } + // rows = await db.any(` + // SELECT id, min(res) + // FROM ${this.id}.map${this.field} + // WHERE id IN (${where}) + // GROUP BY id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + } + + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +PostgresDB.prototype.info = function () { + // todo +}; + +// PostgresDB.prototype.transaction = async function(task){ +// const self = this; +// if(TRX){ +// return TRX.then(function(){ +// return self.transaction(task); +// //task.call(self, TRX); +// }); +// } +// TRX = await this.db.tx(async function(trx){ +// await task.call(self, trx); +// }); +// TRX = null; +// }; + +PostgresDB.prototype.transaction = function (task) { + if (TRX) { + return task.call(this, TRX); + } + const self = this; + return (TRX || this.db).tx(function (trx) { + return task.call(self, TRX = trx); + }).finally(function () { + TRX = null; + }); +}; + +PostgresDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + + //console.log("tasks:", tasks) + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction(function (trx) { + + const batch = []; + + // Datastore + Registry + + if (flexsearch.store) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["id", "doc"], { + table: this.id + ".reg" + }); + + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + // const migration = checkMigration.call(this, id); + // migration && await migration; + data.push({ id, doc: doc && JSON.stringify(doc) }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + // while(data.length){ + // let next; + // if(data.length > MAXIMUM_QUERY_VARS){ + // next = data.slice(MAXIMUM_QUERY_VARS); + // data = data.slice(0, MAXIMUM_QUERY_VARS); + // } + // let insert = pgp.helpers.insert(data, stmt); + // trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')); + // if(next) data = next; + // else break; + // } + } + + // Registry Only + + else if (!flexsearch.bypass) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["id"], { + table: this.id + ".reg" + }); + + for (const id of flexsearch.reg.keys()) { + // const migration = checkMigration.call(this, id); + // migration && await migration; + data.push({ id }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Default Index + + if (flexsearch.map.size) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["key", "res", "id"], { + table: this.id + ".map" + this.field + }); + + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + //this.type || (this.type = typeof ids[0]); + // let stmt = "($1,$2,$3)"; + // let params = [key, i, ids[0]]; + for (let j = 0; j < ids.length; j++) { + // stmt += ",($1,$2,$3)"; + // params.push(key, i, ids[j]); + //trx.none(`INSERT INTO ${config.schema}.map${self.field} (key, res, id) VALUES ($1,$2,$3)`, [key, i, ids[j]]); + data.push({ + key: key, + res: i, + id: ids[j] + }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + } + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Context Index + + if (flexsearch.ctx.size) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["ctx", "key", "res", "id"], { + table: this.id + ".ctx" + this.field + }); + + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + // let stmt = "(?,?,?)"; + // let params = [ctx_key + ":" + key, i, ids[0]]; + for (let j = 0; j < ids.length; j++) { + // stmt += ",(?,?,?)"; + // params.push(ctx_key + ":" + key, i, ids[j]); + //trx.none("INSERT INTO " + config.schema + ".ctx" + self.field + " (ctx, key, res, id) VALUES ($1,$2,$3,$4)", [ctx_key, key, i, ids[j]]); + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: ids[j] + }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + } + } + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Tag Index + + if (flexsearch.tag) { + let data = [], + stmt = new pgp.helpers.ColumnSet(["tag", "id"], { + table: this.id + ".tag" + this.field + }); + + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let j = 0; j < ids.length; j++) { + data.push({ tag, id: ids[j] }); + if (data.length === MAXIMUM_QUERY_VARS) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + data = []; + } + } + } + if (data.length) { + let insert = pgp.helpers.insert(data, stmt); + batch.push(trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2'))); + } + } + + // Field Configuration + + // TODO + // trx.none("INSERT INTO " + this.id + ".cfg" + this.field + " (cfg) VALUES ($1)", [ + // JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }) + // ]); + + //return Promise.all(batch); + + if (batch.length) { + return trx.batch(batch); + } + }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); +}; + +PostgresDB.prototype.remove = function (ids) { + + if (!ids && 0 !== ids) { + return; + } + if ("object" != typeof ids) { + ids = [ids]; + } + if (!ids.length) { + return; + } + + // ids = [ids]; + // return this.db.none( + // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1);", [ids] + // ); + + // ids = [ids]; + // return Promise.all([ + // this.db.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids) + // ]); + + return this.transaction(function (trx) { + + //console.log("remove:", ids); + + // ids = [ids]; + // trx.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids); + + // ids = [ids]; + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", ids); + // return; + + // return trx.none( + // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1);", [ids] + // ); + + // while(ids.length){ + // let next; + // let stmt = ""; + // let chunk = 0; + // let migration; + // for(let i = 0; i < ids.length; i++){ + // stmt += (stmt ? "," : "") + "$" + (i + 1); // + "::text"; + // // maximum count of variables supported + // if(++chunk === MAXIMUM_QUERY_VARS){ + // next = ids.slice(MAXIMUM_QUERY_VARS); + // ids = ids.slice(0, MAXIMUM_QUERY_VARS); + // break; + // } + // } + // + // trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id IN (" + stmt + ")", ids) + // ]); + // + // // trx.none( + // // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".reg WHERE id IN (" + stmt + ");", ids + // // ); + // + // if(next) ids = next; + // else break; + // } + + ids = [ids]; + return trx.batch([trx.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), trx.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), trx.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), trx.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids)]); + + // ids = [ids]; + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", ids) + // ]); + + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id IN ($1:csv)", [ids]) + // ]); + + // const type = self.type === "text" ? "text" : "int"; + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + self.field + " WHERE id = ANY($1::" + type + "[])", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + self.field + " WHERE id = ANY($1::" + type + "[])", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY($1::" + type + "[])", [ids]) + // ]); + + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + self.field + " WHERE id = ANY($1)", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + self.field + " WHERE id = ANY($1)", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY($1)", [ids]) + // ]); + }); +}; + +// if(this.type === "int" && typeof ids[0] === "string"){ +// ids = ids.map(item => parseInt(item, 10)); +// } +// if(this.type === "bigint" && typeof ids[0] === "string"){ +// ids = ids.map(item => BigInt(item)); +// } +// if(this.type === "string" && typeof ids[0] === "number"){ +// ids = ids.map(item => item + ""); +// } +// this.type !== "string" && checkMigration.call(this, ids[0]); + +// function checkMigration(id){ +// if(this.type !== "string"){ +// if(typeof id === "object"){ +// id = id[0]; +// } +// if(typeof id === "string"){ +// this.type = "string"; +// return upgradeTextId.call(this); +// } +// if(this.type !== "bigint"){ +// if(id > 2 ** 31 - 1){ +// this.type = "bigint"; +// return upgradeBigIntId.call(this); +// } +// } +// } +// } +// +// function upgradeTextId(){ +// return this.db.none(` +// ALTER TABLE ${this.id}.map${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.ctx${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.tag${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.reg +// ALTER COLUMN id type varchar(128) +// USING id::text; +// `); +// } +// +// function upgradeBigIntId(){ +// return this.db.none(` +// ALTER TABLE ${this.id}.map${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.ctx${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.tag${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.reg +// ALTER COLUMN id type bigint +// USING id::bigint; +// `); +// } \ No newline at end of file diff --git a/dist/module/db/redis/index.js b/dist/module/db/redis/index.js new file mode 100644 index 0000000..dbf7c83 --- /dev/null +++ b/dist/module/db/redis/index.js @@ -0,0 +1,500 @@ + + +import { createClient } from "redis"; +const defaults = { + host: "localhost", + port: "6379", + user: null, + pass: null +}, + VERSION = 1, + fields = ["map", "ctx", "reg", "tag", "doc", "cfg"]; + +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +let DB, TRX; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function RedisDB(name, config = {}) { + if (!(this instanceof RedisDB)) { + return new RedisDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + this.id = (name ? sanitize(name) : "flexsearch") + "|"; + this.field = config.field ? "-" + sanitize(config.field) : ""; + this.type = config.type || ""; + this.fastupdate = /* tag? */ /* stringify */ /* stringify */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + this.db = config.db || DB || null; + this.support_tag_search = !0; + //this.trx = false; + Object.assign(defaults, config); + this.db && delete defaults.db; +} + +// RedisDB.mount = function(flexsearch){ +// return new this().mount(flexsearch); +// }; + +RedisDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + // todo support + //this.fastupdate = flexsearch.fastupdate; + return this.open(); +}; + +RedisDB.prototype.open = async function () { + if (this.db) { + return this.db; + } + + let url = defaults.url; + if (!url) { + url = defaults.user ? `redis://${defaults.user}:${defaults.pass}@${defaults.host}:${defaults.port}` : `redis://${defaults.host}:${defaults.port}`; + } + return this.db = await createClient(url).on("error", err => console.error(err)).connect(); +}; + +RedisDB.prototype.close = async function () { + await this.db.disconnect(); // this.db.client.disconnect(); + this.db = null; + return this; +}; + +RedisDB.prototype.clear = function () { + return this.db.unlink([this.id + "map" + this.field, this.id + "ctx" + this.field, this.id + "tag" + this.field, this.id + "cfg" + this.field, this.id + "doc", this.id + "reg"]); +}; + +function create_result(range, type, resolve, enrich) { + if (resolve) { + for (let i = 0, tmp, id; i < range.length; i++) { + tmp = range[i]; + id = "number" === type ? parseInt(tmp.value || tmp, 10) : tmp.value || tmp; + range[i] = /*enrich + ? { id, doc: tmp.doc } + :*/id; + } + return range; + } else { + let result = []; + for (let i = 0, tmp, id, score; i < range.length; i++) { + tmp = range[i]; + id = "number" === type ? parseInt(tmp.value, 10) : tmp.value; + score = tmp.score; + result[score] || (result[score] = []); + result[score].push(enrich ? { id, doc: tmp.doc } : id); + } + return result; + } +} + +RedisDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = /* suggest */!1, tags) { + + if (tags) { + // flexsearch dummy + const query = ctx ? [ctx, key] : [key]; + // keyword first + return this.search({ depth: !!ctx }, query, limit, offset, !1, resolve, enrich, tags); + } + + const type = this.type, + self = this; + + let result; + + if (ctx) { + result = this.db[resolve ? "zRange" : "zRangeWithScores"](this.id + "ctx" + this.field + ":" + ctx + ":" + key, "" + offset, "" + (offset + limit - 1), { REV: !0 }); + } else { + result = this.db[resolve ? "zRange" : "zRangeWithScores"](this.id + "map" + this.field + ":" + key, "" + offset, "" + (offset + limit - 1), { REV: !0 }); + } + + return result.then(async function (range) { + if (!range.length) return range; + if (enrich) range = await self.enrich(range); + return create_result(range, type, resolve, enrich); + }); +}; + +RedisDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const self = this; + return this.db.sMembers(this.id + "tag" + this.field + ":" + tag).then(function (ids) { + if (!ids || !ids.length || offset >= ids.length) return []; + if (!limit && !offset) return ids; + const result = ids.slice(offset, offset + limit); + return enrich ? self.enrich(result) : result; + }); +}; + +RedisDB.prototype.enrich = function (ids) { + if ("object" != typeof ids) { + ids = [ids]; + } + return this.db.hmGet(this.id + "doc", ids).then(function (res) { + for (let i = 0; i < res.length; i++) { + res[i] = { + id: ids[i], + doc: res[i] && JSON.parse(res[i]) + }; + } + return res; + }); +}; + +RedisDB.prototype.has = function (id) { + return this.db.sIsMember(this.id + "reg", "" + id); +}; + +RedisDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let result; + + if (1 < query.length && flexsearch.depth) { + + const key = this.id + "ctx" + this.field + ":"; + let params = [], + keyword = query[0], + term; + + + for (let i = 1, swap; i < query.length; i++) { + term = query[i]; + swap = flexsearch.bidirectional && term > keyword; + params.push(key + (swap ? term : keyword) + ":" + (swap ? keyword : term)); + keyword = term; + } + query = params; + } else { + + const key = this.id + "map" + this.field + ":"; + for (let i = 0; i < query.length; i++) { + query[i] = key + query[i]; + } + } + + const type = this.type; + let key = this.id + "tmp:" + Math.random(); + + + if (suggest) { + if (tags) for (let i = 0; i < tags.length; i += 2) { + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + + const multi = this.db.multi().zUnionStore(key, query, { AGGREGATE: "SUM" }); + // Strict Tag Intersection: it does not put tags into union, instead it calculates the + // intersection against the term match union. This was the default behavior + // of Tag-Search. But putting everything into union will also provide suggestions derived + // from tags when no term was matched. + + result = multi[resolve ? "zRange" : "zRangeWithScores"](key, "" + offset, "" + (offset + limit - 1), { REV: !0 }).unlink(key).exec(); + } else { + if (tags) for (let i = 0; i < tags.length; i += 2) { + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + result = this.db.multi().zInterStore(key, query, { AGGREGATE: "MIN" })[resolve ? "zRange" : "zRangeWithScores"](key, "" + offset, "" + (offset + limit - 1), { REV: !0 }).unlink(key).exec(); + } + + const self = this; + return result.then(async function (range) { + range = suggest && tags + // take the 3rd result from batch return + ? range[2] + // take the 2nd result from batch return + : range[1]; + if (!range.length) return range; + if (enrich) range = await self.enrich(range); + return create_result(range, type, resolve, enrich); + }); +}; + +RedisDB.prototype.info = function () { + // todo +}; + +RedisDB.prototype.transaction = async function (task, callback) { + + if (TRX) { + return task.call(this, TRX); + } + + TRX = this.db.multi(); + let promise1 = /*await*/task.call(this, TRX), + promise2 = TRX.exec(); + + TRX = null; + callback && promise.then(callback); + await Promise.all([promise1, promise2]); +}; + +RedisDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = "" + task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg, !0)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction(function (trx) { + + let refs = new Map(); + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + + let result = []; + for (let j = 0; j < ids.length; j++) { + result.push({ + score: i, + value: "" + ids[j] + }); + } + if ("number" == typeof ids[0]) { + this.type = "number"; + } + + const ref = this.id + "map" + this.field + ":" + key; + trx.zAdd(ref, result); + // if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + // trx.sAdd("ref" + this.field + ":" + ids[j], ref); + // } + if (this.fastupdate) for (let j = 0, id; j < ids.length; j++) { + // Map performs bad when pushing numeric-like values as key + // id = ids[j]; + // let tmp = refs.get(id); + // tmp || refs.set(id, tmp = []); + // tmp.push(ref); + id = ids[j]; + let tmp = refs.get(id); + tmp || refs.set(id, tmp = []); + tmp.push(ref); + } + } + } + } + // if(this.fastupdate) for(let item of refs){ + // const key = item[0]; + // const value = item[1]; + // trx.sAdd("ref" + this.field + ":" + key, value); + // } + if (this.fastupdate) for (const item of refs) { + const key = item[0], + value = item[1]; + + trx.sAdd(this.id + "ref" + this.field + ":" + key, value); + } + + refs = new Map(); + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + let result = []; + for (let j = 0; j < ids.length; j++) { + result.push({ score: i, value: "" + ids[j] }); + } + if ("number" == typeof ids[0]) { + this.type = "number"; + } + const ref = this.id + "ctx" + this.field + ":" + ctx_key + ":" + key; + trx.zAdd(ref, result); + // if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + // trx.sAdd("ref" + this.field + ":" + ids[j], ref); + // } + if (this.fastupdate) for (let j = 0, id; j < ids.length; j++) { + // Map performs bad when pushing numeric-like values as key + // id = ids[j]; + // let tmp = refs.get(id); + // tmp || refs.set(id, tmp = []); + // tmp.push(ref); + id = ids[j]; + let tmp = refs.get(id); + tmp || refs.set(id, tmp = []); + tmp.push(ref); + } + } + } + } + } + + if (this.fastupdate) for (const item of refs) { + const key = item[0], + value = item[1]; + + trx.sAdd(this.id + "ref" + this.field + ":" + key, value); + } + + if (flexsearch.store) { + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + doc && trx.hSet(this.id + "doc", "" + id, JSON.stringify(doc)); + } + } + if (!flexsearch.bypass) { + let ids = toArray(flexsearch.reg, !0); + if (ids.length) { + trx.sAdd(this.id + "reg", ids); + } + } + + if (flexsearch.tag) { + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + let result = []; + // for(let i = 0; i < ids.length; i++){ + // result.push({ + // score: 0, + // value: "" + ids[i] + // }); + // } + if ("number" == typeof ids[0]) { + for (let i = 0; i < ids.length; i++) { + result[i] = "" + ids[i]; + } + } else { + result = ids; + } + trx.sAdd(this.id + "tag" + this.field + ":" + tag, result); + } + } + + // TODO + // trx.set(this.id + "cfg" + this.field, JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // })); + }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); +}; + +RedisDB.prototype.remove = function (ids) { + + if (!ids && 0 !== ids) { + return; + } + if ("object" != typeof ids) { + ids = [ids]; + } + if (!ids.length) { + return; + } + + return this.transaction(async function (trx) { + + while (ids.length) { + let next; + if (10000 < ids.length) { + next = ids.slice(10000); + ids = ids.slice(0, 10000); + } + + if ("number" == typeof ids[0]) { + for (let i = 0; i < ids.length; i++) { + ids[i] = "" + ids[i]; + } + } + + const check = await this.db.smIsMember(this.id + "reg", ids); + + for (let i = 0, id; i < ids.length; i++) { + if (!check[i]) continue; + id = "" + ids[i]; + + if (this.fastupdate) { + // const refs = new Map(); + const ref = await this.db.sMembers(this.id + "ref" + this.field + ":" + id); + if (ref) { + for (let j = 0; j < ref.length; j++) { + // let tmp = refs.get(ref[j]); + // tmp || refs.set(ref[j], tmp = []); + // tmp.push(id); + trx.zRem(ref[j], id); + } + trx.unlink(this.id + "ref" + this.field + ":" + id); + } + // for(let item of refs){ + // //console.log(item[0], item[1]) + // trx.zRem(item[0], item[1]); + // } + } + + trx.hDel(this.id + "doc", id); + trx.sRem(this.id + "reg", id); + } + + if (next) ids = next;else break; + } + }); +}; \ No newline at end of file diff --git a/dist/module/db/sqlite/index.js b/dist/module/db/sqlite/index.js new file mode 100644 index 0000000..c71a6dd --- /dev/null +++ b/dist/module/db/sqlite/index.js @@ -0,0 +1,794 @@ + + +//const sqlite3 = require("sqlite3").verbose(); +import sqlite3 from "sqlite3"; +import path from "path"; +import StorageInterface from "../interface.js"; +import { concat, toArray } from "../../common.js"; +import Document from "../../document.js"; + +const VERSION = 1, + MAXIMUM_QUERY_VARS = 16000, + fields = ["map", "ctx", "reg", "tag", "cfg"], + types = { + text: "text", + char: "text", + varchar: "text", + string: "text", + number: "int", + numeric: "int", + integer: "int", + smallint: "int", + tinyint: "int", + mediumint: "int", + int: "int", + int8: "int", + uint8: "int", + int16: "int", + uint16: "int", + int32: "int", + uint32: "bigint", + int64: "bigint", + bigint: "bigint" +}; + + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +// global transaction to keep track of database lock +const TRX = Object.create(null), + DB = Object.create(null); + + +/** + * @constructor + * @implements StorageInterface + */ + +export default function SqliteDB(name, config = {}) { + if (!(this instanceof SqliteDB)) { + return new SqliteDB(name, config); + } + if ("object" == typeof name) { + name = name.name; + config = name; + } + if (!name) { + console.info("Default storage space was used, because a name was not passed."); + } + //field = "Test-456"; + this.id = config.path || (":memory:" === name ? name : "flexsearch" + (name ? "-" + sanitize(name) : "") + ".sqlite"); + this.field = config.field ? "_" + sanitize(config.field) : ""; + this.support_tag_search = /* tag? */ /* stringify */ /* stringify */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + this.db = config.db || DB[this.id] || null; + this.type = config.type ? types[config.type.toLowerCase()] : "string"; + if (!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); +} + +SqliteDB.prototype.mount = function (flexsearch) { + if (flexsearch instanceof Document) { + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +SqliteDB.prototype.open = async function () { + + if (!this.db) { + + if (!(this.db = DB[this.id])) { + + let filepath = this.id; + if (":memory:" !== filepath) { + // skip absolute path + if ("/" !== filepath[0] && "\\" !== filepath[0]) { + // current working directory + const dir = process.cwd(); + filepath = path.join(dir, this.id); + } + } + + this.db = DB[this.id] = new sqlite3.Database(filepath); + } + } + + const db = this.db; + + for (let i = 0; i < fields.length; i++) { + const exist = await this.promisfy({ + method: "get", + stmt: "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?) as exist", + params: [fields[i] + ("reg" === fields[i] ? "" : this.field)] + }); + if (!exist || !exist.exist) { + let stmt, stmt_index; + switch (fields[i]) { + case "map": + stmt = ` + CREATE TABLE IF NOT EXISTS main.map${this.field}( + key TEXT NOT NULL, + res INTEGER NOT NULL, + id ${this.type} NOT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS map_key_index${this.field} + ON map${this.field} (key); + CREATE INDEX IF NOT EXISTS map_id_index${this.field} + ON map${this.field} (id); + `; + break; + + case "ctx": + stmt = ` + CREATE TABLE IF NOT EXISTS main.ctx${this.field}( + ctx TEXT NOT NULL, + key TEXT NOT NULL, + res INTEGER NOT NULL, + id ${this.type} NOT NULL + ); + + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS ctx_key_index${this.field} + ON ctx${this.field} (ctx, key); + CREATE INDEX IF NOT EXISTS ctx_id_index${this.field} + ON ctx${this.field} (id); + `; + break; + + case "tag": + stmt = ` + CREATE TABLE IF NOT EXISTS main.tag${this.field}( + tag TEXT NOT NULL, + id ${this.type} NOT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS tag_index${this.field} + ON tag${this.field} (tag); + CREATE INDEX IF NOT EXISTS tag_id_index${this.field} + ON tag${this.field} (id); + `; + break; + + case "reg": + stmt = ` + CREATE TABLE IF NOT EXISTS main.reg( + id ${this.type} NOT NULL + CONSTRAINT reg_pk${this.field} + PRIMARY KEY, + doc TEXT DEFAULT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS reg_index + ON reg (id); + `; + break; + + case "cfg": + stmt = ` + CREATE TABLE IF NOT EXISTS main.cfg${this.field} ( + cfg TEXT NOT NULL + ); + `; + break; + } + + await new Promise(function (resolve, reject) { + db.exec(stmt, function (err, rows) { + if (err) return reject(err); + stmt_index ? db.exec(stmt_index, function (err, rows) { + if (err) return reject(err); + resolve(rows); + }) : resolve(rows); + }); + }); + } + } + + db.exec("PRAGMA optimize = 0x10002"); + + return db; +}; + +SqliteDB.prototype.close = function () { + this.db.close(); + this.db = null; + return this; +}; + +SqliteDB.prototype.destroy = async function () { + await this.transaction(function () { + this.db.run("DROP TABLE IF EXISTS main.map" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.ctx" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.tag" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.cfg" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.reg;"); + }); + this.close(); +}; + +SqliteDB.prototype.clear = function () { + return this.transaction(function () { + this.db.run("DELETE FROM main.map" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.ctx" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.tag" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.cfg" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.reg WHERE 1;"); + }); +}; + +function create_result(rows, resolve, enrich) { + if (resolve) { + for (let i = 0; i < rows.length; i++) { + if (enrich) { + if (rows[i].doc) { + rows[i].doc = JSON.parse(rows[i].doc); + } + } else { + rows[i] = rows[i].id; + } + } + return rows; + } else { + const arr = []; + for (let i = 0, row; i < rows.length; i++) { + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich ? row : row.id); + } + return arr; + } +} + +SqliteDB.prototype.get = function (key, ctx, limit = 0, offset = 0, resolve = !0, enrich = /* suggest */!1, tags) { + let result, + stmt = '', + params = ctx ? [ctx, key] : [key], + table = "main." + (ctx ? "ctx" : "map") + this.field; + + if (tags) { + for (let i = 0; i < tags.length; i += 2) { + stmt += ` AND ${table}.id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + params.push(tags[i + 1]); + } + } + if (ctx) { + result = this.promisfy({ + method: "all", + stmt: ` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${table}.id + ` : ""} + WHERE ctx = ? AND key = ? ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params + }); + } else { + result = this.promisfy({ + method: "all", + stmt: ` + SELECT ${table}.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${table}.id + ` : ""} + WHERE key = ? ${stmt} + ORDER BY res + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params + }); + } + return result.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +SqliteDB.prototype.tag = function (tag, limit = 0, offset = 0, enrich = !1) { + const table = "main.tag" + this.field, + promise = this.promisfy({ + method: "all", + stmt: ` + SELECT ${table}.id + ${enrich ? ", doc" : ""} + FROM ${table} + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${table}.id + ` : ""} + WHERE tag = ? + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params: [tag] + }); + + enrich || promise.then(function (rows) { + return create_result(rows, !0, !1); + }); + return promise; +}; + +function build_params(length) { + + let stmt = ",?"; + for (let i = 1; i < length;) { + if (i <= length - i) { + stmt += stmt; + i *= 2; + } else { + stmt += stmt.substring(0, 2 * (length - i)); + break; + } + } + return stmt.substring(1); +} + +SqliteDB.prototype.enrich = function (ids) { + const result = [], + promises = []; + + if ("object" != typeof ids) { + ids = [ids]; + } + + for (let count = 0; count < ids.length;) { + + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + + // let stmt = "?"; + // for(let i = 1; i < chunk.length; i++){ + // stmt += ",?"; + // } + + // 10x faster string concatenation + let stmt = build_params(chunk.length); + + promises.push(this.promisfy({ + method: "all", + stmt: `SELECT id, doc FROM main.reg WHERE id IN (${stmt})`, + params: chunk + })); + } + + return Promise.all(promises).then(function (promises) { + + for (let i = 0, res; i < promises.length; i++) { + res = promises[i]; + if (res && res.length) { + for (let i = 0, doc; i < res.length; i++) { + if (doc = res[i].doc) { + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + + return 1 === result.length ? result[0] : 1 < result.length ? concat(result) : result; + }); +}; + +SqliteDB.prototype.has = function (id) { + return this.promisfy({ + method: "get", + stmt: `SELECT EXISTS(SELECT 1 FROM main.reg WHERE id = ?) as exist`, + params: [id] + }).then(function (result) { + return result && result.exist; + }); +}; + +SqliteDB.prototype.search = function (flexsearch, query, limit = 100, offset = 0, suggest = !1, resolve = !0, enrich = !1, tags) { + + let rows; + + if (1 < query.length && flexsearch.depth) { + let stmt = "", + params = [], + keyword = query[0], + term; + + + for (let i = 1; i < query.length; i++) { + term = query[i]; + const swap = flexsearch.bidirectional && term > keyword; + stmt += (stmt ? " OR " : "") + `(ctx = ? AND key = ?)`; + params.push(swap ? term : keyword, swap ? keyword : term); + keyword = term; + } + + if (tags) { + stmt = "(" + stmt + ")"; + for (let i = 0; i < tags.length; i += 2) { + stmt += ` AND id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + params.push(tags[i + 1]); + } + } + + rows = this.promisfy({ + method: "all", + stmt: ` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM main.ctx${this.field} + WHERE ${stmt} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + (query.length - 1)} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params + }); + + // variant 1 + // for(let i = 1; i < query.length; i++){ + // stmt += (stmt ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM main.ctx${this.field} + // WHERE ctx = ? AND key = ? + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params.push(swap ? term : keyword, swap ? keyword : term); + // keyword = term; + // } + // + // rows = await this.promisfy({ + // method: "all", + // stmt: ` + // SELECT id/*, res */ + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${stmt}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, + // params + // }); + } else { + let stmt = "", + query_length = query.length; + + for (let i = 0; i < query_length; i++) { + stmt += (stmt ? " OR " : "") + `key = ?`; + } + + if (tags) { + stmt = "(" + stmt + ")"; + for (let i = 0; i < tags.length; i += 2) { + stmt += ` AND id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + query.push(tags[i + 1]); + } + } + + rows = this.promisfy({ + method: "all", + stmt: ` + SELECT r.id + ${resolve ? "" : ", res"} + ${enrich ? ", doc" : ""} + FROM ( + SELECT id, count(*) as count, + ${suggest ? "SUM" : "MIN"}(res) as res + FROM main.map${this.field} + WHERE ${stmt} + GROUP BY id + ) as r + ${enrich ? ` + LEFT JOIN main.reg ON main.reg.id = r.id + ` : ""} + ${suggest ? "" : "WHERE count = " + query_length} + ORDER BY ${suggest ? "count DESC, res" : "res"} + ${limit ? "LIMIT " + limit : ""} + ${offset ? "OFFSET " + offset : ""} + `, + params: query + }); + + // variant 1 + // for(let i = 0; i < query.length; i++){ + // stmt += (stmt ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM main.map${ this.field } + // WHERE key = ? + // `; + // } + // + // rows = await this.promisfy({ + // method: "all", + // stmt: ` + // SELECT id/*, res*/ + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${stmt}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, + // params: query + // }); + } + + return rows.then(function (rows) { + return create_result(rows, resolve, enrich); + }); +}; + +SqliteDB.prototype.info = function () { + // todo +}; + +SqliteDB.prototype.transaction = function (task, callback) { + + const self = this; + + if (TRX[this.id]) { + return TRX[this.id].then(function () { + return self.transaction(task, callback); + }); + } + + const db = this.db; + + return TRX[this.id] = new Promise(function (resolve, reject) { + db.exec("PRAGMA optimize"); + db.exec('PRAGMA busy_timeout = 5000'); + db.exec("BEGIN"); + db.parallelize(function () { + task.call(self); + }); + db.exec("COMMIT", function (err, rows) { + TRX[self.id] = null; + if (err) return reject(err); + callback && callback(rows); + resolve(rows); + db.exec("PRAGMA shrink_memory"); + }); + }); +}; + +SqliteDB.prototype.commit = async function (flexsearch, _replace, _append) { + + // process cleanup tasks + if (_replace) { + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } else { + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for (let i = 0, task; i < tasks.length; i++) { + task = tasks[i]; + // there are just removals in the task queue + if (task.clear) { + await this.clear(); + _replace = !0; + break; + } else { + tasks[i] = task.del; + } + } + if (!_replace) { + if (!_append) { + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && (await this.remove(tasks)); + } + } + + if (!flexsearch.reg.size) { + return; + } + + await this.transaction(function () { + + for (const item of flexsearch.map) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + let stmt = "", + params = []; + + + for (let j = 0; j < ids.length; j++) { + stmt += (stmt ? "," : "") + "(?,?,?)"; + params.push(key, i, ids[j]); + // maximum count of variables supported + if (j === ids.length - 1 || params.length + 3 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.map" + this.field + " (key, res, id) VALUES " + stmt, params); + stmt = ""; + params = []; + } + //this.db.run("INSERT INTO map (key, res, id) VALUES (?, ?, ?) ON CONFLICT DO NOTHING", [key, i, ids[j]]); + } + } + } + } + //}); + //await this.transaction(function(){ + + for (const ctx of flexsearch.ctx) { + const ctx_key = ctx[0], + ctx_value = ctx[1]; + + + for (const item of ctx_value) { + const key = item[0], + arr = item[1]; + + + for (let i = 0, ids; i < arr.length; i++) { + if ((ids = arr[i]) && ids.length) { + let stmt = "", + params = []; + + + for (let j = 0; j < ids.length; j++) { + stmt += (stmt ? "," : "") + "(?,?,?,?)"; + params.push(ctx_key, key, i, ids[j]); + // maximum count of variables supported + if (j === ids.length - 1 || params.length + 4 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.ctx" + this.field + " (ctx, key, res, id) VALUES " + stmt, params); + stmt = ""; + params = []; + } + } + } + } + } + } + //}); + //await this.transaction(function(){ + + if (flexsearch.store) { + let stmt = "", + chunk = []; + + for (const item of flexsearch.store.entries()) { + const id = item[0], + doc = item[1]; + + stmt += (stmt ? "," : "") + "(?,?)"; + chunk.push(id, "object" == typeof doc ? JSON.stringify(doc) : doc || null); + if (chunk.length + 2 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.reg (id, doc) VALUES " + stmt, chunk); + stmt = ""; + chunk = []; + } + } + if (chunk.length) { + this.db.run("INSERT INTO main.reg (id, doc) VALUES " + stmt, chunk); + } + } else if (!flexsearch.bypass) { + let ids = toArray(flexsearch.reg); + for (let count = 0; count < ids.length;) { + const chunk = ids.length - count > MAXIMUM_QUERY_VARS ? ids.slice(count, count + MAXIMUM_QUERY_VARS) : count ? ids.slice(count) : ids; + count += chunk.length; + const stmt = build_params(chunk.length); + this.db.run("INSERT INTO main.reg (id) VALUES (" + stmt + ")", chunk); + } + } + //}); + //await this.transaction(function(){ + + if (flexsearch.tag) { + let stmt = "", + chunk = []; + + for (const item of flexsearch.tag) { + const tag = item[0], + ids = item[1]; + + if (!ids.length) continue; + for (let i = 0; i < ids.length; i++) { + stmt += (stmt ? "," : "") + "(?,?)"; + chunk.push(tag, ids[i]); + } + if (chunk.length + 2 > MAXIMUM_QUERY_VARS) { + this.db.run("INSERT INTO main.tag" + this.field + " (tag, id) VALUES " + stmt, chunk); + stmt = ""; + chunk = []; + } + } + if (chunk.length) { + this.db.run("INSERT INTO main.tag" + this.field + " (tag, id) VALUES " + stmt, chunk); + } + } + }); + + // TODO + //await this.transaction(function(){ + // this.db.run("INSERT INTO main.cfg" + this.field + " (cfg) VALUES (?)", [JSON.stringify({ + // "charset": flexsearch.charset, + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "fastupdate": flexsearch.fastupdate, + // "compress": flexsearch.compress, + // "encoder": { + // "minlength": flexsearch.encoder.minlength + // }, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // })]); + //}); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && flexsearch.tag.clear(); + flexsearch.store && flexsearch.store.clear(); + flexsearch.document || flexsearch.reg.clear(); +}; + +SqliteDB.prototype.remove = function (ids) { + + if ("object" != typeof ids) { + ids = [ids]; + } + + let next; + // maximum count of variables supported + if (ids.length > MAXIMUM_QUERY_VARS) { + next = ids.slice(MAXIMUM_QUERY_VARS); + ids = ids.slice(0, MAXIMUM_QUERY_VARS); + } + + const self = this; + return this.transaction(function () { + const stmt = build_params(ids.length); + this.db.run("DELETE FROM main.map" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.ctx" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.tag" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.reg WHERE id IN (" + stmt + ")", ids); + }).then(function (res) { + return next ? self.remove(next) : res; + }); +}; + +SqliteDB.prototype.promisfy = function (opt) { + const db = this.db; + return new Promise(function (resolve, reject) { + db[opt.method](opt.stmt, opt.params || [], function (err, rows) { + opt.callback && opt.callback(rows); + err ? reject(err) : resolve(rows); + }); + }); +}; \ No newline at end of file diff --git a/dist/module/document.js b/dist/module/document.js index e0f7196..5c2a8df 100644 --- a/dist/module/document.js +++ b/dist/module/document.js @@ -6,68 +6,168 @@ * https://github.com/nextapps-de/flexsearch */ +import { DocumentOptions } from "./type.js"; import Index from "./index.js"; -import { DocumentInterface } from "./type.js"; -import Cache, { searchCache } from "./cache.js"; -import { create_object, is_array, is_string, is_object, parse_option, get_keys } from "./common.js"; -import apply_async from "./async.js"; -import { intersect, intersect_union } from "./intersect.js"; -import { exportDocument, importDocument } from "./serialize.js"; import WorkerIndex from "./worker/index.js"; +import Cache, { searchCache } from "./cache.js"; +import { is_string, is_object, parse_simple } from "./common.js"; +import apply_async from "./async.js"; +import { exportDocument, importDocument } from "./serialize.js"; +import { KeystoreMap, KeystoreSet } from "./keystore.js"; +import "./document/add.js"; +import "./document/search.js"; /** * @constructor - * @implements {DocumentInterface} - * @param {Object=} options - * @return {Document} + * @param {DocumentOptions=} options */ -function Document(options) { +export default function Document(options) { if (!(this instanceof Document)) { - return new Document(options); } const document = options.document || options.doc || options; - let opt; + let tmp, keystore; this.tree = []; this.field = []; this.marker = []; - this.register = create_object(); - this.key = (opt = document.key || document.id) && parse_tree(opt, this.marker) || "id"; - this.fastupdate = parse_option(options.fastupdate, /* append: */ /* skip update: */ /* skip_update: */!0); + this.key = (tmp = document.key || document.id) && parse_tree(tmp, this.marker) || "id"; - this.storetree = (opt = document.store) && !0 !== opt && []; - this.store = opt && create_object(); + keystore = options.keystore || 0; + keystore && (this.keystore = keystore); + this.fastupdate = !!options.fastupdate; + this.reg = this.fastupdate ? keystore ? new KeystoreMap(keystore) : new Map() : keystore ? new KeystoreSet(keystore) : new Set(); + // todo support custom filter function + this.storetree = (tmp = document.store || null) && /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ !== tmp && []; + this.store = tmp && (keystore ? new KeystoreMap(keystore) : new Map()); - // TODO case-insensitive tags - - this.tag = (opt = document.tag) && parse_tree(opt, this.marker); - this.tagindex = opt && create_object(); - - - this.cache = (opt = options.cache) && new Cache(opt); - - // do not apply cache again for the indexes - - options.cache = !1; - + this.cache = (tmp = options.cache || null) && new Cache(tmp); + // do not apply cache again for the indexes since .searchCache() + // is just a wrapper over .search() + options.cache = /* suggest */ /* append: */ /* enrich */!1; this.worker = options.worker; - // this switch is used by recall of promise callbacks - this.async = !1; /** @export */ this.index = parse_descriptor.call(this, options, document); + + this.tag = null; + // TODO case-insensitive tags? + if (tmp = document.tag) { + if ("string" == typeof tmp) { + tmp = [tmp]; + } + if (tmp.length) { + this.tag = new Map(); + this.tagtree = []; + this.tagfield = []; + for (let i = 0, params, field; i < tmp.length; i++) { + params = tmp[i]; + field = params.field || params; + if (!field) { + throw new Error("The tag field from the document descriptor is undefined."); + } + if (params.custom) { + this.tagtree[i] = params.custom; + } else { + this.tagtree[i] = parse_tree(field, this.marker); + if (params.filter) { + if ("string" == typeof this.tagtree[i]) { + // it needs an object to put a property to it + this.tagtree[i] = new String(this.tagtree[i]); + } + this.tagtree[i]._filter = params.filter; + } + } + // the tag fields needs to be hold by indices + this.tagfield[i] = field; + this.tag.set(field, new Map()); + } + } + } + + options.db && this.mount(options.db); } -export default Document; +Document.prototype.mount = function (db) { + + let fields = this.field; + + if (this.tag) { + // tag indexes are referenced by field + // move tags to their field indexes respectively + for (let i = 0, field; i < this.tagfield.length; i++) { + field = this.tagfield[i]; + let index = this.index.get(field); + if (!index) { + // create raw index when not exists + this.index.set(field, index = new Index({}, this.reg)); + // copy and push to the field selection + if (fields === this.field) { + fields = fields.slice(0); + } + // tag indexes also needs to be upgraded to db instances + fields.push(field); + } + // assign reference + index.tag = this.tag.get(field); + } + } + + const promises = [], + config = { + db: db.db, + type: db.type, + fastupdate: db.fastupdate + }; + + + // upgrade all indexes to db instances + for (let i = 0, index, field; i < fields.length; i++) { + config.field = field = fields[i]; + index = this.index.get(field); + const dbi = new db.constructor(db.id, config); + // take over the storage id + dbi.id = db.id; + promises[i] = dbi.mount(index); + // add an identification property + index.document = !0; + if (i) { + // the register has to export just one time + // also it's needed by the index for ID contain check + index.bypass = !0; + } else { + // the datastore has to export one time + index.store = this.store; + } + } + + this.async = !0; + this.db = !0; + return Promise.all(promises); +}; + +Document.prototype.commit = async function (replace, append) { + // parallel: + const promises = []; + for (const index of this.index.values()) { + promises.push(index.db.commit(index, replace, append)); + } + await Promise.all(promises); + this.reg.clear(); + // queued: + // for(const index of this.index.values()){ + // await index.db.commit(index, replace, append); + // } + // this.reg.clear(); +}; /** * @this Document @@ -75,11 +175,10 @@ export default Document; function parse_descriptor(options, document) { - const index = create_object(); + const index = new Map(); let field = document.index || document.field || document; if (is_string(field)) { - field = [field]; } @@ -88,7 +187,6 @@ function parse_descriptor(options, document) { key = field[i]; if (!is_string(key)) { - opt = key; key = key.field; } @@ -96,36 +194,55 @@ function parse_descriptor(options, document) { opt = is_object(opt) ? Object.assign({}, options, opt) : options; if (this.worker) { - - index[key] = new WorkerIndex(opt); - - if (!index[key].worker) { - + const worker = new WorkerIndex(opt); + index.set(key, worker); + if (!worker.worker) { + // fallback when not supported this.worker = !1; } } if (!this.worker) { - - index[key] = new Index(opt, this.register); + index.set(key, new Index(opt, this.reg)); + } + + if (opt.custom) { + this.tree[i] = opt.custom; + } else { + this.tree[i] = parse_tree(key, this.marker); + if (opt.filter) { + if ("string" == typeof this.tree[i]) { + // it needs an object to put a property to it + this.tree[i] = new String(this.tree[i]); + } + this.tree[i]._filter = opt.filter; + } } - this.tree[i] = parse_tree(key, this.marker); this.field[i] = key; } if (this.storetree) { - let store = document.store; + let stores = document.store; + if (is_string(stores)) stores = [stores]; - if (is_string(store)) { - - store = [store]; - } - - for (let i = 0; i < store.length; i++) { - - this.storetree[i] = parse_tree(store[i], this.marker); + for (let i = 0, store, field; i < stores.length; i++) { + store = stores[i]; + field = store.field || store; + if (store.custom) { + this.storetree[i] = store.custom; + store.custom._field = field; + } else { + this.storetree[i] = parse_tree(field, this.marker); + if (store.filter) { + if ("string" == typeof this.storetree[i]) { + // it needs an object to put a property to it + this.storetree[i] = new String(this.storetree[i]); + } + this.storetree[i]._filter = store.filter; + } + } } } @@ -138,291 +255,57 @@ function parse_tree(key, marker) { let count = 0; for (let i = 0; i < tree.length; i++) { - key = tree[i]; - - if (0 <= key.indexOf("[]")) { - + // todo apply some indexes e.g. [0], [-1], [0-2] + if ("]" === key[key.length - 1]) { key = key.substring(0, key.length - 2); - if (key) { - marker[count] = !0; } } - if (key) { - tree[count++] = key; } } if (count < tree.length) { - tree.length = count; } return 1 < count ? tree : tree[0]; } -// TODO support generic function created from string when tree depth > 1 - -function parse_simple(obj, tree) { - - if (is_string(tree)) { - - obj = obj[tree]; - } else { - - for (let i = 0; obj && i < tree.length; i++) { - - obj = obj[tree[i]]; - } - } - - return obj; -} - -// TODO support generic function created from string when tree depth > 1 - -function store_value(obj, store, tree, pos, key) { - - obj = obj[key]; - - // reached target field - - if (pos === tree.length - 1) { - - // store target value - - store[key] = obj; - } else if (obj) { - - if (is_array(obj)) { - - store = store[key] = Array(obj.length); - - for (let i = 0; i < obj.length; i++) { - - // do not increase pos (an array is not a field) - store_value(obj, store, tree, pos, i); - } - } else { - - store = store[key] || (store[key] = create_object()); - key = tree[++pos]; - - store_value(obj, store, tree, pos, key); - } - } -} - -function add_index(obj, tree, marker, pos, index, id, key, _append) { - - obj = obj[key]; - - if (obj) { - - // reached target field - - if (pos === tree.length - 1) { - - // handle target value - - if (is_array(obj)) { - - // append array contents so each entry gets a new scoring context - - if (marker[pos]) { - - for (let i = 0; i < obj.length; i++) { - - index.add(id, obj[i], !0, !0); - } - - return; - } - - // or join array contents and use one scoring context - - obj = obj.join(" "); - } - - index.add(id, obj, _append, !0); - } else { - - if (is_array(obj)) { - - for (let i = 0; i < obj.length; i++) { - - // do not increase index, an array is not a field - - add_index(obj, tree, marker, pos, index, id, i, _append); - } - } else { - - key = tree[++pos]; - - add_index(obj, tree, marker, pos, index, id, key, _append); - } - } - } -} - -/** - * - * @param id - * @param content - * @param {boolean=} _append - * @returns {Document|Promise} - */ - -Document.prototype.add = function (id, content, _append) { - - if (is_object(id)) { - - content = id; - id = parse_simple(content, this.key); - } - - if (content && (id || 0 === id)) { - - if (!_append && this.register[id]) { - - return this.update(id, content); - } - - for (let i = 0, tree, field; i < this.field.length; i++) { - - field = this.field[i]; - tree = this.tree[i]; - - if (is_string(tree)) { - - tree = [tree]; - } - - add_index(content, tree, this.marker, 0, this.index[field], id, tree[0], _append); - } - - if (this.tag) { - let tag = parse_simple(content, this.tag), - dupes = create_object(); - - - if (is_string(tag)) { - - tag = [tag]; - } - - for (let i = 0, key, arr; i < tag.length; i++) { - - key = tag[i]; - - if (!dupes[key]) { - - dupes[key] = 1; - arr = this.tagindex[key] || (this.tagindex[key] = []); - - if (!_append || !arr.includes(id)) { - - arr[arr.length] = id; - - // add a reference to the register for fast updates - - if (this.fastupdate) { - - const tmp = this.register[id] || (this.register[id] = []); - tmp[tmp.length] = arr; - } - } - } - } - } - - // TODO: how to handle store when appending contents? - - if (this.store && (!_append || !this.store[id])) { - - let store; - - if (this.storetree) { - - store = create_object(); - - for (let i = 0, tree; i < this.storetree.length; i++) { - - tree = this.storetree[i]; - - if (is_string(tree)) { - - store[tree] = content[tree]; - } else { - - store_value(content, store, tree, 0, tree[0]); - } - } - } - - this.store[id] = store || content; - } - } - - return this; -}; - Document.prototype.append = function (id, content) { - return this.add(id, content, !0); }; Document.prototype.update = function (id, content) { - return this.remove(id).add(id, content); }; Document.prototype.remove = function (id) { if (is_object(id)) { - id = parse_simple(id, this.key); } - if (this.register[id]) { + for (const index of this.index.values()) { + index.remove(id, !0); + } - for (let i = 0; i < this.field.length; i++) { - - // workers does not share the register - - this.index[this.field[i]].remove(id, !this.worker); - - if (this.fastupdate) { - - // when fastupdate was enabled all ids are removed - - break; - } - } + if (this.reg.has(id)) { if (this.tag) { - // when fastupdate was enabled all ids are already removed - if (!this.fastupdate) { + for (let field of this.tag.values()) { + for (let item of field) { + const tag = item[0], + ids = item[1], + pos = ids.indexOf(id); - for (let key in this.tagindex) { - const tag = this.tagindex[key], - pos = tag.indexOf(id); - - - if (-1 !== pos) { - - if (1 < tag.length) { - - tag.splice(pos, 1); - } else { - - delete this.tagindex[key]; + if (-1 < pos) { + 1 < ids.length ? ids.splice(pos, 1) : field.delete(tag); } } } @@ -430,297 +313,84 @@ Document.prototype.remove = function (id) { } if (this.store) { - - delete this.store[id]; + this.store.delete(id); } - delete this.register[id]; + this.reg.delete(id); + } + + // the cache could be used outside the InMemory store + if (this.cache) { + this.cache.remove(id); } return this; }; -/** - * @param {!string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @param {Array=} _resolve For internal use only. - * @returns {Promise|Array} - */ +Document.prototype.clear = function () { -Document.prototype.search = function (query, limit, options, _resolve) { + //const promises = []; - if (!options) { + for (const index of this.index.values()) { + // db index will add clear task + index.clear(); + // const promise = index.clear(); + // if(promise instanceof Promise){ + // promises.push(promise); + // } + } - if (!limit && is_object(query)) { - - options = /** @type {Object} */query; - query = ""; - } else if (is_object(limit)) { - - options = /** @type {Object} */limit; - limit = 0; + if (this.tag) { + for (const tags of this.tag.values()) { + tags.clear(); } } - let result = [], - result_field = [], - pluck, - enrich, - field, - tag, - bool, - offset, - count = 0; - - - if (options) { - - if (is_array(options)) { - - field = options; - options = null; - } else { - - query = options.query || query; - pluck = options.pluck; - field = pluck || options.index || options.field /*|| (is_string(options) && [options])*/; - tag = options.tag; - enrich = this.store && options.enrich; - bool = "and" === options.bool; - limit = options.limit || limit || 100; - offset = options.offset || 0; - - if (tag) { - - if (is_string(tag)) { - - tag = [tag]; - } - - // when tags is used and no query was set, - // then just return the tag indexes - - if (!query) { - - for (let i = 0, res; i < tag.length; i++) { - - res = get_tag.call(this, tag[i], limit, offset, enrich); - - if (res) { - - result[result.length] = res; - count++; - } - } - - return count ? result : []; - } - } - - if (is_string(field)) { - - field = [field]; - } - } + if (this.store) { + this.store.clear(); } - field || (field = this.field); - bool = bool && (1 < field.length || tag && 1 < tag.length); - - const promises = !_resolve && (this.worker || this.async) && []; - - // TODO solve this in one loop below - - for (let i = 0, res, key, len; i < field.length; i++) { - - let field_options; - - key = field[i]; - - if (!is_string(key)) { - - field_options = key; - key = field_options.field; - query = field_options.query || query; - limit = field_options.limit || limit; - enrich = field_options.enrich || enrich; - } - - if (promises) { - - promises[i] = this.index[key].searchAsync(query, limit, field_options || options); - - // just collect and continue - - continue; - } else if (_resolve) { - - res = _resolve[i]; - } else { - - // inherit options also when search? it is just for laziness, Object.assign() has a cost - - res = this.index[key].search(query, limit, field_options || options); - } - - len = res && res.length; - - if (tag && len) { - - const arr = []; - let count = 0; - - if (bool) { - - // prepare for intersection - - arr[0] = [res]; - } - - for (let y = 0, key, res; y < tag.length; y++) { - - key = tag[y]; - res = this.tagindex[key]; - len = res && res.length; - - if (len) { - - count++; - arr[arr.length] = bool ? [res] : res; - } - } - - if (count) { - - if (bool) { - - res = intersect(arr, limit || 100, offset || 0); - } else { - - res = intersect_union(res, arr); - } - - len = res.length; - } - } - - if (len) { - - result_field[count] = key; - result[count++] = res; - } else if (bool) { - - return []; - } - } - - if (promises) { - - const self = this; - - // anyone knows a better workaround of optionally having async promises? - // the promise.all() needs to be wrapped into additional promise, - // otherwise the recursive callback wouldn't run before return - - return new Promise(function (resolve) { - - Promise.all(promises).then(function (result) { - - resolve(self.search(query, limit, options, result)); - }); - }); - } - - if (!count) { - - // fast path "not found" - - return []; - } - - if (pluck && (!enrich || !this.store)) { - - // fast path optimization - - return result[0]; - } - - for (let i = 0, res; i < result_field.length; i++) { - - res = result[i]; - - if (res.length) { - - if (enrich) { - - res = apply_enrich.call(this, res); - } - } - - if (pluck) { - - return res; - } - - result[i] = { - - field: result_field[i], - result: res - }; - } - - return result; + return this; /*promises.length + ? Promise.all(promises) + :*/ }; -/** - * @this Document - */ - -function get_tag(key, limit, offset) { - let res = this.tagindex[key], - len = res && res.length - offset; -} - -/** - * @this Document - */ - -function apply_enrich(res) { - - const arr = Array(res.length); - - for (let x = 0, id; x < res.length; x++) { - - id = res[x]; - - arr[x] = { - - id: id, - doc: this.store[id] - }; - } - - return arr; -} - Document.prototype.contain = function (id) { - return !!this.register[id]; + if (this.db) { + return this.index.get(this.field[0]).db.has(id); + } + + return this.reg.has(id); +}; + +Document.prototype.cleanup = function () { + + for (const index of this.index.values()) { + index.cleanup(); + } + + return this; }; Document.prototype.get = function (id) { - return this.store[id]; + if (this.db) { + return this.index.get(this.field[0]).db.enrich(id).then(function (result) { + return result[0] && result[0].doc; + }); + } + + return this.store.get(id); }; -Document.prototype.set = function (id, data) { +Document.prototype.set = function (id, store) { - this.store[id] = data; + this.store.set(id, store); return this; }; - +// todo mo Document.prototype.searchCache = searchCache; diff --git a/dist/module/document/add.js b/dist/module/document/add.js new file mode 100644 index 0000000..d59ac77 --- /dev/null +++ b/dist/module/document/add.js @@ -0,0 +1,243 @@ + +import { create_object, is_array, is_object, is_string, parse_simple } from "../common.js"; +import { KeystoreArray } from "../keystore.js"; +import Document from "../document.js"; + +/** + * + * @param id + * @param content + * @param {boolean=} _append + * @returns {Document|Promise} + */ + +Document.prototype.add = function (id, content, _append) { + + if (is_object(id)) { + + content = id; + id = parse_simple(content, this.key); + } + + if (content && (id || 0 === id)) { + + if (!_append && this.reg.has(id)) { + return this.update(id, content); + } + + // this.field does not include db tag indexes + for (let i = 0, tree; i < this.field.length; i++) { + + tree = this.tree[i]; + + const index = this.index.get(this.field[i]); + if ("function" == typeof tree) { + const tmp = tree(content); + if (tmp) { + index.add(id, tmp, /* suggest */ /* append: */!1, /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/); + } + } else { + const filter = tree._filter; + if (filter && !filter(content)) { + continue; + } + if (tree instanceof String) { + tree = ["" + tree]; + } else if (is_string(tree)) { + tree = [tree]; + } + add_index(content, tree, this.marker, 0, index, id, tree[0], _append); + } + } + + if (this.tag) { + + //console.log(this.tag, this.tagtree) + + for (let x = 0; x < this.tagtree.length; x++) { + let tree = this.tagtree[x], + field = this.tagfield[x], + ref = this.tag.get(field), + dupes = create_object(), + tags; + + + if ("function" == typeof tree) { + tags = tree(content); + if (!tags) continue; + } else { + const filter = tree._filter; + if (filter && !filter(content)) { + continue; + } + if (tree instanceof String) { + tree = "" + tree; + } + tags = parse_simple(content, tree); + } + + if (!ref || !tags) { + continue; + } + + if (is_string(tags)) { + tags = [tags]; + } + + for (let i = 0, tag, arr; i < tags.length; i++) { + + tag = tags[i]; + //console.log(this.tag, tag, key, field) + + if (!dupes[tag]) { + dupes[tag] = 1; + + let tmp = ref.get(tag); + + tmp ? arr = tmp : ref.set(tag, arr = []); + + if (!_append || ! /** @type {!Array|KeystoreArray} */arr.includes(id)) { + + // auto-upgrade to keystore array if max size exceeded + if (2147483647 === arr.length /*|| !(arr instanceof KeystoreArray)*/) { + const keystore = new KeystoreArray(arr); + if (this.fastupdate) { + for (let value of this.reg.values()) { + if (value.includes(arr)) { + value[value.indexOf(arr)] = keystore; + } + } + } + ref.set(tag, arr = keystore); + } + + + arr.push(id); + + // add a reference to the register for fast updates + if (this.fastupdate) { + const tmp = this.reg.get(id); + tmp ? tmp.push(arr) : this.reg.set(id, [arr]); + } + } + } + } + } + } + + if (this.store && (!_append || !this.store.has(id))) { + + let payload; + + if (this.storetree) { + + payload = create_object(); + + for (let i = 0, tree; i < this.storetree.length; i++) { + tree = this.storetree[i]; + + const filter = tree._filter; + if (filter && !filter(content)) { + continue; + } + let custom; + if ("function" == typeof tree) { + custom = tree(content); + if (!custom) continue; + tree = [tree._field]; + } else if (is_string(tree) || tree instanceof String) { + payload[tree] = content[tree]; + continue; + } + + store_value(content, payload, tree, 0, tree[0], custom); + } + } + + this.store.set(id, payload || content); + } + } + + return this; +}; + +// TODO support generic function created from string when tree depth > 1 + +/** + * @param obj + * @param store + * @param tree + * @param pos + * @param key + * @param {*=} custom + */ + +function store_value(obj, store, tree, pos, key, custom) { + + obj = obj[key]; + + // reached target field + if (pos === tree.length - 1) { + + // store target value + store[key] = custom || obj; + } else if (obj) { + + if (is_array(obj)) { + + store = store[key] = Array(obj.length); + + for (let i = 0; i < obj.length; i++) { + // do not increase pos (an array is not a field) + store_value(obj, store, tree, pos, i); + } + } else { + + store = store[key] || (store[key] = create_object()); + key = tree[++pos]; + store_value(obj, store, tree, pos, key); + } + } +} + +function add_index(obj, tree, marker, pos, index, id, key, _append) { + + if (obj = obj[key]) { + + // reached target field + if (pos === tree.length - 1) { + + // handle target value + if (is_array(obj)) { + + // append array contents so each entry gets a new scoring context + if (marker[pos]) { + for (let i = 0; i < obj.length; i++) { + index.add(id, obj[i], !0, !0); + } + return; + } + + // or join array contents and use one scoring context + obj = obj.join(" "); + } + + index.add(id, obj, _append, !0); + } else { + + if (is_array(obj)) { + for (let i = 0; i < obj.length; i++) { + // do not increase index, an array is not a field + add_index(obj, tree, marker, pos, index, id, i, _append); + } + } else { + key = tree[++pos]; + add_index(obj, tree, marker, pos, index, id, key, _append); + } + } + } else { + if (index.db) { + index.remove(id); + } + } +} \ No newline at end of file diff --git a/dist/module/document/search.js b/dist/module/document/search.js new file mode 100644 index 0000000..24837e0 --- /dev/null +++ b/dist/module/document/search.js @@ -0,0 +1,429 @@ + + +import { DocumentSearchOptions } from "../type.js"; +import { create_object, is_array, is_object, is_string } from "../common.js"; +import { intersect_union } from "../intersect.js"; +import Document from "../document.js"; + +let debug = /* suggest */ /* append: */ /* enrich */!1; + +/** + * @param {!string|DocumentSearchOptions} query + * @param {number|DocumentSearchOptions=} limit + * @param {DocumentSearchOptions=} options + * @param {Array=} _resolve For internal use only. + * @returns {Promise|Array} + */ + +Document.prototype.search = function (query, limit, options, _resolve) { + + if (!options) { + if (!limit && is_object(query)) { + options = /** @type {DocumentSearchOptions} */query; + query = ""; + } else if (is_object(limit)) { + options = /** @type {DocumentSearchOptions} */limit; + limit = 0; + } + } + + let result = [], + result_field = [], + pluck, + enrich, + merge, + suggest, + field, + tag, + offset, + count = 0; + + + if (options) { + + // todo remove support? + if (is_array(options)) { + field = options; + options = null; + } else { + + query = options.query || query; + pluck = options.pluck; + merge = options.merge; + field = pluck || options.field || options.index; + tag = this.tag && options.tag; + enrich = this.store && options.enrich; + suggest = options.suggest; + limit = options.limit || limit; + offset = options.offset || 0; + limit || (limit = 100); + + if (tag && (!this.db || !_resolve)) { + + if (tag.constructor !== Array) { + tag = [tag]; + } + + // Tag-Search + // ----------------------------- + + let pairs = []; + + for (let i = 0, field; i < tag.length; i++) { + field = tag[i]; + + // default array notation + if (field.field && field.tag) { + const value = field.tag; + if (value.constructor === Array) { + for (let k = 0; k < value.length; k++) { + pairs.push(field.field, value[k]); + } + } else { + pairs.push(field.field, value); + } + } + // shorter object notation + else { + const keys = Object.keys(field); + for (let j = 0, key, value; j < keys.length; j++) { + key = keys[j]; + value = field[key]; + if (value.constructor === Array) { + for (let k = 0; k < value.length; k++) { + pairs.push(key, value[k]); + } + } else { + pairs.push(key, value); + } + } + } + } + + // tag used as pairs from this point + tag = pairs; + + // when tags is used and no query was set, + // then just return the tag indexes + if (!query) { + + let promises = []; + if (pairs.length) for (let j = 0; j < pairs.length; j += 2) { + let ids; + if (this.db) { + const index = this.index.get(pairs[j]); + if (!index) { + continue; + } + + promises.push(ids = index.db.tag(pairs[j + 1], limit, offset, enrich)); + } else { + ids = get_tag.call(this, pairs[j], pairs[j + 1], limit, offset, enrich); + } + result.push({ + field: pairs[j], + tag: pairs[j + 1], + result: ids + }); + } + + if (promises.length) { + return Promise.all(promises).then(function (promises) { + for (let j = 0; j < promises.length; j++) { + result[j].result = promises[j]; + } + return result; + }); + } + + return result; + } + } + + // extend to multi field search by default + if (is_string(field)) { + field = [field]; + } + } + } + + field || (field = this.field); + let promises = !_resolve && (this.worker || this.async) && [], + db_tag_search; + + + // multi field search + // field could be a custom set of selected fields by this query + // db tag indexes are also included in this field list + for (let i = 0, res, key, len; i < field.length; i++) { + + key = field[i]; + + if (this.db && this.tag) { + // tree is missing when it is a tag-only index (db) + if (!this.tree[i]) { + continue; + } + } + + let field_options; + + if (!is_string(key)) { + field_options = key; + key = field_options.field; + query = field_options.query || query; + limit = field_options.limit || limit; + //offset = field_options.offset || offset; + suggest = field_options.suggest || suggest; + //enrich = SUPPORT_STORE && this.store && (field_options.enrich || enrich); + } + + if (_resolve) { + res = _resolve[i]; + } else { + let opt = field_options || options, + index = this.index.get(key); + + + if (tag) { + if (this.db) { + opt.tag = tag; + db_tag_search = index.db.support_tag_search; + opt.field = field; + } + if (!db_tag_search) { + opt.enrich = !1; + } + } + if (promises) { + promises[i] = index.searchAsync(query, limit, opt); + // restore enrich state + opt && enrich && (opt.enrich = enrich); + // just collect and continue + continue; + } else { + res = index.search(query, limit, opt); + // restore enrich state + opt && enrich && (opt.enrich = enrich); + } + } + + len = res && res.length; + + // todo when no term was matched but tag was retrieved extend suggestion to tags + // every field has to intersect against all selected tag fields + if (tag && len) { + + const arr = []; + let count = 0; + + // tags are only applied in resolve phase when it's a db + if (this.db && _resolve) { + if (!db_tag_search) { + + // retrieve tag results assigned to it's field + for (let y = field.length; y < _resolve.length; y++) { + let ids = _resolve[y], + len = ids && ids.length; + + + if (len) { + count++; + arr.push(ids); + } else if (!suggest) { + // no tags found + return result; + } + } + } + } else { + + // tag[] are pairs at this line + for (let y = 0, ids, len; y < tag.length; y += 2) { + ids = this.tag.get(tag[y]); + + if (!ids) { + if (suggest) { + continue; + } else { + return result; + } + } + + ids = ids && ids.get(tag[y + 1]); + len = ids && ids.length; + + if (len) { + count++; + arr.push(ids); + } else if (!suggest) { + // no tags found + return result; + } + } + } + + if (count) { + res = intersect_union(res, arr); // intersect(arr, limit, offset) + len = res.length; + if (!len && !suggest) { + // nothing matched + return result; + } + // move counter back by 1 + count--; + } + } + + if (len) { + result_field[count] = key; + result.push(res); + count++; + } else if (1 === field.length) { + // fast path: nothing matched + return result; + } + } + + if (promises) { + if (this.db) { + // todo when a tag index is never a search index this could be extracted + // push tag promises to the end + if (tag && tag.length && !db_tag_search) { + for (let y = 0; y < tag.length; y += 2) { + // it needs to retrieve data from tag pairs + const index = this.index.get(tag[y]); + if (!index) { + if (suggest) { + continue; + } else { + return result; + } + } + + promises.push(index.db.tag(tag[y + 1], limit, offset, !1)); + } + } + } + + const self = this; + + // TODO unroll this recursion + return Promise.all(promises).then(function (result) { + return result.length ? self.search(query, limit, options, /* resolve: */result) : result; + }); + } + + if (!count) { + return result; + } + if (pluck && (!enrich || !this.store)) { + return result[0]; + } + + promises = []; + + for (let i = 0, res; i < result_field.length; i++) { + + res = result[i]; + + if (enrich && res.length && !res[0].doc) { + if (!this.db) { + if (res.length) { + res = apply_enrich.call(this, res); + } + } else { + promises.push(res = this.index.get(this.field[0]).db.enrich(res)); + } + } + + if (pluck) { + return res; + } + + result[i] = { + field: result_field[i], + result: res + }; + } + + if (enrich && /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ && this.db && promises.length) { + return Promise.all(promises).then(function (promises) { + for (let j = 0; j < promises.length; j++) { + result[j].result = promises[j]; + } + return merge ? merge_fields(result, limit, offset) : result; + }); + } + + return merge ? merge_fields(result, limit, offset) : result; +}; + +// todo support Resolver +// todo when searching through multiple fields each term should +// be found at least by one field to get a valid match without +// using suggestion explicitly + +function merge_fields(fields, limit) { + const final = [], + set = create_object(); + + for (let i = 0, field, res; i < fields.length; i++) { + field = fields[i]; + res = field.result; + for (let j = 0, id, entry, tmp; j < res.length; j++) { + entry = res[j]; + id = entry.id; + tmp = set[id]; + if (!tmp) { + // offset was already applied on field indexes + // if(offset){ + // offset--; + // continue; + // } + // apply limit from last round, because just fields could + // be pushed without adding new results + if (final.length === limit) { + return final; + } + entry.field = set[id] = [field.field]; + final.push(entry); + } else { + tmp.push(field.field); + } + } + } + return final; +} + +/** + * @this Document + */ + +function get_tag(tag, key, limit, offset) { + let res = this.tag.get(tag); + if (!res) { + return []; + } + res = res && res.get(key); + res && res.length - offset; +} + +/** + * @this Document + */ + +function apply_enrich(res) { + + const arr = Array(res.length); + + for (let x = 0, id; x < res.length; x++) { + id = res[x]; + arr[x] = { + id: id, + doc: this.store.get(id) + }; + } + + return arr; +} \ No newline at end of file diff --git a/dist/module/encoder.js b/dist/module/encoder.js new file mode 100644 index 0000000..0b18c42 --- /dev/null +++ b/dist/module/encoder.js @@ -0,0 +1,568 @@ + +import { parse_option } from "./common.js"; + +/* + +Custom Encoder +---------------- + +// Split a passed string into an Array of words: +function englishEncoder(string){ + return string.toLowerCase().split(/[^a-z]+/) +} + +// For CJK split a passed string into an Array of chars: +function chineseEncoder(string){ + return string.replace(/\s+/, "").split("") +} + +// Alternatively do not split the input: +function fixedEncoder(string){ + return [string] +} + +Built-in Encoder (Workflow) +---------------------------- +Pipeline: + 1. apply this.normalize: charset normalization: + applied on the whole input string e.g. lowercase, + will also apply on: filter, matcher, stemmer, mapper + 2. apply this.split: split input into terms (includes/excludes) + 3. apply this.filter (pre-filter) + 4. apply this.matcher (replace terms) + 5. apply this.stemmer (replace term endings) + 6. apply this.filter (post-filter) + 7. apply this.mapper (replace chars) + 8. apply this.replacer (custom regex) + 9. apply this.dedupe (letter deduplication) + 10. apply this.finalize +*/ + +const whitespace = /[^\p{L}\p{N}]+/u, + numeric_split_length = /(\d{3})/g, + numeric_split_prev_char = /(\D)(\d{3})/g, + numeric_split_next_char = /(\d{3})(\D)/g, + normalize = /[\u0300-\u036f]/g, + normalize_mapper = !normalize && new Map([ + +// Charset Normalization + +["ª", "a"], ["²", "2"], ["³", "3"], ["¹", "1"], ["º", "o"], ["¼", "1⁄4"], ["½", "1⁄2"], ["¾", "3⁄4"], ["à", "a"], ["á", "a"], ["â", "a"], ["ã", "a"], ["ä", "a"], ["å", "a"], ["ç", "c"], ["è", "e"], ["é", "e"], ["ê", "e"], ["ë", "e"], ["ì", "i"], ["í", "i"], ["î", "i"], ["ï", "i"], ["ñ", "n"], ["ò", "o"], ["ó", "o"], ["ô", "o"], ["õ", "o"], ["ö", "o"], ["ù", "u"], ["ú", "u"], ["û", "u"], ["ü", "u"], ["ý", "y"], ["ÿ", "y"], ["ā", "a"], ["ă", "a"], ["ą", "a"], ["ć", "c"], ["ĉ", "c"], ["ċ", "c"], ["č", "c"], ["ď", "d"], ["ē", "e"], ["ĕ", "e"], ["ė", "e"], ["ę", "e"], ["ě", "e"], ["ĝ", "g"], ["ğ", "g"], ["ġ", "g"], ["ģ", "g"], ["ĥ", "h"], ["ĩ", "i"], ["ī", "i"], ["ĭ", "i"], ["į", "i"], ["ij", "ij"], ["ĵ", "j"], ["ķ", "k"], ["ĺ", "l"], ["ļ", "l"], ["ľ", "l"], ["ŀ", "l"], ["ń", "n"], ["ņ", "n"], ["ň", "n"], ["ʼn", "n"], ["ō", "o"], ["ŏ", "o"], ["ő", "o"], ["ŕ", "r"], ["ŗ", "r"], ["ř", "r"], ["ś", "s"], ["ŝ", "s"], ["ş", "s"], ["š", "s"], ["ţ", "t"], ["ť", "t"], ["ũ", "u"], ["ū", "u"], ["ŭ", "u"], ["ů", "u"], ["ű", "u"], ["ų", "u"], ["ŵ", "w"], ["ŷ", "y"], ["ź", "z"], ["ż", "z"], ["ž", "z"], ["ſ", "s"], ["ơ", "o"], ["ư", "u"], ["dž", "dz"], ["lj", "lj"], ["nj", "nj"], ["ǎ", "a"], ["ǐ", "i"], ["ǒ", "o"], ["ǔ", "u"], ["ǖ", "u"], ["ǘ", "u"], ["ǚ", "u"], ["ǜ", "u"], ["ǟ", "a"], ["ǡ", "a"], ["ǣ", "ae"], ["æ", "ae"], ["ǽ", "ae"], ["ǧ", "g"], ["ǩ", "k"], ["ǫ", "o"], ["ǭ", "o"], ["ǯ", "ʒ"], ["ǰ", "j"], ["dz", "dz"], ["ǵ", "g"], ["ǹ", "n"], ["ǻ", "a"], ["ǿ", "ø"], ["ȁ", "a"], ["ȃ", "a"], ["ȅ", "e"], ["ȇ", "e"], ["ȉ", "i"], ["ȋ", "i"], ["ȍ", "o"], ["ȏ", "o"], ["ȑ", "r"], ["ȓ", "r"], ["ȕ", "u"], ["ȗ", "u"], ["ș", "s"], ["ț", "t"], ["ȟ", "h"], ["ȧ", "a"], ["ȩ", "e"], ["ȫ", "o"], ["ȭ", "o"], ["ȯ", "o"], ["ȱ", "o"], ["ȳ", "y"], ["ʰ", "h"], ["ʱ", "h"], ["ɦ", "h"], ["ʲ", "j"], ["ʳ", "r"], ["ʴ", "ɹ"], ["ʵ", "ɻ"], ["ʶ", "ʁ"], ["ʷ", "w"], ["ʸ", "y"], ["ˠ", "ɣ"], ["ˡ", "l"], ["ˢ", "s"], ["ˣ", "x"], ["ˤ", "ʕ"], ["ΐ", "ι"], ["ά", "α"], ["έ", "ε"], ["ή", "η"], ["ί", "ι"], ["ΰ", "υ"], ["ϊ", "ι"], ["ϋ", "υ"], ["ό", "ο"], ["ύ", "υ"], ["ώ", "ω"], ["ϐ", "β"], ["ϑ", "θ"], ["ϒ", "Υ"], ["ϓ", "Υ"], ["ϔ", "Υ"], ["ϕ", "φ"], ["ϖ", "π"], ["ϰ", "κ"], ["ϱ", "ρ"], ["ϲ", "ς"], ["ϵ", "ε"], ["й", "и"], ["ѐ", "е"], ["ё", "е"], ["ѓ", "г"], ["ї", "і"], ["ќ", "к"], ["ѝ", "и"], ["ў", "у"], ["ѷ", "ѵ"], ["ӂ", "ж"], ["ӑ", "а"], ["ӓ", "а"], ["ӗ", "е"], ["ӛ", "ә"], ["ӝ", "ж"], ["ӟ", "з"], ["ӣ", "и"], ["ӥ", "и"], ["ӧ", "о"], ["ӫ", "ө"], ["ӭ", "э"], ["ӯ", "у"], ["ӱ", "у"], ["ӳ", "у"], ["ӵ", "ч"] + +// Term Separators + +// ["'", ""], // it's -> its +// ["´", ""], +// ["`", ""], +// ["’", ""], +// ["ʼ", ""], + +// Numeric-Separators Chars Removal + +// [",", ""], +// [".", ""] + +// Non-Whitespace Separators + +// already was split by default via p{P} +// ["-", " "], +// [":", " "], +// ["_", " "], +// ["|", " "], +// ["/", " "], +// ["\\", " "] +]); // /[\p{Z}\p{S}\p{P}\p{C}]+/u; +//const numeric_split = /(\d{3})/g; + +//.replace(/(\d{3})/g, "$1 ") +//.replace(/([^\d])([\d])/g, "$1 $2") +//.replace(/([\d])([^\d])/g, "$1 $2") +// '´`’ʼ., + +/** + * @param options + * @constructor + */ + +export default function Encoder(options = {}) { + + if (!(this instanceof Encoder)) { + return new Encoder(...arguments); + } + + for (let i = 0; i < arguments.length; i++) { + this.assign(arguments[i]); + } +} + +Encoder.prototype.assign = function (options) { + + // if(options.assign){ + // //options = Object.assign({}, options.assign, options); + // this.assign(options.assign); + // } + + // let tmp = options["normalize"]; + // if(typeof tmp === "function"){ + // const old = this.normalize; + // if(typeof old === "function"){ + // const current = tmp; + // tmp = function(){ + // old(); + // current(); + // } + // } + // } + + /** + * pre-processing string input + * @type {Function|boolean} + */ + this.normalize = /** @type {Function|boolean} */parse_option(options.normalize, /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, this.normalize); + + // { + // letter: true, + // number: true, + // whitespace: true, + // symbol: true, + // punctuation: true, + // control: true, + // char: "" + // } + + let include = options.include, + tmp = include || options.exclude || options.split; + + + if ("object" == typeof tmp) { + let numeric = !include, + regex = ""; + + // split on whitespace by default + options.include || (regex += "\\p{Z}"); + if (tmp.letter) { + regex += "\\p{L}"; + } + if (tmp.number) { + regex += "\\p{N}"; + numeric = !!include; + } + if (tmp.symbol) { + regex += "\\p{S}"; + } + if (tmp.punctuation) { + regex += "\\p{P}"; + } + if (tmp.control) { + regex += "\\p{C}"; + } + if (tmp = tmp.char) { + regex += "object" == typeof tmp ? tmp.join("") : tmp; + } + + this.split = new RegExp("[" + (include ? "^" : "") + regex + "]+", "u"); + this.numeric = numeric; + } else { + + /** + * split string input into terms + * @type {string|RegExp|boolean|null} + */ + this.split = /** @type {string|RegExp|boolean} */parse_option(tmp, whitespace, this.split); + this.numeric = parse_option(this.numeric, !0); + } + + /** + * post-processing terms + * @type {Function|null} + */ + this.prepare = /** @type {Function|null} */parse_option(options.prepare, null, this.prepare); + /** + * final processing + * @type {Function|null} + */ + this.finalize = /** @type {Function|null} */parse_option(options.finalize, null, this.finalize); + + // options + + this.rtl = options.rtl || /* suggest */ /* append: */ /* enrich */!1; + this.dedupe = parse_option(options.dedupe, !0, this.dedupe); + this.filter = parse_option((tmp = options.filter) && new Set(tmp), null, this.filter); + this.matcher = parse_option((tmp = options.matcher) && new Map(tmp), null, this.matcher); + this.mapper = parse_option((tmp = options.mapper) && new Map(tmp), null, this.mapper); + this.stemmer = parse_option((tmp = options.stemmer) && new Map(tmp), null, this.stemmer); + this.replacer = parse_option(options.replacer, null, this.replacer); + this.minlength = parse_option(options.minlength, 1, this.minlength); + this.maxlength = parse_option(options.maxlength, 0, this.maxlength); + + // minimum required tokenizer by this encoder + //this.tokenize = options["tokenize"] || ""; + + // auto-balanced cache + this.cache = tmp = parse_option(options.cache, !0, this.cache); + if (tmp) { + this.timer = null; + this.cache_size = "number" == typeof tmp ? tmp : 2e5; + this.cache_enc = new Map(); + this.cache_prt = new Map(); + this.cache_enc_length = 128; + this.cache_prt_length = 128; + } + + // regex temporary state + this.matcher_str = ""; + this.matcher_test = null; + this.stemmer_str = ""; + this.stemmer_test = null; + + // prebuilt + // if(this.filter && this.split){ + // for(const key of this.filter){ + // const tmp = key.replace(this.split, ""); + // if(key !== tmp){ + // this.filter.delete(key); + // this.filter.add(tmp); + // } + // } + // } + if (this.matcher) { + for (const key of this.matcher.keys()) { + this.matcher_str += (this.matcher_str ? "|" : "") + key; + } + } + if (this.stemmer) { + for (const key of this.stemmer.keys()) { + this.stemmer_str += (this.stemmer_str ? "|" : "") + key; + } + } + + // if(SUPPORT_COMPRESSION){ + // this.compression = parse_option(options.compress || options.compression, 0, this.compression); + // if(this.compression && !table){ + // table = new Array(radix); + // for(let i = 0; i < radix; i++) table[i] = i + 33; + // table = String.fromCharCode.apply(null, table); + // } + // } + + return this; +}; + +Encoder.prototype.addMatcher = function (match, replace) { + // regex: + if ("object" == typeof match) { + return this.addReplacer(match, replace); + } + // a single char: + if (2 > match.length) { + return this.addMapper(match, replace); + } + this.matcher || (this.matcher = new Map()); + this.matcher.set(match, replace); + this.matcher_str += (this.matcher_str ? "|" : "") + match; + this.matcher_test = null; //new RegExp("(" + this.matcher_str + ")"); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addStemmer = function (match, replace) { + this.stemmer || (this.stemmer = new Map()); + this.stemmer.set(match, replace); + this.stemmer_str += (this.stemmer_str ? "|" : "") + match; + this.stemmer_test = null; //new RegExp("(" + this.stemmer_str + ")"); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addFilter = function (str) { + this.filter || (this.filter = new Set()); + this.filter.add(str); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addMapper = function (char_match, char_replace) { + // regex: + if ("object" == typeof char_match) { + return this.addReplacer(char_match, char_replace); + } + // not a char: + if (1 < char_match.length) { + return this.addMatcher(char_match, char_replace); + } + this.mapper || (this.mapper = new Map()); + this.mapper.set(char_match, char_replace); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addReplacer = function (match, replace) { + if ("string" == typeof match) match = new RegExp(match, "g"); + this.replacer || (this.replacer = []); + this.replacer.push(match, replace || ""); + this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.invalidate = function () { + this.cache_enc.clear(); + this.cache_prt.clear(); +}; + + +Encoder.prototype.encode = function (str) { + + //if(!str) return str; + // todo remove dupe terms + + if (this.cache && str.length <= this.cache_enc_length) { + if (this.timer) { + if (this.cache_enc.has(str)) { + return this.cache_enc.get(str); + } + } else { + this.timer = setTimeout(clear, 0, this); + } + } + + // 1. apply charset normalization + if (this.normalize) { + if ("function" == typeof this.normalize) { + str = this.normalize(str); + } else if (normalize) { + str = str.normalize("NFKD").replace(normalize, "").toLowerCase(); + } else { + str = str.toLowerCase(); + + this.mapper = this.mapper + // todo replace spread + ? new Map([... /** @type {!Iterable} */normalize_mapper, ...this.mapper]) : new Map( /** @type {Map} */normalize_mapper); + } + //if(!str) return str; + } + + // 2. apply custom encoder (can replace split) + if (this.prepare) { + str = this.prepare(str); + } + + // 3. split numbers into triplets + if (this.numeric && 3 < str.length) { + str = str.replace(numeric_split_prev_char, "$1 $2").replace(numeric_split_next_char, "$1 $2").replace(numeric_split_length, "$1 "); + } + + // if(this.matcher && (str.length > 1)){ + // this.matcher_test || ( + // this.matcher_test = new RegExp("(" + this.matcher_str + ")", "g") + // ); + // str = str.replace(this.matcher_test, match => this.matcher.get(match)); + // } + // if(this.stemmer){ + // this.stemmer_test || ( + // this.stemmer_test = new RegExp("(?!\\b)(" + this.stemmer_str + ")(\\b|_)", "g") + // ); + // str = str.replace(this.stemmer_test, match => this.stemmer.get(match)); + // } + + const skip = !(this.dedupe || this.mapper || this.filter || this.matcher || this.stemmer || this.replacer); + let final = [], + words = this.split || "" === this.split ? str.split( /** @type {string|RegExp} */this.split) : str; + //[str]; + + for (let i = 0, word, base; i < words.length; i++) { + // filter empty entries + if (!(word = base = words[i])) { + continue; + } + if (word.length < this.minlength) { + continue; + } + if (skip) { + final.push(word); + continue; + } + + // 1. pre-filter before cache + if (this.filter && this.filter.has(word)) { + continue; + } + + if (this.cache && word.length <= this.cache_prt_length) { + if (this.timer) { + const tmp = this.cache_prt.get(word); + //if(this.cache_prt.has(word)){ + if (tmp || "" === tmp) { + //word = this.cache_prt.get(word); + tmp && final.push(tmp); + //word ? words[i] = word : words.splice(i--, 1); + continue; + } + } else { + this.timer = setTimeout(clear, 0, this); + } + } + + let postfilter; + + // if(this.normalize === true && normalize){ + // word = word.normalize("NFKD").replace(normalize, ""); + // postfilter = 1; + // } + + // if(this.normalize){ + // if(typeof this.normalize === "function"){ + // word = this.normalize(word); + // } + // else if(normalize){ + // word = word.normalize("NFKD").replace(normalize, "").toLowerCase(); + // } + // else{ + // word = word.toLowerCase(); + // this.mapper = this.mapper + // ? new Map([...normalize_mapper, ...this.mapper]) + // : new Map(/** @type {Map} */ normalize_mapper); + // } + // postfilter = 1; + // //if(!str) return str; + // } + + // 2. apply stemmer after matcher + if (this.stemmer && 2 < word.length) { + // for(const item of this.stemmer){ + // const key = item[0]; + // const value = item[1]; + // + // if(word.length > key.length && word.endsWith(key)){ + // word = word.substring(0, word.length - key.length) + value; + // break; + // } + // + // // const position = word.length - key.length; + // // if(position > 0 && word.substring(position) === key){ + // // word = word.substring(0, position) + value; + // // break; + // // } + // } + this.stemmer_test || (this.stemmer_test = new RegExp("(?!^)(" + this.stemmer_str + ")$")); + word = word.replace(this.stemmer_test, match => this.stemmer.get(match)); + postfilter = 1; + } + + // 3. apply matcher + if (this.matcher && 1 < word.length) { + this.matcher_test || (this.matcher_test = new RegExp("(" + this.matcher_str + ")", "g")); + word = word.replace(this.matcher_test, match => this.matcher.get(match)); + postfilter = 1; + } + + // 4. post-filter after matcher and stemmer was applied + if (word && postfilter && (word.length < this.minlength || this.filter && this.filter.has(word))) { + word = ""; + } + + // 5. apply mapper and collapsing + if (word && (this.mapper || this.dedupe && 1 < word.length)) { + //word = this.replace_dedupe(word); + //word = replace_deduped(word, this.mapper, true); + let final = ""; + for (let i = 0, prev = "", char, tmp; i < word.length; i++) { + char = word.charAt(i); + if (char !== prev || !this.dedupe) { + tmp = this.mapper && this.mapper.get(char); + if (!tmp && "" !== tmp) final += prev = char;else if ((tmp !== prev || !this.dedupe) && (prev = tmp)) final += tmp; + } + } + word = final; + } + + // apply custom regex + if (word && this.replacer) { + for (let i = 0; word && i < this.replacer.length; i += 2) { + word = word.replace(this.replacer[i], this.replacer[i + 1]); + } + } + + // slower variants for removing same chars in a row: + //word = word.replace(/([^0-9])\1+/g, "$1"); + //word = word.replace(/(.)\1+/g, "$1"); + //word = word.replace(/(?<=(.))\1+/g, ""); + + // if(word){ + // words[i] = word; + // } + + if (this.cache && base.length <= this.cache_prt_length) { + this.cache_prt.set(base, word); + if (this.cache_prt.size > this.cache_size) { + this.cache_prt.clear(); + this.cache_prt_length = 0 | this.cache_prt_length / 1.1; + } + } + + //word || words.splice(i--, 1); + word && final.push(word); + } + + //words = final; + // else if(this.filter){ + // for(let i = 0, word; i < words.length; i++){ + // if((word = words[i]) && !this.filter.has(word)){ + // //filtered.push(word); + // words.splice(i--, 1); + // } + // } + // } + + if (this.finalize) { + final = this.finalize(final) || final; + } + + if (this.cache && str.length <= this.cache_enc_length) { + this.cache_enc.set(str, final); + if (this.cache_enc.size > this.cache_size) { + this.cache_enc.clear(); + this.cache_enc_length = 0 | this.cache_enc_length / 1.1; + } + } + + return final; +}; + +// Encoder.prototype.compress = function(str) { +// +// //return str; +// //if(!str) return str; +// +// if(SUPPORT_CACHE && this.cache && str.length <= this.cache_prt_length){ +// if(this.timer){ +// if(this.cache_cmp.has(str)){ +// return this.cache_cmp.get(str); +// } +// } +// else{ +// this.timer = setTimeout(clear, 0, this); +// } +// } +// +// const result = typeof this.compression === "function" +// ? this.compression(str) +// : hash(str); //window.hash(str); +// +// if(SUPPORT_CACHE && this.cache && str.length <= this.cache_prt_length){ +// this.cache_cmp.set(str, result); +// this.cache_cmp.size > this.cache_size && +// this.cache_cmp.clear(); +// } +// +// return result; +// }; + +// function hash(str){ +// return str; +// } + +function clear(self) { + self.timer = null; + self.cache_enc.clear(); + self.cache_prt.clear(); +} \ No newline at end of file diff --git a/dist/module/engine.js b/dist/module/engine.js deleted file mode 100644 index 715226b..0000000 --- a/dist/module/engine.js +++ /dev/null @@ -1,28 +0,0 @@ - -import { searchCache } from "./cache"; - -/** - * @constructor - * @abstract - */ - -function Engine(index) { - - index.prototype.searchCache = searchCache; - - - index.prototype.addAsync = addAsync; - index.prototype.appendAsync = appendAsync; - index.prototype.searchAsync = searchAsync; - index.prototype.updateAsync = updateAsync; - index.prototype.removeAsync = removeAsync; -} - -Engine.prototype.searchCache = searchCache; - - -Engine.prototype.addAsync = addAsync; -Engine.prototype.appendAsync = appendAsync; -Engine.prototype.searchAsync = searchAsync; -Engine.prototype.updateAsync = updateAsync; -Engine.prototype.removeAsync = removeAsync; \ No newline at end of file diff --git a/dist/module/global.js b/dist/module/global.js index bd7da48..fc9a5ce 100644 --- a/dist/module/global.js +++ b/dist/module/global.js @@ -1,5 +1,7 @@ -export const global_lang = {}; -export const global_charset = {}; +import { create_object } from "./common.js"; + +export const global_lang = create_object(); +export const global_charset = create_object(); /** * @param {!string} name @@ -7,7 +9,6 @@ export const global_charset = {}; */ export function registerCharset(name, charset) { - global_charset[name] = charset; } @@ -17,6 +18,5 @@ export function registerCharset(name, charset) { */ export function registerLanguage(name, lang) { - global_lang[name] = lang; } \ No newline at end of file diff --git a/dist/module/index.js b/dist/module/index.js index d6f8703..27c1d39 100644 --- a/dist/module/index.js +++ b/dist/module/index.js @@ -6,90 +6,152 @@ * https://github.com/nextapps-de/flexsearch */ -import { IndexInterface } from "./type.js"; -import { encode as default_encoder } from "./lang/latin/default.js"; -import { create_object, create_object_array, concat, sort_by_length_down, is_array, is_string, is_object, parse_option } from "./common.js"; -import { pipeline, init_stemmer_or_matcher, init_filter } from "./lang.js"; -import { global_lang, global_charset } from "./global.js"; -import apply_async from "./async.js"; -import { intersect } from "./intersect.js"; +import { IndexOptions } from "./type.js"; +import Encoder from "./encoder.js"; import Cache, { searchCache } from "./cache.js"; -import apply_preset from "./preset.js"; +import { KeystoreMap, KeystoreSet } from "./keystore.js"; +import { is_array, is_string } from "./common.js"; import { exportIndex, importIndex } from "./serialize.js"; +import { global_lang, global_charset } from "./global.js"; +import default_encoder from "./lang/latin/default.js"; +import apply_preset from "./preset.js"; +import apply_async from "./async.js"; +import tick from "./profiler.js"; +import "./index/add.js"; +import "./index/search.js"; +import "./index/remove.js"; /** * @constructor - * @implements IndexInterface - * @param {Object=} options - * @param {Object=} _register - * @return {Index} + * @param {IndexOptions|string=} options + * @param {Map|Set|KeystoreSet|KeystoreMap=} _register */ -function Index(options, _register) { +export default function Index(options, _register) { if (!(this instanceof Index)) { - return new Index(options); } - let charset, lang, tmp; - if (options) { options = apply_preset(options); + // charset = options.charset; + // // lang = options.lang; + // + // if(is_string(charset)){ + // + // if(!charset.includes(":")){ + // charset += ":default"; + // } + // + // charset = global_charset[charset]; + // } - - charset = options.charset; - lang = options.lang; - - if (is_string(charset)) { - - if (-1 === charset.indexOf(":")) { - - charset += ":default"; - } - - charset = global_charset[charset]; - } - - if (is_string(lang)) { - - lang = global_lang[lang]; - } + // if(is_string(lang)){ + // + // lang = global_lang[lang]; + // } } else { - options = {}; } - let resolution, - optimize, - context = options.context || {}; + // let charset, lang, tmp; - this.encode = options.encode || charset && charset.encode || default_encoder; - this.register = _register || create_object(); - this.resolution = resolution = options.resolution || 9; - this.tokenize = tmp = charset && charset.tokenize || options.tokenize || "strict"; - this.depth = "strict" === tmp && context.depth; - this.bidirectional = parse_option(context.bidirectional, /* append: */ /* skip update: */ /* skip_update: */!0); - this.optimize = optimize = parse_option(options.optimize, !0); - this.fastupdate = parse_option(options.fastupdate, !0); - this.minlength = options.minlength || 1; - this.boost = options.boost; + const context = options.context || {}, + encoder = options.encode || options.encoder || default_encoder; - // when not using the memory strategy the score array should not pre-allocated to its full length + this.encoder = encoder.encode ? encoder : "object" == typeof encoder ? new Encoder(encoder) : { encode: encoder }; - this.map = optimize ? create_object_array(resolution) : create_object(); - this.resolution_ctx = resolution = context.resolution || 1; - this.ctx = optimize ? create_object_array(resolution) : create_object(); - this.rtl = charset && charset.rtl || options.rtl; - this.matcher = (tmp = options.matcher || lang && lang.matcher) && init_stemmer_or_matcher(tmp, !1); - this.stemmer = (tmp = options.stemmer || lang && lang.stemmer) && init_stemmer_or_matcher(tmp, !0); - this.filter = (tmp = options.filter || lang && lang.filter) && init_filter(tmp); + this.compress = options.compress || options.compression || /* suggest */ /* append: */ /* enrich */!1; - this.cache = (tmp = options.cache) && new Cache(tmp); + + let tmp; + this.resolution = options.resolution || 9; + this.tokenize = tmp = options.tokenize || "strict"; + this.depth = "strict" === tmp && context.depth || 0; + this.bidirectional = !1 !== context.bidirectional; + this.fastupdate = !!options.fastupdate; + this.score = options.score || null; + + tmp = options.keystore || 0; + tmp && (this.keystore = tmp); + + this.map = tmp ? new KeystoreMap(tmp) : new Map(); + this.ctx = tmp ? new KeystoreMap(tmp) : new Map(); + this.reg = _register || (this.fastupdate ? tmp ? new KeystoreMap(tmp) : new Map() : tmp ? new KeystoreSet(tmp) : new Set()); + this.resolution_ctx = context.resolution || 1; + this.rtl = encoder.rtl || options.rtl || !1; + + this.cache = (tmp = options.cache || null) && new Cache(tmp); + + this.resolve = !1 !== options.resolve; + + if (tmp = options.db) { + this.db = tmp.mount(this); + } + this.commit_auto = !1 !== options.commit; + this.commit_task = []; + this.commit_timer = null; } -export default Index; +Index.prototype.mount = function (db) { + if (this.commit_timer) { + clearTimeout(this.commit_timer); + this.commit_timer = null; + } + return db.mount(this); +}; +Index.prototype.commit = function (replace, append) { + if (this.commit_timer) { + clearTimeout(this.commit_timer); + this.commit_timer = null; + } + return this.db.commit(this, replace, append); +}; + +// if(SUPPORT_RESOLVER){ +// Index.prototype.resolve = function(params){ +// return new Resolver(params); +// }; +// } + +/** + * @param {!Index} self + * @param {boolean=} replace + * @param {boolean=} append + */ + +export function autoCommit(self, replace, append) { + if (!self.commit_timer) { + self.commit_timer = setTimeout(function () { + self.commit_timer = null; + self.db.commit(self, replace, append); + }, 0); + } +} + +Index.prototype.clear = function () { + + //this.map = new Map(); + //this.ctx = new Map(); + //this.reg = this.fastupdate ? new Map() : new Set(); + this.map.clear(); + this.ctx.clear(); + this.reg.clear(); + + this.cache && this.cache.clear(); + + + if (this.db) { + this.commit_timer && clearTimeout(this.commit_timer); + this.commit_timer = null; + this.commit_task = [{ clear: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ }]; + //return this.db.clear(); + } + + return this; +}; //Index.prototype.pipeline = pipeline; @@ -99,681 +161,67 @@ export default Index; */ Index.prototype.append = function (id, content) { - return this.add(id, content, !0); }; -// TODO: -// string + number as text -// boolean, null, undefined as ? - -/** - * @param {!number|string} id - * @param {!string} content - * @param {boolean=} _append - * @param {boolean=} _skip_update - */ - -Index.prototype.add = function (id, content, _append, _skip_update) { - - if (content && (id || 0 === id)) { - - if (!_skip_update && !_append && this.register[id]) { - - return this.update(id, content); - } - - content = this.encode("" + content); - const length = content.length; - - if (length) { - - // check context dupes to skip all contextual redundancy along a document - - const dupes_ctx = create_object(), - dupes = create_object(), - depth = this.depth, - resolution = this.resolution; - - - for (let i = 0; i < length; i++) { - let term = content[this.rtl ? length - 1 - i : i], - term_length = term.length; - - - // skip dupes will break the context chain - - if (term && term_length >= this.minlength && (depth || !dupes[term])) { - let score = get_score(resolution, length, i), - token = ""; - - - switch (this.tokenize) { - - case "full": - - if (2 < term_length) { - - for (let x = 0; x < term_length; x++) { - - for (let y = term_length; y > x; y--) { - - if (y - x >= this.minlength) { - - const partial_score = get_score(resolution, length, i, term_length, x); - token = term.substring(x, y); - this.push_index(dupes, token, partial_score, id, _append); - } - } - } - - break; - } - - // fallthrough to next case when term length < 3 - - case "reverse": - - // skip last round (this token exist already in "forward") - - if (1 < term_length) { - - for (let x = term_length - 1; 0 < x; x--) { - - token = term[x] + token; - - if (token.length >= this.minlength) { - - const partial_score = get_score(resolution, length, i, term_length, x); - this.push_index(dupes, token, partial_score, id, _append); - } - } - - token = ""; - } - - // fallthrough to next case to apply forward also - - case "forward": - - if (1 < term_length) { - - for (let x = 0; x < term_length; x++) { - - token += term[x]; - - if (token.length >= this.minlength) { - - this.push_index(dupes, token, score, id, _append); - } - } - - break; - } - - // fallthrough to next case when token has a length of 1 - - default: - // case "strict": - - if (this.boost) { - - score = Math.min(0 | score / this.boost(content, term, i), resolution - 1); - } - - this.push_index(dupes, term, score, id, _append); - - // context is just supported by tokenizer "strict" - - if (depth) { - - if (1 < length && i < length - 1) { - - // check inner dupes to skip repeating words in the current context - - const dupes_inner = create_object(), - resolution = this.resolution_ctx, - keyword = term, - size = Math.min(depth + 1, length - i); - - - dupes_inner[keyword] = 1; - - for (let x = 1; x < size; x++) { - - term = content[this.rtl ? length - 1 - i - x : i + x]; - - if (term && term.length >= this.minlength && !dupes_inner[term]) { - - dupes_inner[term] = 1; - - const context_score = get_score(resolution + (length / 2 > resolution ? 0 : 1), length, i, size - 1, x - 1), - swap = this.bidirectional && term > keyword; - - this.push_index(dupes_ctx, swap ? keyword : term, context_score, id, _append, swap ? term : keyword); - } - } - } - } - } - } - } - - this.fastupdate || (this.register[id] = 1); - } - } - - return this; -}; - -/** - * @param {number} resolution - * @param {number} length - * @param {number} i - * @param {number=} term_length - * @param {number=} x - * @returns {number} - */ - -function get_score(resolution, length, i, term_length, x) { - - // console.log("resolution", resolution); - // console.log("length", length); - // console.log("term_length", term_length); - // console.log("i", i); - // console.log("x", x); - // console.log((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1); - - // the first resolution slot is reserved for the best match, - // when a query matches the first word(s). - - // also to stretch score to the whole range of resolution, the - // calculation is shift by one and cut the floating point. - // this needs the resolution "1" to be handled additionally. - - // do not stretch the resolution more than the term length will - // improve performance and memory, also it improves scoring in - // most cases between a short document and a long document - - return i && 1 < resolution ? length + (term_length || 0) <= resolution ? i + (x || 0) : 0 | (resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1 : 0; -} - -/** - * @private - * @param dupes - * @param value - * @param score - * @param id - * @param {boolean=} append - * @param {string=} keyword - */ - -Index.prototype.push_index = function (dupes, value, score, id, append, keyword) { - - let arr = keyword ? this.ctx : this.map; - - if (!dupes[value] || keyword && !dupes[value][keyword]) { - - if (this.optimize) { - - arr = arr[score]; - } - - if (keyword) { - - dupes = dupes[value] || (dupes[value] = create_object()); - dupes[keyword] = 1; - - arr = arr[keyword] || (arr[keyword] = create_object()); - } else { - - dupes[value] = 1; - } - - arr = arr[value] || (arr[value] = []); - - if (!this.optimize) { - - arr = arr[score] || (arr[score] = []); - } - - if (!append || !arr.includes(id)) { - - arr[arr.length] = id; - - // add a reference to the register for fast updates - - if (this.fastupdate) { - - const tmp = this.register[id] || (this.register[id] = []); - tmp[tmp.length] = arr; - } - } - } -}; - -/** - * @param {string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @returns {Array} - */ - -Index.prototype.search = function (query, limit, options) { - - if (!options) { - - if (!limit && is_object(query)) { - - options = /** @type {Object} */query; - query = options.query; - } else if (is_object(limit)) { - - options = /** @type {Object} */limit; - } - } - - let result = [], - length, - context, - suggest, - offset = 0; - - - if (options) { - - query = options.query || query; - limit = options.limit; - offset = options.offset || 0; - context = options.context; - suggest = options.suggest; - } - - if (query) { - - query = /** @type {Array} */this.encode("" + query); - length = query.length; - - // TODO: solve this in one single loop below - - if (1 < length) { - const dupes = create_object(), - query_new = []; - - - for (let i = 0, count = 0, term; i < length; i++) { - - term = query[i]; - - if (term && term.length >= this.minlength && !dupes[term]) { - - // this fast path can just apply when not in memory-optimized mode - - if (!this.optimize && !suggest && !this.map[term]) { - - // fast path "not found" - - return result; - } else { - - query_new[count++] = term; - dupes[term] = 1; - } - } - } - - query = query_new; - length = query.length; - } - } - - if (!length) { - - return result; - } - - limit || (limit = 100); - - let depth = this.depth && 1 < length && !1 !== context, - index = 0, - keyword; - - - if (depth) { - - keyword = query[0]; - index = 1; - } else { - - if (1 < length) { - - query.sort(sort_by_length_down); - } - } - - for (let arr, term; index < length; index++) { - - term = query[index]; - - // console.log(keyword); - // console.log(term); - // console.log(""); - - if (depth) { - - arr = this.add_result(result, suggest, limit, offset, 2 === length, term, keyword); - - // console.log(arr); - // console.log(result); - - // when suggestion enabled just forward keyword if term was found - // as long as the result is empty forward the pointer also - - if (!suggest || !1 !== arr || !result.length) { - - keyword = term; - } - } else { - - arr = this.add_result(result, suggest, limit, offset, 1 === length, term); - } - - if (arr) { - - return (/** @type {Array} */arr - ); - } - - // apply suggestions on last loop or fallback - - if (suggest && index == length - 1) { - - let length = result.length; - - if (!length) { - - if (depth) { - - // fallback to non-contextual search when no result was found - - depth = 0; - index = -1; - - continue; - } - - return result; - } else if (1 === length) { - - // fast path optimization - - return single_result(result[0], limit, offset); - } - } - } - - return intersect(result, limit, offset, suggest); -}; - -/** - * Returns an array when the result is done (to stop the process immediately), - * returns false when suggestions is enabled and no result was found, - * or returns nothing when a set was pushed successfully to the results - * - * @private - * @param {Array} result - * @param {Array} suggest - * @param {number} limit - * @param {number} offset - * @param {boolean} single_term - * @param {string} term - * @param {string=} keyword - * @return {Array>|boolean|undefined} - */ - -Index.prototype.add_result = function (result, suggest, limit, offset, single_term, term, keyword) { - let word_arr = [], - arr = keyword ? this.ctx : this.map; - - - if (!this.optimize) { - - arr = get_array(arr, term, keyword, this.bidirectional); - } - - if (arr) { - - let count = 0; - const arr_len = Math.min(arr.length, keyword ? this.resolution_ctx : this.resolution); - - // relevance: - for (let x = 0, size = 0, tmp, len; x < arr_len; x++) { - - tmp = arr[x]; - - if (tmp) { - - if (this.optimize) { - - tmp = get_array(tmp, term, keyword, this.bidirectional); - } - - if (offset) { - - if (tmp && single_term) { - - len = tmp.length; - - if (len <= offset) { - - offset -= len; - tmp = null; - } else { - - tmp = tmp.slice(offset); - offset = 0; - } - } - } - - if (tmp) { - - // keep score (sparse array): - //word_arr[x] = tmp; - - // simplified score order: - word_arr[count++] = tmp; - - if (single_term) { - - size += tmp.length; - - if (size >= limit) { - - // fast path optimization - - break; - } - } - } - } - } - - if (count) { - - if (single_term) { - - // fast path optimization - // offset was already applied at this point - - return single_result(word_arr, limit, 0); - } - - result[result.length] = word_arr; - return; - } - } - - // return an empty array will stop the loop, - // to prevent stop when using suggestions return a false value - - return !suggest && word_arr; -}; - -function single_result(result, limit, offset) { - - if (1 === result.length) { - - result = result[0]; - } else { - - result = concat(result); - } - - return offset || result.length > limit ? result.slice(offset, offset + limit) : result; -} - -function get_array(arr, term, keyword, bidirectional) { - - if (keyword) { - - // the frequency of the starting letter is slightly less - // on the last half of the alphabet (m-z) in almost every latin language, - // so we sort downwards (https://en.wikipedia.org/wiki/Letter_frequency) - - const swap = bidirectional && term > keyword; - - arr = arr[swap ? term : keyword]; - arr = arr && arr[swap ? keyword : term]; - } else { - - arr = arr[term]; - } - - return arr; -} - Index.prototype.contain = function (id) { - return !!this.register[id]; + if (this.db) { + return this.db.has(id); + } + + return this.reg.has(id); }; Index.prototype.update = function (id, content) { + // todo check the async part + if (this.async /*|| (SUPPORT_PERSISTENT && this.db)*/) { + const self = this, + res = this.remove(id); + + return res.then ? res.then(() => self.add(id, content)) : this.add(id, content); + } + return this.remove(id).add(id, content); }; -/** - * @param {boolean=} _skip_deletion - */ - -Index.prototype.remove = function (id, _skip_deletion) { - - const refs = this.register[id]; - - if (refs) { - - if (this.fastupdate) { - - // fast updates performs really fast but did not fully cleanup the key entries - - for (let i = 0, tmp; i < refs.length; i++) { - - tmp = refs[i]; - tmp.splice(tmp.indexOf(id), 1); - } - } else { - - remove_index(this.map, id, this.resolution, this.optimize); - - if (this.depth) { - - remove_index(this.ctx, id, this.resolution_ctx, this.optimize); - } - } - - _skip_deletion || delete this.register[id]; - - if (this.cache) { - - this.cache.del(id); - } - } - - return this; -}; - /** * @param map - * @param id - * @param res - * @param optimize - * @param {number=} resolution * @return {number} */ -function remove_index(map, id, res, optimize, resolution) { +function cleanup_index(map) { let count = 0; if (is_array(map)) { - - // the first array is the score array in both strategies - - if (!resolution) { - - resolution = Math.min(map.length, res); - - for (let x = 0, arr; x < resolution; x++) { - - arr = map[x]; - - if (arr) { - - count = remove_index(arr, id, res, optimize, resolution); - - if (!optimize && !count) { - - // when not memory optimized the score index should removed - - delete map[x]; - } - } - } - } else { - - const pos = map.indexOf(id); - - if (-1 !== pos) { - - // fast path, when length is 1 or lower then the whole field gets deleted - - if (1 < map.length) { - - map.splice(pos, 1); - count++; - } - } else { - - count++; - } + for (let i = 0, arr; i < map.length; i++) { + (arr = map[i]) && (count += arr.length); } - } else { + } else for (const item of map) { + const key = item[0], + value = item[1], + tmp = cleanup_index(value); - for (let key in map) { - - count = remove_index(map[key], id, res, optimize, resolution); - - if (!count) { - - delete map[key]; - } - } + tmp ? count += tmp : map.delete(key); } return count; } +Index.prototype.cleanup = function () { + + if (!this.fastupdate) { + return this; + } + + cleanup_index(this.map); + this.depth && cleanup_index(this.ctx); + + return this; +}; + Index.prototype.searchCache = searchCache; diff --git a/dist/module/index/add.js b/dist/module/index/add.js new file mode 100644 index 0000000..0f47234 --- /dev/null +++ b/dist/module/index/add.js @@ -0,0 +1,261 @@ + +import { create_object } from "../common.js"; +import Index, { autoCommit } from "../index.js"; +import default_compress from "../compress.js"; +import { KeystoreArray } from "../keystore.js"; + +// TODO: +// string + number as text +// boolean, null, undefined as ? + + +/** + * @param {!number|string} id + * @param {!string} content + * @param {boolean=} _append + * @param {boolean=} _skip_update + */ + +Index.prototype.add = function (id, content, _append, _skip_update) { + + if (content && (id || 0 === id)) { + + // todo check skip_update + //_skip_update = false; + + if (!_skip_update && !_append) { + if (this.reg.has(id)) { + return this.update(id, content); + } + } + + // do not force a string as input + // https://github.com/nextapps-de/flexsearch/issues/432 + content = this.encoder.encode(content); + const word_length = content.length; + + if (word_length) { + + // check context dupes to skip all contextual redundancy along a document + + const dupes_ctx = create_object(), + dupes = create_object(), + depth = this.depth, + resolution = this.resolution; + + + for (let i = 0; i < word_length; i++) { + let term = content[this.rtl ? word_length - 1 - i : i], + term_length = term.length; + + + // skip dupes will break the context chain + + if (term_length /*&& (term_length >= this.minlength)*/ && (depth || !dupes[term])) { + let score = this.score ? this.score(content, term, i, null, 0) : get_score(resolution, word_length, i), + token = ""; + + + switch (this.tokenize) { + + case "full": + if (2 < term_length) { + for (let x = 0; x < term_length; x++) { + for (let y = term_length; y > x; y--) { + + //if((y - x) >= this.minlength){ + token = term.substring(x, y); + const partial_score = this.score ? this.score(content, term, i, token, x) : get_score(resolution, word_length, i, term_length, x); + this.push_index(dupes, token, partial_score, id, _append); + //} + } + } + break; + } + // fallthrough to next case when term length < 3 + case "reverse": + // skip last round (this token exist already in "forward") + if (1 < term_length) { + for (let x = term_length - 1; 0 < x; x--) { + token = term[x] + token; + //if(token.length >= this.minlength){ + const partial_score = this.score ? this.score(content, term, i, token, x) : get_score(resolution, word_length, i, term_length, x); + this.push_index(dupes, token, partial_score, id, _append); + //} + } + token = ""; + } + + // fallthrough to next case to apply forward also + case "forward": + if (1 < term_length) { + for (let x = 0; x < term_length; x++) { + token += term[x]; + //if(token.length >= this.minlength){ + this.push_index(dupes, token, score, id, _append); + //} + } + break; + } + + // fallthrough to next case when token has a length of 1 + default: + // case "strict": + + // todo move boost to search + // if(this.boost){ + // score = Math.min((score / this.boost(content, term, i)) | 0, resolution - 1); + // } + + this.push_index(dupes, term, score, id, _append); + + // context is just supported by tokenizer "strict" + if (depth) { + + if (1 < word_length && i < word_length - 1) { + + // check inner dupes to skip repeating words in the current context + const dupes_inner = create_object(), + resolution = this.resolution_ctx, + keyword = term, + size = Math.min(depth + 1, word_length - i); + + + dupes_inner[keyword] = 1; + + for (let x = 1; x < size; x++) { + + term = content[this.rtl ? word_length - 1 - i - x : i + x]; + + if (term /*&& (term.length >= this.minlength)*/ && !dupes_inner[term]) { + + dupes_inner[term] = 1; + + const context_score = this.score ? this.score(content, keyword, i, term, x) : get_score(resolution + (word_length / 2 > resolution ? 0 : 1), word_length, i, size - 1, x - 1), + swap = this.bidirectional && term > keyword; + + this.push_index(dupes_ctx, swap ? keyword : term, context_score, id, _append, swap ? term : keyword); + } + } + } + } + } + } + } + + this.fastupdate || this.reg.add(id); + } else { + content = ""; + } + } + + if (this.db) { + // when the term has no valid content (e.g. empty), + // then it was not added to the ID registry for removal + content || this.commit_task.push({ del: id }); + this.commit_auto && autoCommit(this); + } + + return this; +}; + +/** + * @private + * @param dupes + * @param term + * @param score + * @param id + * @param {boolean=} append + * @param {string=} keyword + */ + +Index.prototype.push_index = function (dupes, term, score, id, append, keyword) { + let arr = keyword ? this.ctx : this.map, + tmp; + + + if (!dupes[term] || !keyword || !(tmp = dupes[term])[keyword]) { + + if (keyword) { + + dupes = tmp || (dupes[term] = create_object()); + dupes[keyword] = 1; + + if (this.compress) { + keyword = default_compress(keyword); + } + + tmp = arr.get(keyword); + tmp ? arr = tmp : arr.set(keyword, arr = new Map()); + } else { + + dupes[term] = 1; + } + + if (this.compress) { + term = default_compress(term); + } + + tmp = arr.get(term); + tmp ? arr = tmp : arr.set(term, arr = tmp = []); + // the ID array will be upgraded dynamically + arr = arr[score] || (arr[score] = []); + + if (!append || !arr.includes(id)) { + + // auto-upgrade to keystore array if max size exceeded + if (2147483647 === arr.length /*|| !(arr instanceof KeystoreArray)*/) { + const keystore = new KeystoreArray(arr); + if (this.fastupdate) { + for (let value of this.reg.values()) { + if (value.includes(arr)) { + value[value.indexOf(arr)] = keystore; + } + } + } + tmp[score] = arr = keystore; + } + + + arr.push(id); + + // add a reference to the register for fast updates + if (this.fastupdate) { + const tmp = this.reg.get(id); + tmp ? tmp.push(arr) : this.reg.set(id, [arr]); + } + } + } +}; + +/** + * @param {number} resolution + * @param {number} length + * @param {number} i + * @param {number=} term_length + * @param {number=} x + * @returns {number} + */ + +function get_score(resolution, length, i, term_length, x) { + + // console.log("resolution", resolution); + // console.log("length", length); + // console.log("term_length", term_length); + // console.log("i", i); + // console.log("x", x); + // console.log((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1); + + // the first resolution slot is reserved for the best match, + // when a query matches the first word(s). + + // also to stretch score to the whole range of resolution, the + // calculation is shift by one and cut the floating point. + // this needs the resolution "1" to be handled additionally. + + // do not stretch the resolution more than the term length will + // improve performance and memory, also it improves scoring in + // most cases between a short document and a long document + + return i && 1 < resolution ? length + (term_length || 0) <= resolution ? i + (x || 0) : 0 | (resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1 : 0; +} \ No newline at end of file diff --git a/dist/module/index/remove.js b/dist/module/index/remove.js new file mode 100644 index 0000000..b79ad2c --- /dev/null +++ b/dist/module/index/remove.js @@ -0,0 +1,137 @@ + +import { create_object, is_array } from "../common.js"; +import Index, { autoCommit } from "../index.js"; +import default_compress from "../compress.js"; + +/** + * @param {boolean=} _skip_deletion + */ + +Index.prototype.remove = function (id, _skip_deletion) { + + const refs = this.reg.size && (this.fastupdate ? this.reg.get(id) : this.reg.has(id)); + + if (refs) { + + if (this.fastupdate) { + + // fast updates did not fully cleanup the key entries + + for (let i = 0, tmp; i < refs.length; i++) { + if (tmp = refs[i]) { + // todo check + //if(tmp.length < 1) throw new Error("invalid length"); + //if(tmp.indexOf(id) < 0) throw new Error("invalid id"); + if (2 > tmp.length) { + tmp.pop(); + } else { + const index = tmp.indexOf(id); + index === refs.length - 1 ? tmp.pop() : tmp.splice(index, 1); + } + } + } + + // todo variation which cleans up, requires to push [ctx, key] instead of arr to the index.reg + // for(let i = 0, arr, term, keyword; i < refs.length; i++){ + // arr = refs[i]; + // if(typeof arr === "string"){ + // arr = this.map.get(term = arr); + // } + // else{ + // arr = this.ctx.get(keyword = arr[0]); + // arr && (arr = arr.get(arr[1])); + // } + // let counter = 0, found; + // if(arr && arr.length){ + // for(let j = 0, tmp; j < arr.length; j++){ + // if((tmp = arr[j])){ + // if(!found && tmp.length){ + // const index = tmp.indexOf(id); + // if(index >= 0){ + // tmp.splice(index, 1); + // // the index [ctx, key]:[res, id] is unique + // found = 1; + // } + // } + // if(tmp.length){ + // counter++; + // if(found){ + // break; + // } + // } + // else{ + // delete arr[j]; + // } + // } + // } + // } + // if(!counter){ + // keyword + // ? this.ctx.delete(keyword) + // : this.map.delete(term); + // } + // } + } else { + + remove_index(this.map, id /*, this.resolution*/); + this.depth && remove_index(this.ctx, id /*, this.resolution_ctx*/); + } + + _skip_deletion || this.reg.delete(id); + } + + if (this.db) { + this.commit_task.push({ del: id }); + this.commit_auto && autoCommit(this); + //return this.db.remove(id); + } + + // the cache could be used outside the InMemory store + if (this.cache) { + this.cache.remove(id); + } + + return this; +}; + +/** + * @param map + * @param id + * @return {number} + */ + +function remove_index(map, id) { + + // a check counter of filled resolution slots + // to prevent removing the field + let count = 0; + + if (is_array(map)) { + for (let x = 0, arr, index; x < map.length; x++) { + if ((arr = map[x]) && arr.length) { + index = arr.indexOf(id); + if (0 <= index) { + if (1 < arr.length) { + arr.splice(index, 1); + count++; + } else { + // remove resolution slot + delete map[x]; + } + // the index key:[res, id] is unique + break; + } else { + count++; + } + } + } + } else for (let item of map) { + const key = item[0], + value = item[1], + tmp = remove_index(value, id); + + tmp ? count += tmp : map.delete(key); + } + + return count; +} \ No newline at end of file diff --git a/dist/module/index/search.js b/dist/module/index/search.js new file mode 100644 index 0000000..44b4cb1 --- /dev/null +++ b/dist/module/index/search.js @@ -0,0 +1,432 @@ + + +import { SearchOptions } from "../type.js"; +import { create_object, is_object, sort_by_length_down } from "../common.js"; +import Index from "../index.js"; +import default_compress from "../compress.js"; +import Resolver from "../resolver.js"; +import { intersect } from "../intersect.js"; +import resolve_default from "../resolve/default.js"; + +let global_resolve = 1; +export function set_resolve(resolve) { + global_resolve = resolve; +} + +/** + * @param {string|SearchOptions} query + * @param {number|SearchOptions=} limit + * @param {SearchOptions=} options + * @returns {Array|Resolver|Promise} + */ + +Index.prototype.search = function (query, limit, options) { + + if (!options) { + if (!limit && is_object(query)) { + options = /** @type {!SearchOptions} */query; + query = ""; + } else if (is_object(limit)) { + options = /** @type {!SearchOptions} */limit; + limit = 0; + } + } + + let result = [], + length, + context, + suggest, + offset = 0, + resolve, + enrich, + tag; + + + if (options) { + query = options.query || query; + limit = options.limit || limit; + offset = options.offset || 0; + context = options.context; + suggest = options.suggest; + resolve = global_resolve && + /* suggest */ /* append: */ /* enrich */!1 !== options.resolve; + resolve || (global_resolve = 0); + enrich = resolve && options.enrich; + tag = this.db && options.tag; + } else { + resolve = this.resolve || global_resolve; + } + + // todo: term deduplication during encoding when context is disabled + + // do not force a string as input + // https://github.com/nextapps-de/flexsearch/issues/432 + query = /** @type {Array} */this.encoder.encode(query); + length = query.length; + limit || !resolve || (limit = 100); + + // fast path single term + if (1 === length) { + return single_term_query.call(this, query[0], // term + "", // ctx + limit, offset, resolve, enrich, tag); + } + + // TODO: dedupe terms within encoder? + // TODO: deduplication will break the context chain + + context = this.depth && !1 !== context; + + // fast path single context + if (2 === length && context && !suggest) { + return single_term_query.call(this, query[0], // term + query[1], // ctx + limit, offset, resolve, enrich, tag); + } + + let maxlength = 0, + minlength = 0; + + + if (1 < length) { + + // term deduplication will break the context chain + // todo add context to dupe check + const dupes = create_object(), + query_new = []; + + + // if(context){ + // keyword = query[0]; + // dupes[keyword] = 1; + // query_new.push(keyword); + // maxlength = minlength = keyword.length; + // i = 1; + // } + + for (let i = 0, term; i < length; i++) { + + term = query[i]; + + if (term && !dupes[term]) { + + // todo add keyword check + // this fast path can't apply to persistent indexes + if (!suggest && !this.db && !this.get_array(term /*, keyword*/)) { + + // fast path "not found" + return resolve ? result : new Resolver(result); + } else { + + query_new.push(term); + dupes[term] = 1; + } + + const term_length = term.length; + maxlength = Math.max(maxlength, term_length); + minlength = minlength ? Math.min(minlength, term_length) : term_length; + } + // else if(term && (!this.depth || context === false)){ + // query_new.push(term); + // } + } + + query = query_new; + length = query.length; + } + + // the term length could be changed after deduplication + + if (!length) { + return resolve ? result : new Resolver(result); + } + + let index = 0, + keyword; + + // fast path single term + if (1 === length) { + return single_term_query.call(this, query[0], // term + "", // ctx + limit, offset, resolve, enrich, tag); + } + + // fast path single context + if (2 === length && context && !suggest) { + return single_term_query.call(this, query[0], // term + query[1], // ctx + limit, offset, resolve, enrich, tag); + } + + if (1 < length) { + if (context) { + // start with context right away + keyword = query[0]; + index = 1; + } + // todo + else if (9 < maxlength && 3 < maxlength / minlength) { + // sorting terms will break the context chain + // bigger terms has less occurrence + // this might also reduce the intersection task + // todo check intersection order + query.sort(sort_by_length_down); + } + } + + if (this.db) { + + if (this.db.search) { + // when the configuration is not supported it returns false + const result = this.db.search(this, query, limit, offset, suggest, resolve, enrich, tag); + if (!1 !== result) return result; + } + + const self = this; + return async function () { + + for (let arr, term; index < length; index++) { + + term = query[index]; + + if (keyword) { + + arr = await self.get_array(term, keyword); + arr = add_result(arr, result, suggest, self.resolution_ctx, + /** @type {!number} */limit, offset, 2 === length + /*, term, keyword*/ + ); + + // the context is a moving window where the keyword is going forward like a cursor + // 1. when suggestion enabled just forward keyword if term was found + // 2. as long as the result is empty forward the pointer also + if (!suggest || !1 !== arr || !result.length) { + keyword = term; + } + } else { + + arr = await self.get_array(term); + arr = add_result(arr, result, suggest, self.resolution, + /** @type {!number} */limit, offset, 1 === length + /*, term*/ + ); + } + + // limit reached + if (arr) { + return arr; + } + + // apply suggestions on last loop + if (suggest && index == length - 1) { + let length = result.length; + if (!length) { + // fallback to non-contextual search when no result was found + if (keyword) { + keyword = ""; + index = -1; + continue; + } + return result; + } else if (1 === length) { + return resolve ? resolve_default(result[0], /** @type {number} */limit, offset) : new Resolver(result[0]); + } + } + } + + return resolve ? intersect(result, /** @type {number} */limit, offset, suggest) : new Resolver(result[0]); + }(); + } + + for (let arr, term; index < length; index++) { + + term = query[index]; + + if (keyword) { + + arr = this.get_array(term, keyword); + arr = /*this.*/add_result(arr, result, suggest, this.resolution_ctx, + /** @type {!number} */limit, offset, 2 === length + /*, term, keyword*/ + ); + + // 1. when suggestion enabled just forward keyword if term was found + // 2. as long as the result is empty forward the pointer also + if (!suggest || !1 !== arr || !result.length) { + keyword = term; + } + } else { + + arr = this.get_array(term); + arr = /*this.*/add_result(arr, result, suggest, this.resolution, + /** @type {!number} */limit, offset, 1 === length + /*, term*/ + ); + } + + // limit reached + if (arr) { + return (/** @type {Array} */arr + ); + } + + // apply suggestions on last loop + if (suggest && index == length - 1) { + const length = result.length; + if (!length) { + // fallback to non-contextual search when no result was found + if (keyword) { + keyword = ""; + index = -1; + continue; + } + return result; + } else if (1 === length) { + return resolve ? resolve_default(result[0], limit, offset) : new Resolver(result[0]); + } + } + } + + return resolve ? intersect(result, limit, offset, suggest) : new Resolver(result[0]); +}; + +/** + * @param term + * @param keyword + * @param limit + * @param offset + * @param resolve + * @param enrich + * @param tag + * @this Index + * @return {Array|Resolver} + */ + +function single_term_query(term, keyword, limit, offset, resolve, enrich, tag) { + + const result = this.get_array(term, keyword, limit, offset, resolve, enrich, tag); + + if (this.db) { + return result.then(function (result) { + if (resolve) return result; + return result && result.length ? resolve ? resolve_default(result, limit, offset) : new Resolver(result) : resolve ? [] : new Resolver([]); + }); + } + + return result && result.length ? resolve ? resolve_default(result, limit, offset) : new Resolver(result) : resolve ? [] : new Resolver([]); +} + +/** + * Returns a 1-dimensional finalized array when the result is done (fast path return), + * returns false when suggestions is enabled and no result was found, + * or returns nothing when a set was pushed successfully to the results + * + * @private + * @param {Array} arr + * @param {Array} result + * @param {Array} suggest + * @param {number} resolution + * @param {number} limit + * @param {number} offset + * @param {boolean} single_term + * @return {Array|boolean|undefined} + */ + +function add_result(arr, result, suggest, resolution, limit, offset, single_term /*, term, keyword*/) { + + let word_arr = []; + //let arr;// = keyword ? this.ctx : this.map; + //arr = this.get_array(term, keyword); + + if (arr) { + + //const resolution = Math.min(arr.length, keyword ? this.resolution_ctx : this.resolution); + // apply reduced resolution for queries + resolution = Math.min(arr.length, resolution); + + for (let x = 0, size = 0, tmp; x < resolution; x++) { + if (tmp = arr[x]) { + + if (offset) { + // apply offset right here on single terms + if (tmp && single_term) { + if (tmp.length <= offset) { + offset -= tmp.length; + tmp = null; + } else { + tmp = tmp.slice(offset); + offset = 0; + } + } + } + + if (tmp) { + + // keep score (sparse array): + word_arr[x] = tmp; + // simplified score order: + //word_arr.push(tmp); + + if (single_term) { + size += tmp.length; + if (size >= limit) { + // fast path: + // a single term does not need to pre-collect results + break; + } + } + } + } + } + + if (word_arr.length) { + if (single_term) { + // fast path optimization + // offset was already applied at this point + // return an array will stop the query process immediately + return resolve_default(word_arr, limit, 0); + } + + result.push(word_arr); + // return nothing will continue the query + return; + } + } + + // 1. return an empty array will stop the loop + // 2. return a false value to prevent stop when using suggestions + return !suggest && word_arr; +} + +Index.prototype.get_array = function (term, keyword, limit, offset, resolve, enrich, tag) { + + let arr, swap; + + if (keyword) { + swap = this.bidirectional && term > keyword; + } + + if (this.compress) { + term = default_compress(term); + keyword && (keyword = default_compress(keyword)); + } + + if (this.db) { + return keyword ? this.db.get(swap ? keyword : term, // term + swap ? term : keyword, // ctx + limit, offset, resolve, enrich, tag) : this.db.get(term, "", // ctx + limit, offset, resolve, enrich, tag); + } + + if (keyword) { + // the frequency of the starting letter is slightly less + // on the last half of the alphabet (m-z) in almost every latin language, + // so we sort downwards (https://en.wikipedia.org/wiki/Letter_frequency) + arr = this.ctx.get(swap ? term : keyword); + arr = arr && arr.get(swap ? keyword : term); + } else { + arr = this.map.get(term); + } + + return arr; +}; \ No newline at end of file diff --git a/dist/module/intersect.js b/dist/module/intersect.js index 2935747..8b5575a 100644 --- a/dist/module/intersect.js +++ b/dist/module/intersect.js @@ -1,4 +1,231 @@ -import { create_object, concat } from "./common.js"; +import { create_object, concat, sort_by_length_up, get_max_len } from "./common.js"; + +/* + + from -> result[ + res[score][id], + res[score][id], + ] + + to -> [id] + + */ + +/** + * Implementation based on Object[key] provides better suggestions + * capabilities and has less performance scaling issues on large indexes. + * + * @param arrays + * @param limit + * @param offset + * @param {boolean|Array=} suggest + * @param {boolean=} resolve + * @returns {Array} + */ + +export function intersect(arrays, limit, offset, suggest) { + + const length = arrays.length; + + // todo remove + // if(length < 2){ + // throw new Error("Not optimized intersect"); + // } + + let result = [], + size = 0, + check, + check_suggest, + check_new, + res_arr; + + + if (suggest) { + suggest = []; + } + + // 1. a reversed order prioritize the order of words from a query + // because it ends with the first term. + // 2. process terms in reversed order often has advantage for + // the fast path "end reached". + + // alternatively the results could be sorted by length up + //arrays.sort(sort_by_length_up); + + // todo the outer loop should be the res array instead of term array, + // this isn't just simple because the intersection calculation needs to reflect this + //const maxlen = get_max_len(arrays); + + for (let x = length - 1, found; 0 <= x; x--) { + //for(let x = 0, found; x < length; x++){ + + res_arr = arrays[x]; + check_new = create_object(); + found = !check; + + // process relevance in forward order (direction is + // important for adding IDs during the last round) + + for (let y = 0, ids; y < res_arr.length; y++) { + + ids = res_arr[y]; + if (!ids || !ids.length) continue; + + for (let z = 0, id; z < ids.length; z++) { + + id = ids[z]; + + // check exists starting from the 2nd slot + if (check) { + if (check[id]) { + + // check if in last round + if (!x) { + //if(x === length - 1){ + + if (offset) { + offset--; + } else { + + result[size++] = id; + + if (size === limit) { + // fast path "end reached" + return result; /*resolve === false + ? { result, suggest } + :*/ + } + } + } + + if (x || suggest) { + //if((x < length - 1) || suggest){ + check_new[id] = 1; + } + + found = /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */!0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; + } + + if (suggest) { + + if (!check_suggest[id]) { + check_suggest[id] = 1; + const arr = suggest[y] || (suggest[y] = []); + arr.push(id); + } + + // OLD: + // + // check_idx = (check_suggest[id] || 0) + 1; + // check_suggest[id] = check_idx; + // + // // do not adding IDs which are already included in the result (saves one loop) + // // the first intersection match has the check index 2, so shift by -2 + // + // if(check_idx < length){ + // + // const tmp = suggest[check_idx - 2] || (suggest[check_idx - 2] = []); + // tmp[tmp.length] = id; + // } + } + } else { + + // pre-fill in first round + check_new[id] = 1; + } + } + } + + if (suggest) { + + // re-use the first pre-filled check for suggestions + check || (check_suggest = check_new); + } else if (!found) { + + return []; + } + + check = check_new; + } + + // return intermediate result + // if(resolve === false){ + // return { result, suggest }; + // } + + if (suggest) { + + // needs to iterate in reverse direction + for (let x = suggest.length - 1, ids, len; 0 <= x; x--) { + + ids = suggest[x]; + len = ids.length; + + for (let y = 0, id; y < len; y++) { + + id = ids[y]; + + if (!check[id]) { + + if (offset) { + offset--; + } else { + + result[size++] = id; + + if (size === limit) { + // fast path "end reached" + return result; + } + } + + check[id] = 1; + } + } + } + } + + return result; +} + +/** + * @param mandatory + * @param arrays + * @returns {Array} + */ + +export function intersect_union(mandatory, arrays) { + const check = create_object(), + union = create_object(), + result = []; + + + for (let x = 0; x < mandatory.length; x++) { + + check[mandatory[x]] = 1; + } + + for (let x = 0, arr; x < arrays.length; x++) { + + arr = arrays[x]; + + for (let y = 0, id; y < arr.length; y++) { + + id = arr[y]; + + if (check[id]) { + + if (!union[id]) { + + union[id] = 1; + result.push(id); + } + } + } + } + + return result; +} /** * Implementation based on Array.includes() provides better performance, @@ -192,203 +419,4 @@ import { create_object, concat } from "./common.js"; // } // // return result; -// } - -/** - * Implementation based on Object[key] provides better suggestions - * capabilities and has less performance scaling issues on large indexes. - * - * @param arrays - * @param limit - * @param offset - * @param {boolean|Array=} suggest - * @returns {Array} - */ - -export function intersect(arrays, limit, offset, suggest) { - - const length = arrays.length; - let result = [], - check, - check_suggest, - size = 0; - - - if (suggest) { - - suggest = []; - } - - // process terms in reversed order often has advantage for the fast path "end reached". - // also a reversed order prioritize the order of words from a query. - - for (let x = length - 1; 0 <= x; x--) { - const word_arr = arrays[x], - word_arr_len = word_arr.length, - check_new = create_object(); - - - let found = !check; - - // process relevance in forward order (direction is - // important for adding IDs during the last round) - - for (let y = 0; y < word_arr_len; y++) { - const arr = word_arr[y], - arr_len = arr.length; - - - if (arr_len) { - - // loop through IDs - - for (let z = 0, check_idx, id; z < arr_len; z++) { - - id = arr[z]; - - if (check) { - - if (check[id]) { - - // check if in last round - - if (!x) { - - if (offset) { - - offset--; - } else { - - result[size++] = id; - - if (size === limit) { - - // fast path "end reached" - - return result; - } - } - } - - if (x || suggest) { - - check_new[id] = 1; - } - - found = /* append: */ /* skip update: */ /* skip_update: */!0; - } - - if (suggest) { - - check_idx = (check_suggest[id] || 0) + 1; - check_suggest[id] = check_idx; - - // do not adding IDs which are already included in the result (saves one loop) - // the first intersection match has the check index 2, so shift by -2 - - if (check_idx < length) { - - const tmp = suggest[check_idx - 2] || (suggest[check_idx - 2] = []); - tmp[tmp.length] = id; - } - } - } else { - - // pre-fill in first round - - check_new[id] = 1; - } - } - } - } - - if (suggest) { - - // re-use the first pre-filled check for suggestions - - check || (check_suggest = check_new); - } else if (!found) { - - return []; - } - - check = check_new; - } - - if (suggest) { - - // needs to iterate in reverse direction - - for (let x = suggest.length - 1, arr, len; 0 <= x; x--) { - - arr = suggest[x]; - len = arr.length; - - for (let y = 0, id; y < len; y++) { - - id = arr[y]; - - if (!check[id]) { - - if (offset) { - - offset--; - } else { - - result[size++] = id; - - if (size === limit) { - - // fast path "end reached" - - return result; - } - } - - check[id] = 1; - } - } - } - } - - return result; -} - -/** - * @param mandatory - * @param arrays - * @returns {Array} - */ - -export function intersect_union(mandatory, arrays) { - const check = create_object(), - union = create_object(), - result = []; - - - for (let x = 0; x < mandatory.length; x++) { - - check[mandatory[x]] = 1; - } - - for (let x = 0, arr; x < arrays.length; x++) { - - arr = arrays[x]; - - for (let y = 0, id; y < arr.length; y++) { - - id = arr[y]; - - if (check[id]) { - - if (!union[id]) { - - union[id] = 1; - result[result.length] = id; - } - } - } - } - - return result; -} \ No newline at end of file +// } \ No newline at end of file diff --git a/dist/module/keystore.js b/dist/module/keystore.js new file mode 100644 index 0000000..3f7b6d7 --- /dev/null +++ b/dist/module/keystore.js @@ -0,0 +1,388 @@ +import { create_object } from "./common.js"; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreObj(bitlength = 8) { + + if (!(this instanceof KeystoreObj)) { + return new KeystoreObj(bitlength); + } + + this.index = create_object(); + this.keys = []; + + if (32 < bitlength) { + this.crc = lcg64; + this.bit = BigInt(bitlength); + } else { + this.crc = lcg; + this.bit = bitlength; + } + + return (/*this.proxy =*/new Proxy(this, { + get(target, key) { + const address = target.crc(key), + obj = target.index[address]; + + return obj && obj[key]; + }, + set(target, key, value) { + const address = target.crc(key); + let obj = target.index[address]; + if (!obj) { + target.index[address] = obj = create_object(); + target.keys.push(address); + } + obj[key] = value; + return (/* tag? */ /* stringify */ + /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ + // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/ + ); + }, + delete(target, key) { + const address = target.crc(key), + obj = target.index[address]; + + obj && delete obj[key]; + return !0; + } + }) + ); +} + +KeystoreObj.prototype.clear = function () { + this.index = create_object(); + this.keys = []; +}; + +// KeystoreObj.prototype.destroy = function(){ +// this.index = null; +// this.keys = null; +// this.proxy = null; +// }; + +function _slice(self, start, end, splice) { + let arr = []; + for (let i = 0, index; i < self.index.length; i++) { + index = self.index[i]; + if (start >= index.length) { + start -= index.length; + } else { + const tmp = index[splice ? "splice" : "slice"](start, end), + length = tmp.length; + + if (length) { + arr = arr.length ? arr.concat(tmp) : tmp; + end -= length; + if (splice) self.length -= length; + if (!end) break; + } + start = 0; + } + } + return arr; +} + +/** + * @param arr + * @constructor + */ + +export function KeystoreArray(arr) { + + if (!(this instanceof KeystoreArray)) { + return new KeystoreArray(arr); + } + + this.index = arr ? [arr] : []; + this.length = arr ? arr.length : 0; + const self = this; + + return (/*this.proxy =*/new Proxy([], { + get(target, key) { + if ("length" === key) { + return self.length; + } + if ("push" === key) { + return function (value) { + self.index[self.index.length - 1].push(value); + self.length++; + }; + } + if ("pop" === key) { + return function () { + if (self.length) { + self.length--; + return self.index[self.index.length - 1].pop(); + } + }; + } + if ("indexOf" === key) { + return function (key) { + let index = 0; + for (let i = 0, arr, tmp; i < self.index.length; i++) { + arr = self.index[i]; + //if(!arr.includes(key)) continue; + tmp = arr.indexOf(key); + if (0 <= tmp) return index + tmp; + index += arr.length; + } + return -1; + }; + } + if ("includes" === key) { + return function (key) { + for (let i = 0; i < self.index.length; i++) { + if (self.index[i].includes(key)) { + return !0; + } + } + return (/* suggest */ /* append: */ /* enrich */!1 + ); + }; + } + if ("slice" === key) { + return function (start, end) { + return _slice(self, start || 0, end || self.length, !1); + }; + } + if ("splice" === key) { + return function (start, end) { + return _slice(self, start || 0, end || self.length, !0); + }; + } + if ("constructor" === key) { + return Array; + } + if ("symbol" == typeof key /*|| isNaN(key)*/) { + // not supported + return; + } + const arr = self.index[0 | key / 2147483648]; + + return arr && arr[key]; + }, + set(target, key, value) { + const index = 0 | key / 2147483648, + arr = self.index[index] || (self.index[index] = []); + + arr[key] = value; + self.length++; + return !0; + } + }) + ); +} + +KeystoreArray.prototype.clear = function () { + this.index.length = 0; +}; + +KeystoreArray.prototype.destroy = function () { + this.index = null; + this.proxy = null; +}; + +KeystoreArray.prototype.push = function () {}; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreMap(bitlength = 8) { + + if (!(this instanceof KeystoreMap)) { + return new KeystoreMap(bitlength); + } + + this.index = create_object(); + this.refs = []; + this.size = 0; + + if (32 < bitlength) { + this.crc = lcg64; + this.bit = BigInt(bitlength); + } else { + this.crc = lcg; + this.bit = bitlength; + } +} + +KeystoreMap.prototype.get = function (key) { + const address = this.crc(key), + map = this.index[address]; + + return map && map.get(key); +}; + +KeystoreMap.prototype.set = function (key, value) { + const address = this.crc(key); + let map = this.index[address]; + if (map) { + let size = map.size; + map.set(key, value); + size -= map.size; + size && this.size++; + } else { + this.index[address] = map = new Map([[key, value]]); + this.refs.push(map); + } +}; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreSet(bitlength = 8) { + + if (!(this instanceof KeystoreSet)) { + return new KeystoreSet(bitlength); + } + + // using plain Object with numeric key access + // just for max performance + this.index = create_object(); + this.refs = []; + + if (32 < bitlength) { + this.crc = lcg64; + this.bit = BigInt(bitlength); + } else { + this.crc = lcg; + this.bit = bitlength; + } +} + +KeystoreSet.prototype.add = function (key) { + const address = this.crc(key); + let set = this.index[address]; + if (set) { + let size = set.size; + set.add(key); + size -= set.size; + size && this.size++; + } else { + this.index[address] = set = new Set([key]); + this.refs.push(set); + } +}; + +KeystoreMap.prototype.has = KeystoreSet.prototype.has = function (key) { + const address = this.crc(key), + map_or_set = this.index[address]; + + return map_or_set && map_or_set.has(key); +}; + +/* +KeystoreMap.prototype.size = +KeystoreSet.prototype.size = function(){ + let size = 0; + const values = Object.values(this.index); + for(let i = 0; i < values.length; i++){ + size += values[i].size; + } + return size; +}; +*/ + +KeystoreMap.prototype.delete = KeystoreSet.prototype.delete = function (key) { + const address = this.crc(key), + map_or_set = this.index[address]; + + // set && (set.size === 1 + // ? this.index.delete(address) + // : set.delete(key)); + map_or_set && map_or_set.delete(key) && this.size--; +}; + +KeystoreMap.prototype.clear = KeystoreSet.prototype.clear = function () { + this.index = create_object(); + this.refs = []; + this.size = 0; +}; + +// KeystoreMap.prototype.destroy = +// KeystoreSet.prototype.destroy = function(){ +// this.index = null; +// this.refs = null; +// this.proxy = null; +// }; + +KeystoreMap.prototype.values = KeystoreSet.prototype.values = function* () { + // alternatively iterate through this.keys[] + //const refs = Object.values(this.index); + for (let i = 0; i < this.refs.length; i++) { + for (let value of this.refs[i].values()) { + yield value; + } + } +}; + +KeystoreMap.prototype.keys = KeystoreSet.prototype.keys = function* () { + //const values = Object.values(this.index); + for (let i = 0; i < this.refs.length; i++) { + for (let key of this.refs[i].keys()) { + yield key; + } + } +}; + +KeystoreMap.prototype.entries = KeystoreSet.prototype.entries = function* () { + //const values = Object.values(this.index); + for (let i = 0; i < this.refs.length; i++) { + for (let entry of this.refs[i].entries()) { + yield entry; + } + } +}; + +/** + * Linear Congruential Generator (LCG) + * @param str + * @this {KeystoreMap|KeystoreSet} + */ + +function lcg(str) { + let range = 2 ** this.bit - 1; + if ("number" == typeof str) { + return str & range; + } + let crc = 0, + bit = this.bit + 1; + for (let i = 0; i < str.length; i++) { + crc = (crc * bit ^ str.charCodeAt(i)) & range; + } + // shift Int32 to UInt32 because negative numbers + // extremely slows down key lookup + return 32 === this.bit ? crc + 2147483648 : crc; // & 0xFFFF; +} + +/** + * @param str + * @this {KeystoreMap|KeystoreSet} + */ + +function lcg64(str) { + let range = BigInt(2) ** /** @type {!BigInt} */this.bit - BigInt(1), + type = typeof str; + if ("bigint" == type) { + return (/** @type {!BigInt} */str & range + ); + } + if ("number" == type) { + return BigInt(str) & range; + } + let crc = BigInt(0), + bit = /** @type {!BigInt} */this.bit + BigInt(1); + for (let i = 0; i < str.length; i++) { + crc = (crc * bit ^ BigInt(str.charCodeAt(i))) & range; + } + return crc; // & 0xFFFFFFFFFFFFFFFF; +} \ No newline at end of file diff --git a/dist/module/lang.js b/dist/module/lang.js deleted file mode 100644 index 954ae86..0000000 --- a/dist/module/lang.js +++ /dev/null @@ -1,321 +0,0 @@ -import { IndexInterface } from "./type.js"; -import { create_object, get_keys } from "./common.js"; - -/** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} _collapse - * @returns {string|Array} - * @this IndexInterface - */ - -export function pipeline(str, normalize, split, _collapse) { - - if (str) { - - if (normalize) { - - str = replace(str, /** @type {Array} */normalize); - } - - if (this.matcher) { - - str = replace(str, this.matcher); - } - - if (this.stemmer && 1 < str.length) { - - str = replace(str, this.stemmer); - } - - if (_collapse && 1 < str.length) { - - str = collapse(str); - } - - if (split || "" === split) { - - const words = str.split( /** @type {string|RegExp} */split); - - return this.filter ? filter(words, this.filter) : words; - } - } - - return str; -} - -// TODO improve normalize + remove non-delimited chars like in "I'm" + split on whitespace+ - -export const regex_whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; -// https://github.com/nextapps-de/flexsearch/pull/414 -//export const regex_whitespace = /[\s\xA0\u2000-\u200B\u2028\u2029\u3000\ufeff!"#$%&'()*+,\-./:;<=>?@[\\\]^_`{|}~]/ -const regex_normalize = /[\u0300-\u036f]/g; - -export function normalize(str) { - - if (str.normalize) { - - str = str.normalize("NFD").replace(regex_normalize, ""); - } - - return str; -} - -/** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} _collapse - * @returns {string|Array} - */ - -// FlexSearch.prototype.pipeline = function(str, normalize, split, _collapse){ -// -// if(str){ -// -// if(normalize && str){ -// -// str = replace(str, /** @type {Array} */ (normalize)); -// } -// -// if(str && this.matcher){ -// -// str = replace(str, this.matcher); -// } -// -// if(this.stemmer && str.length > 1){ -// -// str = replace(str, this.stemmer); -// } -// -// if(_collapse && str.length > 1){ -// -// str = collapse(str); -// } -// -// if(str){ -// -// if(split || (split === "")){ -// -// const words = str.split(/** @type {string|RegExp} */ (split)); -// -// return this.filter ? filter(words, this.filter) : words; -// } -// } -// } -// -// return str; -// }; - -// export function pipeline(str, normalize, matcher, stemmer, split, _filter, _collapse){ -// -// if(str){ -// -// if(normalize && str){ -// -// str = replace(str, normalize); -// } -// -// if(matcher && str){ -// -// str = replace(str, matcher); -// } -// -// if(stemmer && str.length > 1){ -// -// str = replace(str, stemmer); -// } -// -// if(_collapse && str.length > 1){ -// -// str = collapse(str); -// } -// -// if(str){ -// -// if(split !== false){ -// -// str = str.split(split); -// -// if(_filter){ -// -// str = filter(str, _filter); -// } -// } -// } -// } -// -// return str; -// } - - -/** - * @param {Array} words - * @returns {Object} - */ - -export function init_filter(words) { - - const filter = create_object(); - - for (let i = 0, length = words.length; i < length; i++) { - - filter[words[i]] = 1; - } - - return filter; -} - -/** - * @param {!Object} obj - * @param {boolean} is_stemmer - * @returns {Array} - */ - -export function init_stemmer_or_matcher(obj, is_stemmer) { - const keys = get_keys(obj), - length = keys.length, - final = []; - - - let removal = "", - count = 0; - - for (let i = 0, key, tmp; i < length; i++) { - - key = keys[i]; - tmp = obj[key]; - - if (tmp) { - - final[count++] = regex(is_stemmer ? "(?!\\b)" + key + "(\\b|_)" : key); - final[count++] = tmp; - } else { - - removal += (removal ? "|" : "") + key; - } - } - - if (removal) { - - final[count++] = regex(is_stemmer ? "(?!\\b)(" + removal + ")(\\b|_)" : "(" + removal + ")"); - final[count] = ""; - } - - return final; -} - -/** - * @param {!string} str - * @param {Array} regexp - * @returns {string} - */ - -export function replace(str, regexp) { - - for (let i = 0, len = regexp.length; i < len; i += 2) { - - str = str.replace(regexp[i], regexp[i + 1]); - - if (!str) { - - break; - } - } - - return str; -} - -/** - * @param {!string} str - * @returns {RegExp} - */ - -export function regex(str) { - - return new RegExp(str, "g"); -} - -/** - * Regex: replace(/(?:(\w)(?:\1)*)/g, "$1") - * @param {!string} string - * @returns {string} - */ - -export function collapse(string) { - - let final = "", - prev = ""; - - for (let i = 0, len = string.length, char; i < len; i++) { - - if ((char = string[i]) !== prev) { - - final += prev = char; - } - } - - return final; -} - -// TODO using fast-swap -export function filter(words, map) { - const length = words.length, - filtered = []; - - - for (let i = 0, count = 0; i < length; i++) { - - const word = words[i]; - - if (word && !map[word]) { - - filtered[count++] = word; - } - } - - return filtered; -} - -// const chars = {a:1, e:1, i:1, o:1, u:1, y:1}; -// -// function collapse_repeating_chars(string){ -// -// let collapsed_string = "", -// char_prev = "", -// char_next = ""; -// -// for(let i = 0; i < string.length; i++){ -// -// const char = string[i]; -// -// if(char !== char_prev){ -// -// if(i && (char === "h")){ -// -// if((chars[char_prev] && chars[char_next]) || (char_prev === " ")){ -// -// collapsed_string += char; -// } -// } -// else{ -// -// collapsed_string += char; -// } -// } -// -// char_next = ( -// -// (i === (string.length - 1)) ? -// -// "" -// : -// string[i + 1] -// ); -// -// char_prev = char; -// } -// -// return collapsed_string; -// } \ No newline at end of file diff --git a/dist/module/lang/arabic/default.js b/dist/module/lang/arabic/default.js deleted file mode 100644 index 89f65a6..0000000 --- a/dist/module/lang/arabic/default.js +++ /dev/null @@ -1,27 +0,0 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline } from "../../lang.js"; - -export const rtl = /* append: */ /* skip update: */ /* skip_update: */!0; -export const tokenize = ""; -export default { - encode: encode, - rtl: !0 -}; - -const regex = /[\x00-\x7F]+/g, - split = /\s+/; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).replace(regex, " "), - /* normalize: */ - /* collapse: */!1, - /* split: */split, !1); -} \ No newline at end of file diff --git a/dist/module/lang/at.js b/dist/module/lang/at.js deleted file mode 100644 index af404df..0000000 --- a/dist/module/lang/at.js +++ /dev/null @@ -1,41 +0,0 @@ -/** - * http://www.ranks.nl/stopwords - * @type {Array} - */ - -export const filter = ["aber", "als", "am", "an", "auch", "auf", "aus", "bei", "bin", "bis", "bist", "da", "dadurch", "daher", "darum", "das", "daß", "dass", "dein", "deine", "dem", "den", "der", "des", "dessen", "deshalb", "die", "dies", "dieser", "dieses", "doch", "dort", "du", "durch", "ein", "eine", "einem", "einen", "einer", "eines", "er", "es", "euer", "eure", "für", "hatte", "hatten", "hattest", "hattet", "hier", "hinter", "ich", "ihr", "ihre", "im", "in", "ist", "ja", "jede", "jedem", "jeden", "jeder", "jedes", "jener", "jenes", "jetzt", "kann", "kannst", "können", "könnt", "machen", "mein", "meine", "mit", "muß", "mußt", "musst", "müssen", "müßt", "nach", "nachdem", "nein", "nicht", "nun", "oder", "seid", "sein", "seine", "sich", "sie", "sind", "soll", "sollen", "sollst", "sollt", "sonst", "soweit", "sowie", "und", "unser", "unsere", "unter", "vom", "von", "vor", "wann", "warum", "was", "weiter", "weitere", "wenn", "wer", "werde", "werden", "werdet", "weshalb", "wie", "wieder", "wieso", "wir", "wird", "wirst", "wo", "woher", "wohin", "zu", "zum", "zur", "über"]; - -/** - * @type {Object} - */ - -export const stemmer = { - - niss: "", - isch: "", - lich: "", - heit: "", - keit: "", - end: "", - ung: "", - est: "", - ern: "", - em: "", - er: "", - en: "", - es: "", - st: "", - ig: "", - ik: "", - e: "", - s: "" -}; - -export const matcher = {}; - -export default { - - filter: filter, - stemmer: stemmer, - matcher: matcher -}; \ No newline at end of file diff --git a/dist/module/lang/cjk/default.js b/dist/module/lang/cjk/default.js deleted file mode 100644 index a0e94f2..0000000 --- a/dist/module/lang/cjk/default.js +++ /dev/null @@ -1,26 +0,0 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline } from "../../lang.js"; - -export const rtl = /* normalize: */ /* collapse: */ -/* normalize: */ -/* collapse: */!1; -export const tokenize = "strict"; -export default { - encode: encode, - rtl: !1, - tokenize: "strict" -}; - -const regex = /[\x00-\x7F]+/g; - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).replace(regex, ""), !1, - /* split: */"", !1); -} \ No newline at end of file diff --git a/dist/module/lang/cyrillic/default.js b/dist/module/lang/cyrillic/default.js deleted file mode 100644 index de45263..0000000 --- a/dist/module/lang/cyrillic/default.js +++ /dev/null @@ -1,27 +0,0 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline } from "../../lang.js"; - -export const rtl = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ -/* normalize: */ -/* collapse: */!1; -export const tokenize = ""; -export default { - encode: encode, - rtl: !1 -}; - -const regex = /[\x00-\x7F]+/g, - split = /\s+/; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).replace(regex, " "), !1, - /* split: */split, !1); -} \ No newline at end of file diff --git a/dist/module/lang/de.js b/dist/module/lang/de.js index 8196777..717c431 100644 --- a/dist/module/lang/de.js +++ b/dist/module/lang/de.js @@ -2,53 +2,38 @@ * Filter are also known as "stopwords", they completely filter out words from being indexed. * Source: http://www.ranks.nl/stopwords * Object Definition: Just provide an array of words. - * @type {Array} + * @type {Set} */ -export const filter = ["aber", "als", "am", "an", "auch", "auf", "aus", "bei", "bin", "bis", "bist", "da", "dadurch", "daher", "darum", "das", "daß", "dass", "dein", "deine", "dem", "den", "der", "des", "dessen", "deshalb", "die", "dies", "dieser", "dieses", "doch", "dort", "du", "durch", "ein", "eine", "einem", "einen", "einer", "eines", "er", "es", "euer", "eure", "für", "hatte", "hatten", "hattest", "hattet", "hier", "hinter", "ich", "ihr", "ihre", "im", "in", "ist", "ja", "jede", "jedem", "jeden", "jeder", "jedes", "jener", "jenes", "jetzt", "kann", "kannst", "können", "könnt", "machen", "mein", "meine", "mit", "muß", "mußt", "musst", "müssen", "müßt", "nach", "nachdem", "nein", "nicht", "nun", "oder", "seid", "sein", "seine", "sich", "sie", "sind", "soll", "sollen", "sollst", "sollt", "sonst", "soweit", "sowie", "und", "unser", "unsere", "unter", "vom", "von", "vor", "wann", "warum", "was", "weiter", "weitere", "wenn", "wer", "werde", "werden", "werdet", "weshalb", "wie", "wieder", "wieso", "wir", "wird", "wirst", "wo", "woher", "wohin", "zu", "zum", "zur", "über"]; +export const filter = new Set(["aber", "als", "am", "an", "auch", "auf", "aus", "bei", "bin", "bis", "bist", "da", "dadurch", "daher", "darum", "das", "dass", "dass", "dein", "deine", "dem", "den", "der", "des", "dessen", "deshalb", "die", "dies", "dieser", "dieses", "doch", "dort", "du", "durch", "ein", "eine", "einem", "einen", "einer", "eines", "er", "es", "euer", "eure", "fuer", "hatte", "hatten", "hattest", "hattet", "hier", "hinter", "ich", "ihr", "ihre", "im", "in", "ist", "ja", "jede", "jedem", "jeden", "jeder", "jedes", "jener", "jenes", "jetzt", "ggf", "kann", "kannst", "koennen", "koennt", "machen", "mein", "meine", "mit", "muss", "musst", "musst", "muessen", "muesst", "nach", "nachdem", "nein", "nicht", "noch", "nun", "oder", "seid", "sein", "seine", "sich", "sie", "sind", "soll", "sollen", "sollst", "sollt", "sonst", "soweit", "sowie", "und", "unser", "unsere", "unter", "usw", "uvm", "vom", "von", "vor", "wann", "warum", "was", "weiter", "weitere", "wenn", "wer", "werde", "werden", "werdet", "weshalb", "wie", "wieder", "wieso", "wir", "wird", "wirst", "wo", "woher", "wohin", "zu", "zum", "zur", "ueber"]); /** * Stemmer removes word endings and is a kind of "partial normalization". A word ending just matched when the word length is bigger than the matched partial. * Example: The word "correct" and "correctness" could be the same word, so you can define {"ness": ""} to normalize the ending. * Object Definition: the key represents the word ending, the value contains the replacement (or empty string for removal). - * @type {Object} + * http://snowball.tartarus.org/algorithms/german/stemmer.html + * @type {Map} */ -export const stemmer = { - - niss: "", - isch: "", - lich: "", - heit: "", - keit: "", - ell: "", - bar: "", - end: "", - ung: "", - est: "", - ern: "", - em: "", - er: "", - en: "", - es: "", - st: "", - ig: "", - ik: "", - e: "", - s: "" -}; +export const stemmer = new Map([["niss", ""], ["isch", ""], ["lich", ""], ["heit", ""], ["keit", ""], ["ell", ""], ["bar", ""], ["end", ""], ["ung", ""], ["est", ""], ["ern", ""], ["em", ""], ["er", ""], ["en", ""], ["es", ""], ["st", ""], ["ig", ""], ["ik", ""], ["e", ""], ["s", ""]]); /** * Matcher replaces all occurrences of a given string regardless of its position and is also a kind of "partial normalization". * Object Definition: the key represents the target term, the value contains the search string which should be replaced (could also be an array of multiple terms). - * @type {Object|string>} + * @type {Map} */ - -export const matcher = {}; +const map = new Map([["_", " "], ["ä", "ae"], ["ö", "oe"], ["ü", "ue"], ["ß", "ss"], ["&", " und "], ["€", " EUR "]]); export default { - + normalize: function (str) { + return str.toLowerCase(); + }, + prepare: function (str) { + // normalization + if (/[_äöüß&€]/.test(str)) str = str.replace(/[_äöüß&€]/g, match => map.get(match)); + // street names + return str.replace(/str\b/g, "strasse").replace(/(?!\b)strasse\b/g, " strasse").replace(/\bst\b/g, "sankt"); + }, filter: filter, - stemmer: stemmer, - matcher: matcher + stemmer: stemmer }; \ No newline at end of file diff --git a/dist/module/lang/en.js b/dist/module/lang/en.js index 7b1ba3a..65b6220 100644 --- a/dist/module/lang/en.js +++ b/dist/module/lang/en.js @@ -1,100 +1,128 @@ /** * http://www.ranks.nl/stopwords - * @type {Array} + * @type {Set} */ -export const filter = ["a", "about", "above", "after", "again", "against", "all", "also", "am", "an", "and", "any", "are", "aren't", "as", "at", -//"back", -"be", "because", "been", "before", "being", "below", -//"between", -"both", "but", "by", "can", "cannot", "can't", "come", "could", "couldn't", +// todo filter out minlength +export const filter = new Set(["a", "about", "above", "after", "again", "against", "all", "also", "am", "an", "and", "any", "are", "arent", "as", "at", "back", "be", "because", "been", "before", "being", "below", "between", "both", "but", "by", "can", "cannot", "cant", "come", "could", "couldnt", //"day", -"did", "didn't", "do", "does", "doesn't", "doing", "dont", "down", "during", "each", "even", "few", "first", "for", "from", "further", "get", +"did", "didnt", "do", "does", "doesnt", "doing", "dont", "down", "during", "each", "even", "few", +//"first", +"for", "from", "further", "get", //"give", -"go", -//"good", -"had", "hadn't", "has", "hasn't", "have", "haven't", "having", "he", "hed", +"go", "good", "had", "hadnt", "has", "hasnt", "have", "havent", "having", "he", "hed", //"hell", -"her", "here", "here's", "hers", "herself", "hes", "him", "himself", "his", "how", "how's", "i", "id", "if", "ill", "im", "in", "into", "is", "isn't", "it", "it's", "itself", "i've", "just", "know", "let's", "like", +"her", "here", "heres", "hers", "herself", "hes", "him", "himself", "his", "how", "hows", "i", "id", "if", "ill", "im", "in", "into", "is", "isnt", "it", "its", "itself", "ive", "just", "know", "lets", "like", //"look", -"make", "me", "more", "most", "mustn't", "my", "myself", "new", "no", "nor", "not", "now", "of", "off", "on", "once", -//"one", -"only", "or", "other", "ought", "our", "our's", "ourselves", "out", "over", "own", -//"people", -"same", "say", "see", "shan't", "she", "she'd", "shell", "shes", "should", "shouldn't", "so", "some", "such", -//"take", -"than", "that", "that's", "the", "their", "theirs", "them", "themselves", "then", "there", "there's", "these", "they", "they'd", "they'll", "they're", "they've", -//"think", -"this", "those", "through", "time", "to", "too", +"lot", "make", "made", "me", "more", "most", "mustnt", "my", "myself", "new", "no", "nor", "not", "now", "of", "off", "on", "once", "one", "only", "or", "other", "ought", "our", "ours", "ourselves", "out", "over", "own", "people", "same", "say", "see", "shant", "she", "shed", "shell", "shes", "should", "shouldnt", "so", "some", "such", "take", "than", "that", "thats", "the", "their", "theirs", "them", "themselves", "then", "there", "theres", "these", "they", "theyd", "theyll", "theyre", "theyve", "think", "this", "those", "through", "time", "times", "to", "too", //"two", -//"under", -"until", "up", "us", -//"use", -"very", "want", "was", "wasn't", "way", "we", "wed", "well", "were", "weren't", "we've", "what", "what's", "when", "when's", "where", "where's", "which", "while", "who", "whom", "who's", "why", "why's", "will", "with", "won't", -//"work", -"would", "wouldn't", +"under", "until", "up", "us", "use", "very", "want", "was", "wasnt", "way", "we", "wed", "well", "were", "werent", "weve", "what", "whats", "when", "whens", "where", "wheres", "which", "while", "who", "whom", "whos", "why", "whys", "will", "with", "wont", "work", "would", "wouldnt", //"year", -"you", "you'd", "you'll", "your", "you're", "your's", "yourself", "yourselves", "you've"]; +"ya", "you", "youd", "youll", "your", "youre", "yours", "yourself", "yourselves", "youve"]); /** * @type {Object} */ -export const stemmer = { +export const stemmer = new Map([["ational", ""], ["iveness", ""], ["fulness", ""], ["ousness", ""], ["ization", ""], ["tional", ""], ["biliti", ""], ["icate", ""], ["ative", ""], ["alize", ""], ["iciti", ""], ["entli", ""], ["ousli", ""], ["alism", ""], ["ation", ""], ["aliti", ""], ["iviti", ""], ["ement", ""], ["izer", ""], ["able", ""], ["alli", ""], ["ator", ""], ["logi", ""], ["ical", ""], ["ance", ""], ["ence", ""], ["ness", ""], ["ble", ""], ["ment", ""], ["eli", ""], ["bli", ""], ["ful", ""], ["ant", ""], ["ent", ""], ["ism", ""], ["ate", ""], ["iti", ""], ["ous", ""], ["ive", ""], ["ize", ""], ["ing", ""], ["ion", ""], ["al", ""], ["ou", ""], ["er", ""], ["ic", ""], ["ly", ""]]); - ational: "ate", - iveness: "ive", - fulness: "ful", - ousness: "ous", - ization: "ize", - tional: "tion", - biliti: "ble", - icate: "ic", - ative: "", - alize: "al", - iciti: "ic", - entli: "ent", - ousli: "ous", - alism: "al", - ation: "ate", - aliti: "al", - iviti: "ive", - ement: "", - enci: "ence", - anci: "ance", - izer: "ize", - alli: "al", - ator: "ate", - logi: "log", - ical: "ic", - ance: "", - ence: "", - ness: "", - able: "", - ible: "", - ment: "", - eli: "e", - bli: "ble", - ful: "", - ant: "", - ent: "", - ism: "", - ate: "", - iti: "", - ous: "", - ive: "", - ize: "", - al: "", - ou: "", - er: "", - ic: "" -}; +// export const replacer = new Map([ +// ["&", " and "], +// ]); -export const matcher = {}; +/* + he’s (= he is / he has) + she’s (= she is / she has) + I’ll (= I will) + I’ve (= I have) + I’d (= I would / I had) + don’t (= do not) + doesn’t (= does not) + didn’t (= did not) + isn’t (= is not) + hasn’t (= has not) + can’t (= cannot) + won’t (= will not) +*/ + +// const explode = new Map([ +// ["^i'm$", "i am"], +// ["^can't$", "can not"], +// ["^cannot$", "can not"], +// ["^won't$", "will not"], +// ["'s$", " is has"], +// ["n't$", " not"], +// ["'ll$", " will"], +// ["'re$", " are"], +// ["'ve$", " have"], +// ["'d$", " would had"], +// ]); + +// const pairs = [ +// /´`’ʼ/, /´`’ʼ/g, "'", +// /_/, /_+/g, " ", +// /&/, /&/g, " and ", +// /\bi'm\b/, /\bi'm\b/g, "i am", +// /\b(can't|cannot)\b/, /\b(can't|cannot)\b/g, "can not", +// /\bwon't\b/, /\bwon't\b/g, "will not", +// /[a-z]n't\b/, /[a-z]n't\b/g, "$1 not", +// /[a-z]'s\b/, /([a-z])'s\b/g, "$1 is has", +// /[a-z]'ll\b/, /[a-z]'ll\b/g, "$1 will", +// /[a-z]'re\b/, /[a-z]'re\b/g, "$1 are", +// /[a-z]'ve\b/, /[a-z]'ve\b/g, "$1 have", +// /[a-z]'d\b/, /[a-z]'d\b/g, "$1 is has" +// ]; + +// const map = new Map([ +// ["´", "'"], +// ["`", "'"], +// ["’", "'"], +// ["ʼ", "'"], +// ["_", " "], +// ["&", " and "] +// ]); export default { + prepare: function (str) { + // if(/[´`’ʼ_&]/.test(str)) + // str = str.replace(/[´`’ʼ_&]/g, match => map.get(match)); + // if(/´`’ʼ/.test(str)) + // str = str.replace(/´`’ʼ/g, "'"); + // if(/_/.test(str)) + // str = str.replace(/_+/g, " "); + // if(/&/.test(str)) + // str = str.replace(/&/g, " and "); + + // if(/\bi'm\b/.test(str)) + // str = str.replace(/\bi'm\b/g, "i am"); + // if(/\b(can't|cannot)\b/.test(str)) + // str = str.replace(/\b(can't|cannot)\b/g, "can not"); + // if(/\bwon't\b/.test(str)) + // str = str.replace(/\bwon't\b/g, "will not"); + // if(/[a-z]n't\b/.test(str)) + // str = str.replace(/([a-z])n't\b/g, "$1 not"); + // if(/[a-z]'s\b/.test(str)) + // str = str.replace(/([a-z])'s\b/g, "$1 is has"); + // if(/[a-z]'ll\b/.test(str)) + // str = str.replace(/([a-z])'ll\b/g, "$1 will"); + // if(/[a-z]'re\b/.test(str)) + // str = str.replace(/([a-z])'re\b/g, "$1 are"); + // if(/[a-z]'ve\b/.test(str)) + // str = str.replace(/([a-z])'ve\b/g, "$1 have"); + // if(/[a-z]'d\b/.test(str)) + // str = str.replace(/([a-z])'d\b/g, "$1 would had"); + // return str; + + return str //.replace(/[´`’ʼ_&]/g, match => map.get(match)) + // normalization + .replace(/´`’ʼ/g, "'").replace(/_+/g, " ").replace(/&/g, " and ") + //.replace(/([0-9 ]|^)\$([0-9 ]|$)/g, "$1 USD $2") + //.replace(/([0-9 ]|^)£([0-9 ]|$)/g, "$1 GBP $2") + .replace(/\$/g, " USD ").replace(/£/g, " GBP ") + // explode short forms + .replace(/([a-z])'s\b/g, "$1 is").replace(/\bi'm\b/g, "i am").replace(/\b(can't|cannot)\b/g, "can not").replace(/\bwon't\b/g, "will not").replace(/([a-z])n't\b/g, "$1 not").replace(/([a-z])'ll\b/g, "$1 will").replace(/([a-z])'re\b/g, "$1 are").replace(/([a-z])'ve\b/g, "$1 have").replace(/([a-z])'d\b/g, "$1 would"); + }, filter: filter, - stemmer: stemmer, - matcher: matcher + stemmer: stemmer }; \ No newline at end of file diff --git a/dist/module/lang/fr.js b/dist/module/lang/fr.js new file mode 100644 index 0000000..573652f --- /dev/null +++ b/dist/module/lang/fr.js @@ -0,0 +1,21 @@ +/** + * http://www.ranks.nl/stopwords + * http://snowball.tartarus.org/algorithms/french/stop.txt + * @type {Set} + */ + +export const filter = new Set(["au", "aux", "avec", "ce", "ces", "dans", "de", "des", "du", "elle", "en", "et", "eux", "il", "je", "la", "le", "leur", "lui", "ma", "mais", "me", "meme", "mes", "moi", "mon", "ne", "nos", "notre", "nous", "on", "ou", "par", "pas", "pour", "qu", "que", "qui", "sa", "se", "ses", "son", "sur", "ta", "te", "tes", "toi", "ton", "tu", "un", "une", "vos", "votre", "vous", "c", "d", "j", "l", "m", "n", "s", "t", "a", "y", "ete", "etee", "etees", "etes", "etant", "suis", "es", "est", "sommes", "etes", "sont", "serai", "seras", "sera", "serons", "serez", "seront", "serais", "serait", "serions", "seriez", "seraient", "etais", "etait", "etions", "etiez", "etaient", "fus", "fut", "fumes", "futes", "furent", "sois", "soit", "soyons", "soyez", "soient", "fusse", "fusses", "fut", "fussions", "fussiez", "fussent", "ayant", "eu", "eue", "eues", "eus", "ai", "as", "avons", "avez", "ont", "aurai", "auras", "aura", "aurons", "aurez", "auront", "aurais", "aurait", "aurions", "auriez", "auraient", "avais", "avait", "avions", "aviez", "avaient", "eut", "eumes", "eutes", "eurent", "aie", "aies", "ait", "ayons", "ayez", "aient", "eusse", "eusses", "eut", "eussions", "eussiez", "eussent", "ceci", "cela", "cela", "cet", "cette", "ici", "ils", "les", "leurs", "quel", "quels", "quelle", "quelles", "sans", "soi"]); + +/** + * @type {Object} + */ + +export const stemmer = new Map([["lement", ""], ["ient", ""], ["nera", ""], ["ment", ""], ["ais", ""], ["ait", ""], ["ant", ""], ["ent", ""], ["iez", ""], ["ion", ""], ["nez", ""], ["ai", ""], ["es", ""], ["er", ""], ["ez", ""], ["le", ""], ["na", ""], ["ne", ""], ["a", ""], ["e", ""]]); + +export default { + prepare: function (str) { + return str.replace(/´`’ʼ/g, "'").replace(/_+/g, " ").replace(/&/g, " et ").replace(/€/g, " EUR ").replace(/\bl'([^\b])/g, "la le $1").replace(/\bt'([^\b])/g, "ta te $1").replace(/\bc'([^\b])/g, "ca ce $1").replace(/\bd'([^\b])/g, "da de $1").replace(/\bj'([^\b])/g, "ja je $1").replace(/\bn'([^\b])/g, "na ne $1").replace(/\bm'([^\b])/g, "ma me $1").replace(/\bs'([^\b])/g, "sa se $1").replace(/\bau\b/g, "a le").replace(/\baux\b/g, "a les").replace(/\bdu\b/g, "de le").replace(/\bdes\b/g, "de les"); + }, + filter: filter, + stemmer: stemmer +}; \ No newline at end of file diff --git a/dist/module/lang/latin/advanced.js b/dist/module/lang/latin/advanced.js index 94f5cb1..fa5ea16 100644 --- a/dist/module/lang/latin/advanced.js +++ b/dist/module/lang/latin/advanced.js @@ -1,89 +1,144 @@ -import { IndexInterface } from "../../type.js"; -import { regex, replace, collapse } from "../../lang.js"; -import { encode as encode_balance } from "./balance.js"; +import Encoder from "../../encoder.js"; +import { soundex } from "./balance.js"; + +// const soundex = new Map([ +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// +// //["e", "e"], +// ["i", "e"], +// ["y", "e"], +// +// //["o", "o"], +// ["u", "o"] +// ]); + +export const matcher = new Map([["ai", "ei"], ["ae", "a"], ["oe", "o"], ["ue", "u"], ["sh", "s"], ["ch", "c"], ["th", "t"], ["ph", "f"], ["pf", "f"]]); + +export const replacer = [/([^aeo])h([aeo$])/g, "$1$2", /([aeo])h([^aeo]|$)/g, "$1$2"]; -export const rtl = /* normalize: */ -/* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0, + mapper: soundex, + replacer: replacer, + matcher: matcher +}; - // Phonetic Normalization - -};const regex_ae = regex("ae"), - -//regex_ai = regex("ai"), -//regex_ay = regex("ay"), -//regex_ey = regex("ey"), -regex_oe = regex("oe"), - //regex_ue = regex("ue"), -//regex_ie = regex("ie"), -//regex_sz = regex("sz"), -//regex_zs = regex("zs"), -//regex_ck = regex("ck"), -//regex_cc = regex("cc"), -regex_sh = regex("sh"), - regex_th = regex("th"), - -//regex_dt = regex("dt"), -regex_ph = regex("ph"), - regex_pf = regex("pf"), - pairs = [regex_ae, "a", -// regex_ai, "ei", -// regex_ay, "ei", -// regex_ey, "ei", -regex_oe, "o", -// regex_ue, "u", -// regex_ie, "i", -// regex_sz, "s", -// regex_zs, "s", -regex_sh, "s", -// regex_ck, "k", -// regex_cc, "k", -regex_th, "t", -// regex_dt, "t", -regex_ph, "f", regex_pf, "f", -// regex_ou, "o", -// regex_uo, "u" - -// regex("(?![aeiouy])h(?![aeiouy])"), "", -// regex("(?!^[aeiouy])h(?!^[aeiouy])"), "" -regex("(?![aeo])h(?![aeo])"), "", regex("(?!^[aeo])h(?!^[aeo])"), ""]; -//regex_ou = regex("ou"), -//regex_uo = regex("uo"); - -/** - * @param {string|number} str - * @param {boolean=} _skip_postprocessing - * @this IndexInterface - */ - -export function encode(str, _skip_postprocessing) { - - if (str) { - - str = encode_balance.call(this, str).join(" "); - - if (2 < str.length) { - - str = replace(str, pairs); - } - - if (!_skip_postprocessing) { - - if (1 < str.length) { - - str = collapse(str); - } - - if (str) { - - str = str.split(" "); - } - } - } - - return str || []; -} \ No newline at end of file +// import { regex, replace, collapse } from "../../lang.js"; +// import { encode as encode_balance } from "./balance.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // Phonetic Normalization +// +// const regex_ae = regex("ae"), +// //regex_ai = regex("ai"), +// //regex_ay = regex("ay"), +// //regex_ey = regex("ey"), +// regex_oe = regex("oe"), +// //regex_ue = regex("ue"), +// //regex_ie = regex("ie"), +// //regex_sz = regex("sz"), +// //regex_zs = regex("zs"), +// //regex_ck = regex("ck"), +// //regex_cc = regex("cc"), +// regex_sh = regex("sh"), +// regex_th = regex("th"), +// //regex_dt = regex("dt"), +// regex_ph = regex("ph"), +// regex_pf = regex("pf"); +// //regex_ou = regex("ou"), +// //regex_uo = regex("uo"); +// +// const pairs = [ +// regex_ae, "a", +// // regex_ai, "ei", +// // regex_ay, "ei", +// // regex_ey, "ei", +// regex_oe, "o", +// // regex_ue, "u", +// // regex_ie, "i", +// // regex_sz, "s", +// // regex_zs, "s", +// regex_sh, "s", +// // regex_ck, "k", +// // regex_cc, "k", +// regex_th, "t", +// // regex_dt, "t", +// regex_ph, "f", +// regex_pf, "f", +// // regex_ou, "o", +// // regex_uo, "u" +// +// // regex("(?![aeiouy])h(?![aeiouy])"), "", +// // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "" +// regex("(?![aeo])h(?![aeo])"), "", +// regex("(?!^[aeo])h(?!^[aeo])"), "" +// ]; +// +// /** +// * @param {string|number} str +// * @param {boolean=} _skip_postprocessing +// */ +// +// export function encode(str, _skip_postprocessing){ +// +// if(str){ +// +// str = encode_balance.call(this, str).join(" "); +// +// if(str.length > 2){ +// +// str = replace(str, pairs); +// } +// +// if(!_skip_postprocessing){ +// +// if(str.length > 1){ +// +// str = collapse(str); +// } +// +// if(str){ +// +// str = str.split(" "); +// } +// } +// } +// +// return str || []; +// } \ No newline at end of file diff --git a/dist/module/lang/latin/balance.js b/dist/module/lang/latin/balance.js index 10de393..bd8fd20 100644 --- a/dist/module/lang/latin/balance.js +++ b/dist/module/lang/latin/balance.js @@ -1,119 +1,279 @@ -import { IndexInterface } from "../../type.js"; -import { encode as encode_simple } from "./simple.js"; +import Encoder from "../../encoder.js"; -// custom soundex implementation +export const soundex = new Map([["b", "p"], +//["p", "p"], + +//["f", "f"], +["v", "f"], ["w", "f"], + +//["s", "s"], +["z", "s"], ["x", "s"], ["d", "t"], +//["t", "t"], + +//["m", "m"], +["n", "m"], + +//["k", "k"], +["c", "k"], ["g", "k"], ["j", "k"], ["q", "k"], + +//["r", "r"], +//["h", "h"], +//["l", "l"], + +//["a", "a"], + +//["e", "e"], +["i", "e"], ["y", "e"], + +//["o", "o"], +["u", "o"]]); -export const rtl = /* normalize: */ /* collapse: */ -/* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */!1; -export const tokenize = "strict"; export default { - encode: encode, - rtl: !1, - tokenize: "strict" - - //const regex_whitespace = /[\W_]+/g; -};const regex_strip = /[^a-z0-9]+/, - soundex = { - - b: "p", - //"p": "p", - - //"f": "f", - v: "f", w: "f", - - //"s": "s", - z: "s", - x: "s", - ß: "s", - - d: "t", - //"t": "t", - - //"l": "l", - - //"m": "m", - n: "m", - - c: "k", - g: "k", - j: "k", - //"k": "k", - q: "k", - - //"r": "r", - //"h": "h", - //"a": "a", - - //"e": "e", - i: "e", - y: "e", - - //"o": "o", - u: "o" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0, + mapper: soundex }; -// const pairs = [ -// regex_whitespace, " ", -// regex_strip, "" -// ]; - -// modified - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - str = encode_simple.call(this, str).join(" "); - - // str = this.pipeline( - // - // /* string: */ normalize("" + str).toLowerCase(), - // /* normalize: */ false, - // /* split: */ false, - // /* collapse: */ false - // ); - - const result = []; - - if (str) { - const words = str.split(regex_strip), - length = words.length; - - - for (let x = 0, tmp, count = 0; x < length; x++) { - - if ((str = words[x]) && ( /*&& (str.length > 2)*/!this.filter || !this.filter[str])) { - - tmp = str[0]; - let code = soundex[tmp] || tmp, - previous = code; //str[0]; - - //soundex[code] || code; - - for (let i = 1; i < str.length; i++) { - - tmp = str[i]; - const current = soundex[tmp] || tmp; - - if (current && current !== previous) { - - code += current; - previous = current; - - // if(code.length === 7){ - // - // break; - // } - } - } - - result[count++] = code; //(code + "0000").substring(0, 4); - } - } - } - - return result; -} \ No newline at end of file +// //import { encode as encode_simple } from "./simple.js"; +// import { pipeline } from "../../lang.js"; +// +// // custom soundex implementation +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// //const regex_whitespace = /[\W_]+/g; +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// const normalize = "".normalize && /[\u0300-\u036f]/g; +// //const regex_strip = /[^a-z0-9]+/; +// +// // const pairs = [ +// // regex_whitespace, " ", +// // regex_strip, "" +// // ]; +// +// // modified +// const pairs = new Map([ +// +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["ß", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// ["ñ", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["ç", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// ["à", "a"], +// ["á", "a"], +// ["â", "a"], +// ["ã", "a"], +// ["ä", "a"], +// ["å", "a"], +// +// //["e", "e"], +// ["è", "e"], +// ["é", "e"], +// ["ê", "e"], +// ["ë", "e"], +// ["i", "e"], +// ["ì", "e"], +// ["í", "e"], +// ["î", "e"], +// ["ï", "e"], +// ["y", "e"], +// ["ý", "e"], +// ["ŷ", "e"], +// ["ÿ", "e"], +// +// //["o", "o"], +// ["ò", "o"], +// ["ó", "o"], +// ["ô", "o"], +// ["õ", "o"], +// ["ö", "o"], +// ["ő", "o"], +// ["u", "o"], +// ["ù", "o"], +// ["ú", "o"], +// ["û", "o"], +// ["ü", "o"], +// ["ű", "o"], +// ]); +// +// const map_soundex = new Map([ +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// +// //["e", "e"], +// ["i", "e"], +// ["y", "e"], +// +// //["o", "o"], +// ["u", "o"] +// ]); +// // const soundex = { +// // +// // "b": "p", +// // //"p": "p", +// // +// // //"f": "f", +// // "v": "f", +// // "w": "f", +// // +// // //"s": "s", +// // "z": "s", +// // "x": "s", +// // "ß": "s", +// // +// // "d": "t", +// // //"t": "t", +// // +// // //"l": "l", +// // +// // //"m": "m", +// // "n": "m", +// // +// // "c": "k", +// // "g": "k", +// // "j": "k", +// // //"k": "k", +// // "q": "k", +// // +// // //"r": "r", +// // //"h": "h", +// // //"a": "a", +// // +// // //"e": "e", +// // "i": "e", +// // "y": "e", +// // +// // //"o": "o", +// // "u": "o" +// // }; +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// return pipeline.call( +// this, +// /* string: */ ("" + str).normalize("NFD").replace(normalize, "").toLowerCase(), +// /* normalize: */ map_soundex, +// /* split: */ whitespace, +// ///* collapse: */ false +// ); +// +// // return pipeline.call( +// // this, +// // /* string: */ ("" + str).toLowerCase(), +// // /* normalize: */ /*pairs*/ new Map(), +// // /* split: */ whitespace, +// // ///* collapse: */ false +// // ); +// +// // str = encode_simple.call(this, str).join(" "); +// // +// // // str = this.pipeline( +// // // +// // // /* string: */ normalize("" + str).toLowerCase(), +// // // /* normalize: */ false, +// // // /* split: */ false, +// // // /* collapse: */ false +// // // ); +// // +// // const result = []; +// // +// // if(str){ +// // +// // const words = str.split(regex_strip); +// // const length = words.length; +// // +// // for(let x = 0, tmp, count = 0; x < length; x++){ +// // +// // if((str = words[x]) /*&& (str.length > 2)*/ && (!this.filter || !this.filter.has(str))){ +// // +// // tmp = str[0]; +// // let code = soundex[tmp] || tmp; //str[0]; +// // let previous = code; //soundex[code] || code; +// // +// // for(let i = 1; i < str.length; i++){ +// // +// // tmp = str[i]; +// // const current = soundex[tmp] || tmp; +// // +// // if(current && (current !== previous)){ +// // +// // code += current; +// // previous = current; +// // +// // // if(code.length === 7){ +// // // +// // // break; +// // // } +// // } +// // } +// // +// // result[count++] = code; //(code + "0000").substring(0, 4); +// // } +// // } +// // } +// // +// // return result; +// } \ No newline at end of file diff --git a/dist/module/lang/latin/default.js b/dist/module/lang/latin/default.js index dac5a9c..a7e126e 100644 --- a/dist/module/lang/latin/default.js +++ b/dist/module/lang/latin/default.js @@ -1,23 +1,36 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline, normalize, regex_whitespace } from "../../lang.js"; +import Encoder from "../../encoder.js"; -export const rtl = /* normalize: */ -/* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ -/* normalize: */ -/* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: function (str) { + return str.toLowerCase(); + }, + dedupe: /* suggest */ /* append: */ /* enrich */!1 +}; - /** - * @param {string|number} str - * @this IndexInterface - */ - -};export function encode(str) { - - return pipeline.call(this, - /* string: */("" + str).toLowerCase(), !1, /* split: */regex_whitespace, !1); -} \ No newline at end of file +// import { pipeline } from "../../lang.js"; +// +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// return pipeline.call( +// +// this, +// /* string: */ ("" + str).toLowerCase(), +// /* normalize: */ false, +// /* split: */ whitespace, +// /* collapse: */ false +// ); +// } \ No newline at end of file diff --git a/dist/module/lang/latin/exact.js b/dist/module/lang/latin/exact.js new file mode 100644 index 0000000..1ef1dae --- /dev/null +++ b/dist/module/lang/latin/exact.js @@ -0,0 +1,4 @@ +export default { + normalize: /* suggest */ /* append: */ /* enrich */!1, + dedupe: !1 +}; \ No newline at end of file diff --git a/dist/module/lang/latin/extra.js b/dist/module/lang/latin/extra.js index c55ea6a..ec53420 100644 --- a/dist/module/lang/latin/extra.js +++ b/dist/module/lang/latin/extra.js @@ -1,65 +1,82 @@ -import { IndexInterface } from "../../type.js"; -import { regex, replace, collapse } from "../../lang.js"; -import { encode as encode_advanced } from "./advanced.js"; +import Encoder from "../../encoder.js"; +import { soundex } from "./balance.js"; +import { matcher, replacer } from "./advanced.js"; + +export const compact = [/(?!^)[aeoy]/g, "" // old: aioy +]; -export const rtl = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0, + mapper: soundex, + replacer: replacer.concat(compact), + matcher: matcher +}; - // Soundex Normalization - -};const prefix = "(?!\\b)", - //soundex_b = regex(prefix + "p"), -// soundex_s = regex(prefix + "z"), -// soundex_k = regex(prefix + "[cgq]"), -// soundex_m = regex(prefix + "n"), -// soundex_t = regex(prefix + "d"), -// soundex_f = regex(prefix + "[vw]"), -//regex_vowel = regex(prefix + "[aeiouy]"); -regex_vowel = regex("(?!\\b)[aeo]"), - pairs = [ - -// soundex_b, "b", -// soundex_s, "s", -// soundex_k, "k", -// soundex_m, "m", -// soundex_t, "t", -// soundex_f, "f", -// regex("(?![aeiouy])h(?![aeiouy])"), "", -// regex("(?!^[aeiouy])h(?!^[aeiouy])"), "", -regex_vowel, ""]; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - if (str) { - - str = encode_advanced.call(this, str, /* append: */ /* skip update: */ /* skip_update: */ /* skip post-processing: */!0); - - if (1 < str.length) { - - //str = replace(str, pairs); - str = str.replace(regex_vowel, ""); - } - - if (1 < str.length) { - - str = collapse(str); - } - - if (str) { - - str = str.split(" "); - } - } - - return str || []; -} \ No newline at end of file +// import { regex, replace, collapse } from "../../lang.js"; +// import { encode as encode_advanced } from "./advanced.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // Soundex Normalization +// +// const prefix = "(?!\\b)"; +// const //soundex_b = regex(prefix + "p"), +// // soundex_s = regex(prefix + "z"), +// // soundex_k = regex(prefix + "[cgq]"), +// // soundex_m = regex(prefix + "n"), +// // soundex_t = regex(prefix + "d"), +// // soundex_f = regex(prefix + "[vw]"), +// //regex_vowel = regex(prefix + "[aeiouy]"); +// regex_vowel = regex(prefix + "[aeo]"); +// +// const pairs = [ +// +// // soundex_b, "b", +// // soundex_s, "s", +// // soundex_k, "k", +// // soundex_m, "m", +// // soundex_t, "t", +// // soundex_f, "f", +// // regex("(?![aeiouy])h(?![aeiouy])"), "", +// // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "", +// regex_vowel, "" +// ]; +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// if(str){ +// +// str = encode_advanced.call(this, str, /* skip post-processing: */ true); +// +// if(str.length > 1){ +// +// //str = replace(str, pairs); +// //str = str.replace(regex_vowel, ""); +// str = str.charAt(0) + str.substring(1).replace(regex_vowel, ""); +// } +// +// if(str.length > 1){ +// +// str = collapse(str); +// } +// +// if(str){ +// +// str = str.split(" "); +// } +// } +// +// return str || []; +// } \ No newline at end of file diff --git a/dist/module/lang/latin/simple.js b/dist/module/lang/latin/simple.js index 1b9ca40..b01236e 100644 --- a/dist/module/lang/latin/simple.js +++ b/dist/module/lang/latin/simple.js @@ -1,45 +1,343 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline, normalize, regex_whitespace, regex } from "../../lang.js"; +import Encoder from "../../encoder.js"; -export const rtl = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ -/* collapse: */!1; -export const tokenize = ""; export default { - encode: encode, - rtl: !1, - tokenize: "" + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: !0 +}; - // Charset Normalization - -};const //regex_whitespace = /\W+/, -//regex_strip = regex("[^a-z0-9 ]"), -regex_a = regex("[àáâãäå]"), - regex_e = regex("[èéêë]"), - regex_i = regex("[ìíîï]"), - regex_o = regex("[òóôõöő]"), - regex_u = regex("[ùúûüű]"), - regex_y = regex("[ýŷÿ]"), - regex_n = regex("ñ"), - regex_c = regex("[çc]"), - regex_s = regex("ß"), - regex_and = regex(" & "), - pairs = [regex_a, "a", regex_e, "e", regex_i, "i", regex_o, "o", regex_u, "u", regex_y, "y", regex_n, "n", regex_c, "k", regex_s, "s", regex_and, " and " -//regex_whitespace, " " -//regex_strip, "" -]; - - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str) { - - str = "" + str; - - return pipeline.call(this, - /* string: */normalize(str).toLowerCase(), - /* normalize: */!str.normalize && pairs, - /* split: */regex_whitespace, !1); -} \ No newline at end of file +// import { pipeline, regex } from "../../lang.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // p{Z} = whitespaces +// // p{S} = symbols (emotes) +// // p{P} = special chars +// // p{C} = controls (linebreak, tabulator) +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// const normalize = "".normalize && /[\u0300-\u036f]/g; +// const pairs = !normalize && new Map([ +// +// // Charset Normalization +// // String.normalize("NFKD").replace(/[\u0300-\u036f]/g, "") +// +// ["ª","a"], +// ["²","2"], +// ["³","3"], +// ["¹","1"], +// ["º","o"], +// ["¼","1⁄4"], +// ["½","1⁄2"], +// ["¾","3⁄4"], +// ["à","a"], +// ["á","a"], +// ["â","a"], +// ["ã","a"], +// ["ä","a"], +// ["å","a"], +// ["ç","c"], +// ["è","e"], +// ["é","e"], +// ["ê","e"], +// ["ë","e"], +// ["ì","i"], +// ["í","i"], +// ["î","i"], +// ["ï","i"], +// ["ñ","n"], +// ["ò","o"], +// ["ó","o"], +// ["ô","o"], +// ["õ","o"], +// ["ö","o"], +// ["ù","u"], +// ["ú","u"], +// ["û","u"], +// ["ü","u"], +// ["ý","y"], +// ["ÿ","y"], +// ["ā","a"], +// ["ă","a"], +// ["ą","a"], +// ["ć","c"], +// ["ĉ","c"], +// ["ċ","c"], +// ["č","c"], +// ["ď","d"], +// ["ē","e"], +// ["ĕ","e"], +// ["ė","e"], +// ["ę","e"], +// ["ě","e"], +// ["ĝ","g"], +// ["ğ","g"], +// ["ġ","g"], +// ["ģ","g"], +// ["ĥ","h"], +// ["ĩ","i"], +// ["ī","i"], +// ["ĭ","i"], +// ["į","i"], +// ["ij","ij"], +// ["ĵ","j"], +// ["ķ","k"], +// ["ĺ","l"], +// ["ļ","l"], +// ["ľ","l"], +// ["ŀ","l"], +// ["ń","n"], +// ["ņ","n"], +// ["ň","n"], +// ["ʼn","n"], +// ["ō","o"], +// ["ŏ","o"], +// ["ő","o"], +// ["ŕ","r"], +// ["ŗ","r"], +// ["ř","r"], +// ["ś","s"], +// ["ŝ","s"], +// ["ş","s"], +// ["š","s"], +// ["ţ","t"], +// ["ť","t"], +// ["ũ","u"], +// ["ū","u"], +// ["ŭ","u"], +// ["ů","u"], +// ["ű","u"], +// ["ų","u"], +// ["ŵ","w"], +// ["ŷ","y"], +// ["ź","z"], +// ["ż","z"], +// ["ž","z"], +// ["ſ","s"], +// ["ơ","o"], +// ["ư","u"], +// ["dž","dz"], +// ["lj","lj"], +// ["nj","nj"], +// ["ǎ","a"], +// ["ǐ","i"], +// ["ǒ","o"], +// ["ǔ","u"], +// ["ǖ","u"], +// ["ǘ","u"], +// ["ǚ","u"], +// ["ǜ","u"], +// ["ǟ","a"], +// ["ǡ","a"], +// ["ǣ","ae"], +// ["æ","ae"], +// ["ǽ","ae"], +// ["ǧ","g"], +// ["ǩ","k"], +// ["ǫ","o"], +// ["ǭ","o"], +// ["ǯ","ʒ"], +// ["ǰ","j"], +// ["dz","dz"], +// ["ǵ","g"], +// ["ǹ","n"], +// ["ǻ","a"], +// ["ǿ","ø"], +// ["ȁ","a"], +// ["ȃ","a"], +// ["ȅ","e"], +// ["ȇ","e"], +// ["ȉ","i"], +// ["ȋ","i"], +// ["ȍ","o"], +// ["ȏ","o"], +// ["ȑ","r"], +// ["ȓ","r"], +// ["ȕ","u"], +// ["ȗ","u"], +// ["ș","s"], +// ["ț","t"], +// ["ȟ","h"], +// ["ȧ","a"], +// ["ȩ","e"], +// ["ȫ","o"], +// ["ȭ","o"], +// ["ȯ","o"], +// ["ȱ","o"], +// ["ȳ","y"], +// ["ʰ","h"], +// ["ʱ","h"], +// ["ɦ","h"], +// ["ʲ","j"], +// ["ʳ","r"], +// ["ʴ","ɹ"], +// ["ʵ","ɻ"], +// ["ʶ","ʁ"], +// ["ʷ","w"], +// ["ʸ","y"], +// ["ˠ","ɣ"], +// ["ˡ","l"], +// ["ˢ","s"], +// ["ˣ","x"], +// ["ˤ","ʕ"], +// ["ΐ","ι"], +// ["ά","α"], +// ["έ","ε"], +// ["ή","η"], +// ["ί","ι"], +// ["ΰ","υ"], +// ["ϊ","ι"], +// ["ϋ","υ"], +// ["ό","ο"], +// ["ύ","υ"], +// ["ώ","ω"], +// ["ϐ","β"], +// ["ϑ","θ"], +// ["ϒ","Υ"], +// ["ϓ","Υ"], +// ["ϔ","Υ"], +// ["ϕ","φ"], +// ["ϖ","π"], +// ["ϰ","κ"], +// ["ϱ","ρ"], +// ["ϲ","ς"], +// ["ϵ","ε"], +// ["й","и"], +// ["ѐ","е"], +// ["ё","е"], +// ["ѓ","г"], +// ["ї","і"], +// ["ќ","к"], +// ["ѝ","и"], +// ["ў","у"], +// ["ѷ","ѵ"], +// ["ӂ","ж"], +// ["ӑ","а"], +// ["ӓ","а"], +// ["ӗ","е"], +// ["ӛ","ә"], +// ["ӝ","ж"], +// ["ӟ","з"], +// ["ӣ","и"], +// ["ӥ","и"], +// ["ӧ","о"], +// ["ӫ","ө"], +// ["ӭ","э"], +// ["ӯ","у"], +// ["ӱ","у"], +// ["ӳ","у"], +// ["ӵ","ч"] +// +// // Charset Normalization +// +// // ["à", "a"], +// // ["á", "a"], +// // ["â", "a"], +// // ["ã", "a"], +// // ["ä", "a"], +// // ["å", "a"], +// // +// // ["è", "e"], +// // ["é", "e"], +// // ["ê", "e"], +// // ["ë", "e"], +// // +// // ["ì", "i"], +// // ["í", "i"], +// // ["î", "i"], +// // ["ï", "i"], +// // +// // ["ò", "o"], +// // ["ó", "o"], +// // ["ô", "o"], +// // ["õ", "o"], +// // ["ö", "o"], +// // ["ő", "o"], +// // +// // ["ù", "u"], +// // ["ú", "u"], +// // ["û", "u"], +// // ["ü", "u"], +// // ["ű", "u"], +// // +// // ["ý", "y"], +// // ["ŷ", "y"], +// // ["ÿ", "y"], +// // +// // ["ñ", "n"], +// // ["ç", "c"], +// // ["ß", "s"] +// +// // Special Chars Removal +// +// // [",", ""], +// // [".", ""], +// // ["'", ""] +// +// // Non-Whitespace Separators +// +// // split by default p{P} +// // ["-", " "], +// // [":", " "], +// // ["_", " "], +// // ["|", " "], +// // ["/", " "], +// // ["\\", " "] +// ]); +// +// // let pairs; +// // +// // if(!normalize){ +// // +// // // Charset Normalization +// // +// // const regex_a = regex("[àáâãäå]"), +// // regex_e = regex("[èéêë]"), +// // regex_i = regex("[ìíîï]"), +// // regex_o = regex("[òóôõöő]"), +// // regex_u = regex("[ùúûüű]"), +// // regex_y = regex("[ýŷÿ]"), +// // regex_n = regex("ñ"), +// // //regex_c = regex("[çc]"), +// // regex_c = regex("ç"), +// // regex_s = regex("ß"), +// // //regex_and = regex(" & "), +// // regex_and = regex("&"); +// // +// // pairs = [ +// // regex_a, "a", +// // regex_e, "e", +// // regex_i, "i", +// // regex_o, "o", +// // regex_u, "u", +// // regex_y, "y", +// // regex_n, "n", +// // //regex_c, "k", +// // regex_c, "c", +// // regex_s, "s", +// // regex_and, " and " +// // ]; +// // } +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// //str = "" + str; +// +// return pipeline.call( +// +// this, +// /* string: */ (/*normalize ? str.normalize("NFD").replace(normalize, "") :*/ "" + str).toLowerCase(), +// /* normalize: */ pairs, +// /* split: */ whitespace, +// ///* collapse: */ false +// ); +// } \ No newline at end of file diff --git a/dist/module/lang/latin/soundex.js b/dist/module/lang/latin/soundex.js new file mode 100644 index 0000000..0a38309 --- /dev/null +++ b/dist/module/lang/latin/soundex.js @@ -0,0 +1,52 @@ +export default { + normalize: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, + dedupe: /* suggest */ /* append: */ /* enrich */!1, + include: { + letter: !0 + }, + finalize: function (arr) { + for (let i = 0; i < arr.length; i++) { + arr[i] = soundex(arr[i]); + } + } +}; + +const codes = { + a: "", e: "", i: "", o: "", u: "", y: "", + b: 1, f: 1, p: 1, v: 1, + c: 2, g: 2, j: 2, k: 2, q: 2, s: 2, x: 2, z: 2, ß: 2, + d: 3, t: 3, + l: 4, + m: 5, n: 5, + r: 6 +}; + +function soundex(stringToEncode) { + let encodedString = stringToEncode.charAt(0), + last = codes[encodedString]; + + for (let i = 1, char; i < stringToEncode.length; i++) { + char = stringToEncode.charAt(i); + // Remove all occurrences of "h" and "w" + if ("h" !== char && "w" !== char) { + // Replace all consonants with digits + char = codes[char]; + // Remove all occurrences of a,e,i,o,u,y except first letter + if (char) { + // Replace all adjacent same digits with one digit + if (char !== last) { + encodedString += char; + last = char; + if (4 === encodedString.length) { + break; + } + } + } + } + } + // while(encodedString.length < 4){ + // encodedString += "0"; + // } + return encodedString; +} \ No newline at end of file diff --git a/dist/module/polyfill.js b/dist/module/polyfill.js deleted file mode 100644 index e8c7408..0000000 --- a/dist/module/polyfill.js +++ /dev/null @@ -1,74 +0,0 @@ - -export let promise = Promise; - -Object.assign || (Object.assign = function () { - const args = arguments, - size = args.length, - obj = args[0]; - - - for (let x = 1, current, keys, length; x < size; x++) { - - current = args[x]; - keys = Object.keys(current); - length = keys.length; - - for (let i = 0, key; i < length; i++) { - - key = keys[i]; - obj[key] = current[key]; - } - } - - return obj; -}); - -// Object.values || (Object.values = function(obj){ -// -// const keys = Object.keys(obj); -// const length = keys.length; -// const values = new Array(length); -// -// for(let x = 0; x < length; x++){ -// -// values[x] = obj[keys[x]]; -// } -// -// return values; -// }); - -if (!promise) { - - /** - * @param {Function} fn - * @constructor - */ - - function SimplePromise(fn) { - - this.callback = null; - - const self = this; - - fn(function (val) { - - if (self.callback) { - - self.callback(val); - // self.callback = null; - // self = null; - } - }); - } - - /** - * @param {Function} callback - */ - - SimplePromise.prototype.then = function (callback) { - - this.callback = callback; - }; - - promise = SimplePromise; -} \ No newline at end of file diff --git a/dist/module/preset.js b/dist/module/preset.js index 5101a98..30c0171 100644 --- a/dist/module/preset.js +++ b/dist/module/preset.js @@ -1,86 +1,53 @@ import { is_string } from "./common.js"; +import { IndexOptions } from "./type.js"; /** * @enum {Object} * @const */ -const preset = { +const presets = { memory: { - charset: "latin:extra", - //tokenize: "strict", - resolution: 3, - //threshold: 0, - minlength: 4, - fastupdate: /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ - /* collapse: */ - /* collapse: */!1 + resolution: 1 }, performance: { - //charset: "latin", - //tokenize: "strict", - resolution: 3, - minlength: 3, - //fastupdate: true, - optimize: !1, //fastupdate: true, + resolution: 6, + fastupdate: /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/, context: { - depth: 2, resolution: 1 - //bidirectional: false + depth: 1, + resolution: 3 } }, match: { - charset: "latin:extra", - tokenize: "reverse" - //resolution: 9, - //threshold: 0 + tokenize: "forward" }, score: { - charset: "latin:advanced", - //tokenize: "strict", - resolution: 20, - minlength: 3, + resolution: 9, context: { - depth: 3, + depth: 2, resolution: 9 - //bidirectional: true } - }, - - default: { - // charset: "latin:default", - // tokenize: "strict", - // resolution: 3, - // threshold: 0, - // depth: 3 } - - // "fast": { - // //charset: "latin", - // //tokenize: "strict", - // threshold: 8, - // resolution: 9, - // depth: 1 - // } }; +/** + * + * @param {!IndexOptions|string} options + * @return {IndexOptions} + */ + export default function apply_preset(options) { - if (is_string(options)) { + const preset = is_string(options) ? options : options.preset; - options = preset[options]; - } else { - - const preset = options.preset; - - if (preset) { - - options = Object.assign({}, preset[preset], /** @type {Object} */options); - } + if (preset) { + options = Object.assign({}, presets[preset], /** @type {Object} */options); } return options; diff --git a/dist/module/resolve/and.js b/dist/module/resolve/and.js new file mode 100644 index 0000000..9e7994f --- /dev/null +++ b/dist/module/resolve/and.js @@ -0,0 +1,242 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object, get_max_len } from "../common.js"; +// import xor from "./xor.js"; +// import or from "./or.js"; +// import not from "./not.js"; + +Resolver.prototype.and = function () { + if (this.result.length) { + + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.and.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.and.apply(this, first_argument); + } + } + + // for(let i = 0; i < args.length; i++){ + // if(args[i].result instanceof Promise){ + // return; + // } + // } + + // if(args.length < 2){ + // if(first_argument.index){ + // first_argument.resolve = false; + // return first_argument.index.search(first_argument); + // } + // } + + let final = [], + promises = [], + limit = 0, + offset = 0, + enrich, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.or) { + result = this.or(query.or); + } else if (query.xor) { + result = this.xor(query.xor); + } else if (query.not) { + result = this.not(query.not); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + final = [self.result].concat(final); + self.result = intersect(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + final = [this.result].concat(final); + this.result = intersect(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? this.result : this; + } + return this; +}; + +/** + * Aggregate the intersection of N raw results + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function intersect(result, limit, offset, enrich, resolve, boost) { + + // if(!result.length){ + // // todo remove + // console.log("Empty Result") + // return result; + // } + + if (2 > result.length) { + // todo remove + //console.log("Single Result") + return []; + // if(resolve){ + // return default_resolver(result[0], limit, offset, enrich); + // } + // else{ + // return result[0]; + // } + } + + let final = [], + count = 0, + contain = create_object(), + maxres = get_max_len(result); + + // fast path single slot + // if(result.length < 2){ + // if(limit || offset){ + // let res = result[0]; + // for(let j = 0, ids; j < res.length; j++){ + // ids = res[j]; + // if(!ids) continue; + // for(let k = 0, id; k < ids.length; k++){ + // id = ids[k]; + // if(offset){ + // offset--; + // continue; + // } + // if(resolve){ + // final.push(id); + // } + // else{ + // final[j + this.boost] || (final[j + this.boost] = []); + // final[j + this.boost].push(id); + // } + // if(limit && ++count === limit){ + // this.boost = 0; + // return final; + // } + // } + // } + // } + // this.boost = 0; + // return result[0]; + // } + + if (!maxres) return final; + + // for(let j = 0, ids, res = result[0]; j < res.length; j++){ + // ids = res[j]; + // for(let k = 0; k < ids.length; k++){ + // contain[ids[k]] = 1; + // } + // } + + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res || !res.length) return []; + let contain_new = create_object(), + match = 0, + last_round = i === result.length - 1; + + + for (let j = 0, ids; j < maxres; j++) { + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id, min; k < ids.length; k++) { + id = ids[k]; + // fill in first round + if (!i) { + // shift resolution +1 + // shift resolution by boost (inverse) + contain_new[id] = j + 1 + (i ? boost : 0); + match = 1; + } + // result in last round + else if (last_round) { + if (min = contain[id]) { + match = 1; + //if(!contain_new[id]){ + if (offset) { + offset--; + continue; + } + if (resolve) { + final.push(id); + } else { + // reduce resolution -1 + min--; + if (j < min) min = j; + final[min] || (final[min] = []); + final[min].push(id); + } + if (limit && ++count === limit) { + //this.boost = 0; + return final; + } + // shift resolution +1 + //contain_new[id] = min + 1; + //} + } + } + // check for intersection + else if (min = contain[id]) { + // shift resolution +1 + if (j + 1 < min) min = j + 1; + contain_new[id] = min; + match = 1; + } + } + } + + if (!match) { + //this.boost = 0; + return []; + } + + contain = contain_new; + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/dist/module/resolve/default.js b/dist/module/resolve/default.js new file mode 100644 index 0000000..2edeb6d --- /dev/null +++ b/dist/module/resolve/default.js @@ -0,0 +1,98 @@ +import { concat } from "../common.js"; + +/* + from -> res[score][id] + to -> [id] +*/ + +/** + * Aggregate the union of a single raw result + * @param {!Array} result + * @param {!number} limit + * @param {number=} offset + * @param {boolean=} enrich + * @return Array + */ + +export default function (result, limit, offset, enrich) { + + // fast path: when there is just one slot in the result + if (1 === result.length) { + result = result[0]; + result = offset || result.length > limit ? limit ? result.slice(offset, offset + limit) : result.slice(offset) : result; + return enrich ? enrich_result(result) : result; + } + + // this is an optimized workaround instead of + // just doing result = concat(result) + + let final = []; + + for (let i = 0, arr, len; i < result.length; i++) { + if (!(arr = result[i]) || !(len = arr.length)) continue; + + if (offset) { + // forward offset pointer + if (offset >= len) { + offset -= len; + continue; + } + // apply offset pointer when length differs + if (offset < len) { + arr = limit ? arr.slice(offset, offset + limit) : arr.slice(offset); + len = arr.length; + offset = 0; + } + } + + if (!final.length) { + // fast path: when limit was reached in first slot + if (len >= limit) { + if (len > limit) { + arr = arr.slice(0, limit); + } + return enrich ? enrich_result(arr) : arr; + } + final = [arr]; + } else { + if (len > limit) { + arr = arr.slice(0, limit); + len = arr.length; + } + final.push(arr); + } + + // reduce limit + limit -= len; + + // todo remove + // if(limit < 0){ + // throw new Error("Impl.Error"); + // } + + // break if limit was reached + if (!limit) { + break; + } + } + + // todo remove + if (!final.length) { + //throw new Error("No results found"); + return final; + } + + final = 1 < final.length ? concat(final) : final[0]; + + return enrich ? enrich_result(final) : final; +} + +function enrich_result(ids) { + for (let i = 0; i < ids.length; i++) { + ids[i] = { + score: i, + id: ids[i] + }; + } + return ids; +} \ No newline at end of file diff --git a/dist/module/resolve/not.js b/dist/module/resolve/not.js new file mode 100644 index 0000000..13364fb --- /dev/null +++ b/dist/module/resolve/not.js @@ -0,0 +1,115 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object } from "../common.js"; +// import or from "./or.js"; +// import and from "./and.js"; +// import xor from "./xor.js"; + +Resolver.prototype.not = function () { + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.not.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.not.apply(this, first_argument); + } + } + + let final = [], + promises = [], + limit = 0, + offset = 0, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.or) { + result = this.or(query.or); + } else if (query.and) { + result = this.and(query.and); + } else if (query.xor) { + result = this.xor(query.xor); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + self.result = exclusion.call(self, final, limit, offset, resolve); + return resolve ? self.result : self; + }); + } + + this.result = exclusion.call(this, final, limit, offset, resolve); + return resolve ? this.result : this; +}; + +/** + * @param result + * @param limit + * @param offset + * @param resolve + * @this Resolver + * @return {Array} + */ + +function exclusion(result, limit, offset, resolve) { + + if (!result.length) { + return this.result; + } + + const final = [], + exclude = new Set(result.flat().flat()); + + + for (let j = 0, ids; j < this.result.length; j++) { + ids = this.result[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + if (!exclude.has(id)) { + if (resolve) { + final.push(id); + } else { + final[j] || (final[j] = []); + final[j].push(id); + } + } + } + } + + return final; +} \ No newline at end of file diff --git a/dist/module/resolve/or.js b/dist/module/resolve/or.js new file mode 100644 index 0000000..d14981c --- /dev/null +++ b/dist/module/resolve/or.js @@ -0,0 +1,182 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object, get_max_len } from "../common.js"; +// import xor from "./xor.js"; +// import and from "./and.js"; +// import not from "./not.js"; + +Resolver.prototype.or = function () { + + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.or.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.or.apply(this, first_argument); + } + } + + // for(let i = 0; i < args.length; i++){ + // if(args[i].result instanceof Promise){ + // return; + // } + // } + + // if(args.length < 2){ + // if(first_argument.index){ + // first_argument.resolve = false; + // const result = first_argument.index.search(first_argument); + // if(result instanceof Promise){ + // result.then(function(result){ + // final = self.result.concat(result); + // self.result = resolver(final, limit, offset, enrich, !resolve); + // return resolve ? self.result : self; + // }); + // } + // else{ + // final = this.result.concat(result); + // this.result = resolver(final, limit, offset, enrich, !resolve); + // return resolve ? this.result : this; + // } + // } + // } + + let final = [], + promises = [], + limit = 0, + offset = 0, + enrich, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.and) { + result = this.and(query.and); + } else if (query.xor) { + result = this.xor(query.xor); + } else if (query.not) { + result = this.not(query.not); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + self.result.length && (final = [self.result].concat(final)); + self.result = resolver(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + this.result.length && (final = [this.result].concat(final)); + this.result = resolver(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? this.result : this; +}; + +/** + * Aggregate the union of N raw results + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function resolver(result, limit, offset, enrich, resolve, boost) { + + if (!result.length) { + // todo remove + //console.log("Empty Result") + return result; + } + + if ("object" == typeof limit) { + offset = limit.offset || 0; + enrich = limit.enrich || !1; + limit = limit.limit || 0; + } + + if (2 > result.length) { + // todo remove + //console.log("Single Result") + if (resolve) { + return default_resolver(result[0], limit, offset, enrich); + } else { + return result[0]; + } + } + + let final = [], + count = 0, + dupe = create_object(), + maxres = get_max_len(result); + + + for (let j = 0, ids; j < maxres; j++) { + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res) continue; + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + if (!dupe[id]) { + dupe[id] = 1; + if (offset) { + offset--; + continue; + } + if (resolve) { + final.push(id); + } else { + // shift resolution by boost (inverse) + const index = j + (i ? boost : 0); + final[index] || (final[index] = []); + final[index].push(id); + } + if (limit && ++count === limit) { + //this.boost = 0; + return final; + } + } + } + } + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/dist/module/resolve/xor.js b/dist/module/resolve/xor.js new file mode 100644 index 0000000..1eed9e8 --- /dev/null +++ b/dist/module/resolve/xor.js @@ -0,0 +1,154 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object } from "../common.js"; +// import or from "./or.js"; +// import and from "./and.js"; +// import not from "./not.js"; + +Resolver.prototype.xor = function () { + const self = this; + let args = arguments, + first_argument = args[0]; + + + if (first_argument instanceof Promise) { + return first_argument.then(function () { + return self.xor.apply(self, args); + }); + } + + if (first_argument[0]) { + // fix false passed parameter style + if (first_argument[0].index) { + return this.xor.apply(this, first_argument); + } + } + + let final = [], + promises = [], + limit = 0, + offset = 0, + enrich, + resolve; + + + for (let i = 0, query; i < args.length; i++) { + if (query = args[i]) { + + let result; + if (query instanceof Resolver) { + result = query.result; + } else if (query.constructor === Array) { + result = query; + } else if (query.index) { + query.resolve = /* suggest */ /* append: */ /* enrich */!1; + result = query.index.search(query).result; + } else if (query.or) { + result = this.or(query.or); + } else if (query.and) { + result = this.and(query.and); + } else if (query.not) { + result = this.not(query.not); + } else { + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if (result instanceof Promise) { + promises.push(result); //{ query, result }; + } + } + } + + if (promises.length) { + return Promise.all(promises).then(function () { + self.result.length && (final = [self.result].concat(final)); + self.result = exclusive(final, limit, offset, enrich, !resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + this.result.length && (final = [this.result].concat(final)); + this.result = exclusive(final, limit, offset, enrich, !resolve, self.boostval); + return resolve ? this.result : this; +}; + +/** + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function exclusive(result, limit, offset, enrich, resolve, boost) { + + if (!result.length) { + // todo remove + //console.log("Empty Result") + return result; + } + + if (2 > result.length) { + // todo remove + //console.log("Single Result") + if (resolve) { + return default_resolver(result[0], limit, offset, enrich); + } else { + return result[0]; + } + } + + const final = [], + check = create_object(); + + + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res) continue; + + for (let j = 0, ids; j < res.length; j++) { + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + check[id] ? check[id]++ : check[id] = 1; + } + } + } + + for (let i = 0, res; i < result.length; i++) { + res = result[i]; + if (!res) continue; + + for (let j = 0, ids; j < res.length; j++) { + ids = res[j]; + if (!ids) continue; + + for (let k = 0, id; k < ids.length; k++) { + id = ids[k]; + if (1 === check[id]) { + if (resolve) { + final.push(id); + } else { + // shift resolution by boost (inverse) + const index = j + (i ? boost : 0); + final[index] || (final[index] = []); + final[index].push(id); + } + } + } + } + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/dist/module/resolver.js b/dist/module/resolver.js new file mode 100644 index 0000000..fbd1bf7 --- /dev/null +++ b/dist/module/resolver.js @@ -0,0 +1,99 @@ +import default_resolver from "./resolve/default.js"; +import { set_resolve } from "./index/search.js"; +// import or from "./resolve/or.js"; +// import and from "./resolve/and.js"; +// import xor from "./resolve/xor.js"; +// import not from "./resolve/not.js"; +import "./resolve/or.js"; +import "./resolve/and.js"; +import "./resolve/xor.js"; +import "./resolve/not.js"; + +/** + * @param result + * @constructor + */ + +export default function Resolver(result) { + if (result && result.index) { + result.resolve = /* suggest */ /* append: */ /* enrich */!1; + this.index = result.index; + return result.index.search(result); + } + if (!(this instanceof Resolver)) { + return new Resolver(result); + } + if (result instanceof Resolver) { + // todo remove + //console.log("Resolver Loopback") + return result; + } + this.index = null; + this.result = result || []; + this.boostval = 0; +} + +// Resolver.prototype.or = or; +// Resolver.prototype.and = and; +// Resolver.prototype.not = not; +// Resolver.prototype.xor = xor; + +Resolver.prototype.limit = function (limit) { + if (this.result.length) { + const final = []; + let count = 0; + for (let j = 0, ids; j < this.result.length; j++) { + ids = this.result[j]; + if (ids.length + count < limit) { + final[j] = ids; + count += ids.length; + } else { + final[j] = ids.slice(0, limit - count); + this.result = final; + break; + } + } + } + return this; +}; + +Resolver.prototype.offset = function (offset) { + if (this.result.length) { + const final = []; + let count = 0; + for (let j = 0, ids; j < this.result.length; j++) { + ids = this.result[j]; + if (ids.length + count < offset) { + count += ids.length; + } else { + final[j] = ids.slice(offset - count); + count = offset; + } + } + this.result = final; + } + return this; +}; + +Resolver.prototype.boost = function (boost) { + this.boostval += boost; + return this; +}; + +Resolver.prototype.resolve = function (limit, offset, enrich) { + set_resolve(1); + const result = this.result; + this.index = null; + this.result = null; + + if (result.length) { + if ("object" == typeof limit) { + enrich = limit.enrich; + offset = limit.offset; + limit = limit.limit; + } + return default_resolver(result, limit || 100, offset, enrich); + } + + return result; +}; \ No newline at end of file diff --git a/dist/module/serialize.js b/dist/module/serialize.js index 0baab85..66ad30b 100644 --- a/dist/module/serialize.js +++ b/dist/module/serialize.js @@ -1,36 +1,44 @@ // TODO return promises instead of inner await -import { IndexInterface, DocumentInterface } from "./type.js"; +import Index from "./index.js"; +import Document from "./document.js"; import { create_object, is_string } from "./common.js"; function async(callback, self, field, key, index_doc, index, data, on_done) { - setTimeout(function () { + //setTimeout(function(){ - const res = callback(field ? field + "." + key : key, JSON.stringify(data)); + const res = callback(field ? field + "." + key : key, JSON.stringify(data)); - // await isn't supported by ES5 + // await isn't supported by ES5 - if (res && res.then) { + if (res && res.then) { - res.then(function () { - - self.export(callback, self, field, index_doc, index + 1, on_done); - }); - } else { + res.then(function () { self.export(callback, self, field, index_doc, index + 1, on_done); - } - }); + }); + } else { + + self.export(callback, self, field, index_doc, index + 1, on_done); + } + //}); } /** - * @this IndexInterface + * @param callback + * @param self + * @param field + * @param index_doc + * @param index + * @param on_done + * @this {Index|Document} */ export function exportIndex(callback, self, field, index_doc, index, on_done) { - let return_value = /* append: */ /* skip update: */ /* skip_update: */ /* skip post-processing: */!0; + let return_value = /* tag? */ /* stringify */ /* stringify */ /* skip update: */ /* append: */ /* skip update: */ /* skip_update: */ /* skip deletion */ // splice: + !0 /*await rows.hasNext()*/ /*await rows.hasNext()*/ /*await rows.hasNext()*/; if ('undefined' == typeof on_done) { return_value = new Promise(resolve => { on_done = resolve; @@ -51,13 +59,13 @@ export function exportIndex(callback, self, field, index_doc, index, on_done) { data = create_object(); - for (let key in this.register) { + for (let key of this.reg.keys()) { data[key] = 1; } } else { - data = this.register; + data = this.reg; } break; @@ -100,7 +108,7 @@ export function exportIndex(callback, self, field, index_doc, index, on_done) { } /** - * @this IndexInterface + * @this Index */ export function importIndex(key, data) { @@ -126,8 +134,8 @@ export function importIndex(key, data) { // fastupdate isn't supported by import - this.fastupdate = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* collapse: */!1; - this.register = data; + this.fastupdate = /* suggest */ /* append: */ /* enrich */!1; + this.reg = data; break; case "map": @@ -143,7 +151,7 @@ export function importIndex(key, data) { } /** - * @this DocumentInterface + * @this Document */ export function exportDocument(callback, self, field, index_doc, index, on_done) { @@ -165,16 +173,16 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) self = this; - setTimeout(function () { + //setTimeout(function(){ - if (!idx.export(callback, self, index ? field /*.replace(":", "-")*/ : "", index_doc, index++, on_done)) { + if (!idx.export(callback, self, index ? field /*.replace(":", "-")*/ : "", index_doc, index++, on_done)) { - index_doc++; - index = 1; + index_doc++; + index = 1; - self.export(callback, self, field, index_doc, index, on_done); - } - }); + self.export(callback, self, field, index_doc, index, on_done); + } + //}); } else { let key, data; @@ -214,7 +222,7 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) } /** - * @this DocumentInterface + * @this Document */ export function importDocument(key, data) { @@ -241,12 +249,12 @@ export function importDocument(key, data) { // fastupdate isn't supported by import this.fastupdate = !1; - this.register = data; + this.reg = data; for (let i = 0, index; i < this.field.length; i++) { index = this.index[this.field[i]]; - index.register = data; + index.reg = data; index.fastupdate = !1; } diff --git a/dist/module/type.js b/dist/module/type.js index 1a784b3..0c377c0 100644 --- a/dist/module/type.js +++ b/dist/module/type.js @@ -1,69 +1,144 @@ -/** - * @interface - */ - -export function IndexInterface() { - - this.cache = null; - this.matcher = null; - this.stemmer = null; - this.filter = null; -} +// When you are looking for type definitions which fully describes the usage take a look into the index.d.ts file. +// Some of the types here aren't supposed to be used as public, they might be defined just for internal state. +import Encoder from "./encoder.js"; +import StorageInterface from "./db/interface.js"; /** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} collapse - * @returns {string|Array} + * @typedef IndexOptions {{ + * preset: string|undefined, + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * resolve: [boolean=true], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -//IndexInterface.prototype.pipeline; +export let IndexOptions; /** - * @param {!number|string} id - * @param {!string} content + * @typedef DocumentOptions {{ + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * db: StorageInterface|undefined, + * doc: DocumentDescriptor|Array|undefined, + * document: DocumentDescriptor|Array|undefined, + * worker: boolean|string|undefined + * }} */ - -IndexInterface.prototype.add; +export let DocumentOptions; /** - * @param {!number|string} id - * @param {!string} content + * @typedef ContextOptions {{ + * depth: number, + * bidirectional: boolean|undefined, + * resolution: number|undefined + * }} */ - -IndexInterface.prototype.append; +export let ContextOptions; /** - * @param {!string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @returns {Array} + * @typedef DocumentDescriptor {{ + * field: FieldOptions|Array|undefined, + * index: FieldOptions|Array|undefined, + * tag: TagOptions|Array|undefined, + * store: StoreOptions|Array|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.search; +export let DocumentDescriptor; /** - * @param {!number|string} id - * @param {!string} content + * @typedef FieldOptions {{ + * field: string, + * filter: Function|undefined, + * custom: Function|undefined, + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.update; +export let FieldOptions; /** - * @param {!number|string} id + * @typedef TagOptions {{ + * field: string, + * tag: Object>|Array|string, + * filter: Function|undefined, + * custom: Function|undefined, + * keystore: [number=0], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.remove; +export let TagOptions; /** - * @interface + * @typedef StoreOptions {{ + * field: string, + * filter: Function|undefined, + * custom: Function|undefined, + * keystore: [number=0], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ +export let StoreOptions; -export function DocumentInterface() { +/** + * @typedef SearchOptions {{ + * query: string=, + * limit: [number=100], + * offset: [number=0], + * context: boolean|undefined, + * suggest: [boolean=false], + * resolve: [boolean=true], + * enrich: [boolean=false], + * tag: Array|undefined + * }} + */ +export let SearchOptions; - this.field = null; +/** + * @typedef DocumentSearchOptions {{ + * query: string=, + * limit: [number=100], + * offset: [number=0], + * context: boolean|undefined, + * suggest: [boolean=false], + * enrich: [boolean=false], + * tag: Array|undefined, + * field: FieldOptions|Array|undefined, + * index: FieldOptions|Array|undefined, + * pluck: boolean|undefined, + * merge: [boolean=false] + * }} + */ +export let DocumentSearchOptions; - /** @type IndexInterface */ - this.index = null; -} \ No newline at end of file +export let EncoderOptions; +export let ResolverOptions; \ No newline at end of file diff --git a/dist/module/webpack.js b/dist/module/webpack.js new file mode 100644 index 0000000..723eaee --- /dev/null +++ b/dist/module/webpack.js @@ -0,0 +1,173 @@ + +import { SearchOptions, ContextOptions, DocumentDescriptor, DocumentSearchOptions, FieldOptions, IndexOptions, DocumentOptions } from "./type.js"; +import Document from "./document.js"; +import Index from "./index.js"; +import WorkerIndex from "./worker/index.js"; +import Resolver from "./resolver.js"; +import Encoder from "./encoder.js"; +import IdxDB from "./db/indexeddb/index.js"; +import { global_charset, global_lang } from "./global.js"; +import charset_exact from "./lang/latin/exact.js"; +import charset_default from "./lang/latin/default.js"; +import charset_simple from "./lang/latin/simple.js"; +import charset_balance from "./lang/latin/balance.js"; +import charset_advanced from "./lang/latin/advanced.js"; +import charset_extra from "./lang/latin/extra.js"; +import charset_soundex from "./lang/latin/soundex.js"; + +/** @export */Index.prototype.add; +/** @export */Index.prototype.append; +/** @export */Index.prototype.search; +/** @export */Index.prototype.update; +/** @export */Index.prototype.remove; +/** @export */Index.prototype.contain; +/** @export */Index.prototype.clear; +/** @export */Index.prototype.cleanup; + +/** @export */Document.prototype.add; +/** @export */Document.prototype.append; +/** @export */Document.prototype.search; +/** @export */Document.prototype.update; +/** @export */Document.prototype.remove; +/** @export */Document.prototype.contain; +/** @export */Document.prototype.clear; +/** @export */Document.prototype.cleanup; + +/** @export */Document.prototype.get; +/** @export */Document.prototype.set; + +/** @export */Index.prototype.searchCache; + +/** @export */Document.prototype.searchCache; + +/** @export */Index.prototype.addAsync; +/** @export */Index.prototype.appendAsync; +/** @export */Index.prototype.searchAsync; +/** @export */Index.prototype.updateAsync; +/** @export */Index.prototype.removeAsync; + +/** @export */Document.prototype.addAsync; +/** @export */Document.prototype.appendAsync; +/** @export */Document.prototype.searchAsync; +/** @export */Document.prototype.updateAsync; +/** @export */Document.prototype.removeAsync; + +/** @export */Index.prototype.export; +/** @export */Index.prototype.import; + +/** @export */Document.prototype.export; +/** @export */Document.prototype.import; + +/** @export */Index.prototype.mount; +/** @export */Index.prototype.commit; +/** @export */Index.db; + +/** @export */Document.prototype.mount; +/** @export */Document.prototype.commit; +/** @export */Document.db; + +/** @export */IndexOptions.preset; +/** @export */IndexOptions.context; +/** @export */IndexOptions.encoder; +/** @export */IndexOptions.encode; +/** @export */IndexOptions.resolution; +/** @export */IndexOptions.tokenize; +/** @export */IndexOptions.fastupdate; +/** @export */IndexOptions.score; +/** @export */IndexOptions.keystore; +/** @export */IndexOptions.rtl; +/** @export */IndexOptions.cache; +/** @export */IndexOptions.resolve; +/** @export */IndexOptions.db; + +/** @export */DocumentOptions.context; +/** @export */DocumentOptions.encoder; +/** @export */DocumentOptions.encode; +/** @export */DocumentOptions.resolution; +/** @export */DocumentOptions.tokenize; +/** @export */DocumentOptions.fastupdate; +/** @export */DocumentOptions.score; +/** @export */DocumentOptions.keystore; +/** @export */DocumentOptions.rtl; +/** @export */DocumentOptions.cache; +/** @export */DocumentOptions.db; +/** @export */DocumentOptions.doc; +/** @export */DocumentOptions.document; +/** @export */DocumentOptions.worker; + +/** @export */DocumentDescriptor.field; +/** @export */DocumentDescriptor.index; +/** @export */DocumentDescriptor.tag; +/** @export */DocumentDescriptor.store; + +/** @export */ContextOptions.depth; +/** @export */ContextOptions.bidirectional; +/** @export */ContextOptions.resolution; + +/** @export */SearchOptions.query; +/** @export */SearchOptions.limit; +/** @export */SearchOptions.offset; +/** @export */SearchOptions.context; +/** @export */SearchOptions.suggest; +/** @export */SearchOptions.resolve; +/** @export */SearchOptions.enrich; +/** @export */SearchOptions.tag; + +/** @export */DocumentSearchOptions.query; +/** @export */DocumentSearchOptions.limit; +/** @export */DocumentSearchOptions.offset; +/** @export */DocumentSearchOptions.context; +/** @export */DocumentSearchOptions.suggest; +/** @export */DocumentSearchOptions.enrich; +/** @export */DocumentSearchOptions.tag; +/** @export */DocumentSearchOptions.field; +/** @export */DocumentSearchOptions.index; +/** @export */DocumentSearchOptions.pluck; +/** @export */DocumentSearchOptions.merge; + +global_charset["latin:exact"] = charset_exact; +global_charset["latin:default"] = charset_default; +global_charset["latin:simple"] = charset_simple; +global_charset["latin:balance"] = charset_balance; +global_charset["latin:advanced"] = charset_advanced; +global_charset["latin:extra"] = charset_extra; +global_charset["latin:soundex"] = charset_soundex; + + +const FlexSearch = { + Index: Index, + Encoder: Encoder, + Charset: global_charset, + Language: global_lang, + Document: Document, + Worker: WorkerIndex, + Resolver: Resolver, + IndexedDB: IdxDB + //"registerCharset": registerCharset, + //"registerLanguage": registerLanguage +}; + +// Export as library (Bundle) +// -------------------------------- + +{ + + const root = self; + let prop; + + // AMD (RequireJS) + if ((prop = root.define) && prop.amd) { + prop([], function () { + return FlexSearch; + }); + } + // CommonJS + else if ("object" == typeof root.exports) { + root.exports = FlexSearch; + } + // Global (window) + else { + /** @export */ + root.FlexSearch = FlexSearch; + } +} \ No newline at end of file diff --git a/dist/module/worker/handler.js b/dist/module/worker/handler.js index 914439c..0ada932 100644 --- a/dist/module/worker/handler.js +++ b/dist/module/worker/handler.js @@ -1,6 +1,7 @@ import Index from "../index.js"; +import { IndexOptions } from "../type.js"; -export default function (data) { +export default (async function (data) { data = data.data; @@ -13,19 +14,26 @@ export default function (data) { switch (task) { case "init": - const options = data.options || {}, - factory = data.factory, - encode = options.encode; + /** @type IndexOptions */ + let options = data.options || {}, + filepath = options.config; - options.cache = /* normalize: */ /* collapse: */ /* normalize: */ - - /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* collapse: */!1; - - if (encode && 0 === encode.indexOf("function")) { - options.encode = Function("return " + encode)(); + if (filepath) { + options = filepath; + // will be replaced after build with the line below because + // there is an issue with closure compiler dynamic import + //options = await import(filepath); } + // deprecated: + // const encode = options.encode; + // if(encode && (encode.indexOf("function") === 0)){ + // options.encode = Function("return " + encode)(); + // } + + const factory = data.factory; + if (factory) { // export the FlexSearch global payload to "self" @@ -41,6 +49,7 @@ export default function (data) { self._index = new Index(options); } + postMessage({ id: data.id }); break; default: @@ -49,4 +58,4 @@ export default function (data) { postMessage("search" === task ? { id: id, msg: message } : { id: id }); } -} \ No newline at end of file +}); \ No newline at end of file diff --git a/dist/module/worker/index.js b/dist/module/worker/index.js index 5f42c56..8d7df5e 100644 --- a/dist/module/worker/index.js +++ b/dist/module/worker/index.js @@ -1,31 +1,22 @@ //import { promise as Promise } from "../polyfill.js"; +import { IndexOptions } from "../type.js"; import { create_object, is_function, is_object, is_string } from "../common.js"; import handler from "./handler.js"; let pid = 0; /** - * @param {Object=} options + * @param {IndexOptions=} options * @constructor */ function WorkerIndex(options) { if (!(this instanceof WorkerIndex)) { - return new WorkerIndex(options); } - let opt; - - if (options) { - - if (is_function(opt = options.encode)) { - - options.encode = opt.toString(); - } - } else { - + if (!options) { options = {}; } @@ -33,9 +24,7 @@ function WorkerIndex(options) { // we use "self" as a trap for node.js let factory = (self || window)._factory; - if (factory) { - factory = factory.toString(); } @@ -50,25 +39,37 @@ function WorkerIndex(options) { return; } - if (is_node_js) { + function onmessage(msg) { + msg = msg.data || msg; + const id = msg.id, + res = id && _self.resolver[id]; - this.worker.on("message", function (msg) { + if (res) { + res(msg.msg); + delete _self.resolver[id]; + } + } - _self.resolver[msg.id](msg.msg); - delete _self.resolver[msg.id]; + is_node_js ? this.worker.on("message", onmessage) : this.worker.onmessage = onmessage; + + if (options.config) { + + // when extern configuration needs to be loaded + // it needs to return a promise to await for + return new Promise(function (resolve) { + _self.resolver[++pid] = function () { + resolve(_self); + }; + _self.worker.postMessage({ + id: pid, + task: "init", + factory: factory, + options: options + }); }); - } else { - - this.worker.onmessage = function (msg) { - - msg = msg.data; - _self.resolver[msg.id](msg.msg); - delete _self.resolver[msg.id]; - }; } this.worker.postMessage({ - task: "init", factory: factory, options: options @@ -93,27 +94,22 @@ function register(key) { let callback; if (is_function(arg)) { - callback = arg; args.splice(args.length - 1, 1); } const promise = new Promise(function (resolve) { - - setTimeout(function () { - - self.resolver[++pid] = resolve; - self.worker.postMessage({ - - task: key, - id: pid, - args: args - }); + //setTimeout(function(){ + self.resolver[++pid] = resolve; + self.worker.postMessage({ + task: key, + id: pid, + args: args }); + //}); }); if (callback) { - promise.then(callback); return this; } else { @@ -125,12 +121,11 @@ function register(key) { function create(factory, is_node_js, worker_path) { - let worker; - - try { - - worker = is_node_js ? eval('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') : factory ? new Worker(URL.createObjectURL(new Blob(["onmessage=" + handler.toString()], { type: "text/javascript" }))) : new Worker(is_string(worker_path) ? worker_path : "worker/worker.js", { type: "module" }); - } catch (e) {} + let worker = is_node_js ? + // This eval will be removed when compiling, it isn't there in final build + (0, eval)('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') + //eval('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') + : factory ? new window.Worker(URL.createObjectURL(new Blob(["onmessage=" + handler.toString()], { type: "text/javascript" }))) : new window.Worker(is_string(worker_path) ? worker_path : "worker/worker.js", { type: "module" }); return worker; } \ No newline at end of file diff --git a/dist/module/worker/node.js b/dist/module/worker/node.js index 5326018..ba05f24 100644 --- a/dist/module/worker/node.js +++ b/dist/module/worker/node.js @@ -1,5 +1,10 @@ const { parentPort } = require("worker_threads"), - { Index } = require("../flexsearch.bundle.min.js"); + { join } = require("path"), + { Index } = require("../../dist/flexsearch.bundle.min.js"); +// TODO EXCHANGE + + +//const { Index } = require("../flexsearch.bundle.min.js"); let index; @@ -14,16 +19,25 @@ parentPort.on("message", function (data) { switch (task) { case "init": - const options = data.options || {}, - encode = options.encode; + let options = data.options || {}, + filepath = options.config; + // load extern field configuration - options.cache = /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* normalize: */ /* collapse: */ /* collapse: */!1; - - if (encode && 0 === encode.indexOf("function")) { - - options.encode = new Function("return " + encode)(); + if ("/" !== filepath[0] && "\\" !== filepath[0]) { + // current working directory + const dir = process.cwd(); + filepath = join(dir, filepath); } + if (filepath) { + options = require(filepath); + } + + // deprecated: + // const encode = options["encode"]; + // if(encode && (encode.indexOf("function") === 0)){ + // options["encode"] = new Function("return " + encode)(); + // } index = new Index(options); break; diff --git a/dist/node/node.js b/dist/node/node.js index 8a8a468..bea6cd6 100644 --- a/dist/node/node.js +++ b/dist/node/node.js @@ -1,5 +1,8 @@ const { parentPort } = require("worker_threads"); -const { Index } = require("../flexsearch.bundle.min.js"); +const { join } = require("path"); +// TODO EXCHANGE +const { Index } = require("../../dist/flexsearch.bundle.min.js"); +//const { Index } = require("../flexsearch.bundle.min.js"); let index; @@ -14,15 +17,24 @@ parentPort.on("message", function(data){ case "init": - const options = data["options"] || {}; - const encode = options["encode"]; + let options = data["options"] || {}; - options["cache"] = false; - - if(encode && (encode.indexOf("function") === 0)){ - - options["encode"] = new Function("return " + encode)(); + // load extern field configuration + let filepath = options["config"]; + if(filepath[0] !== "/" && filepath[0] !== "\\"){ + // current working directory + const dir = process.cwd(); + filepath = join(dir, filepath); } + if(filepath){ + options = require(filepath); + } + + // deprecated: + // const encode = options["encode"]; + // if(encode && (encode.indexOf("function") === 0)){ + // options["encode"] = new Function("return " + encode)(); + // } index = new Index(options); break; diff --git a/package.json b/package.json index 328e053..e38b05f 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "public": true, "preferGlobal": false, "name": "flexsearch", - "version": "0.7.43", + "version": "0.8.0", "description": "Next-Generation full text search library with zero dependencies.", "homepage": "https://github.com/nextapps-de/flexsearch/", "author": "Thomas Wilkerling", @@ -32,31 +32,32 @@ "browser": { "dist/flexsearch.bundle.min.js": "./dist/flexsearch.bundle.min.js", "dist/flexsearch.bundle.module.min.js": "./dist/flexsearch.bundle.module.min.js", - "worker_threads": false + "worker_threads": false, + "path": false }, "types": "./index.d.ts", "scripts": { - "build": "npm run copy && npm run build:bundle", - "build:bundle": "node task/build RELEASE=bundle DEBUG=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false", - "build:bundle:debug": "node task/build RELEASE=bundle DEBUG=true SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false FORMATTING=PRETTY_PRINT", - "build:compact": "node task/build RELEASE=compact DEBUG=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=false SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=true POLYFILL=false", - "build:compact:debug": "node task/build RELEASE=compact DEBUG=true SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=false SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=true POLYFILL=false FORMATTING=PRETTY_PRINT", - "build:light": "node task/build RELEASE=light DEBUG=false SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=false SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false", - "build:light:debug": "node task/build RELEASE=light DEBUG=true SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=false SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false FORMATTING=PRETTY_PRINT", + "build": "npm run build:bundle && npm run build:bundle:debug", + "build:bundle": "node task/build RELEASE=bundle DEBUG=false PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false", + "build:bundle:debug": "node task/build RELEASE=bundle DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT", + "build:compact": "node task/build RELEASE=compact DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false", + "build:compact:debug": "node task/build RELEASE=compact DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT", + "build:light": "node task/build RELEASE=light DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=true SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false", + "build:light:debug": "node task/build RELEASE=light DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=true SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT", "build:custom": "node task/build RELEASE=custom", - "build:es5": "node task/build RELEASE=es5 DEBUG=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true LANGUAGE_OUT=ECMASCRIPT5_STRICT POLYFILL=true", - "build:es5:debug": "node task/build RELEASE=es5 DEBUG=true SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true LANGUAGE_OUT=ECMASCRIPT5_STRICT POLYFILL=true FORMATTING=PRETTY_PRINT", + "build:es5": "node task/build RELEASE=es5 DEBUG=false PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=true SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false LANGUAGE_OUT=ECMASCRIPT5_STRICT", + "build:es5:debug": "node task/build RELEASE=es5 DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=true SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT LANGUAGE_OUT=ECMASCRIPT5_STRICT", "build:lang": "node task/build RELEASE=lang", "build:module": "node task/babel && exit 0", "build:module:debug": "node task/babel DEBUG=true && exit 0", "build:module:min": "node task/babel RELEASE=min && exit 0", - "build:module:bundle": "node task/build RELEASE=bundle.module DEBUG=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false", - "build:module:bundle:debug": "node task/build RELEASE=bundle.module DEBUG=true SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false FORMATTING=PRETTY_PRINT", - "build:module:compact": "node task/build RELEASE=compact.module DEBUG=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=false SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=true POLYFILL=false", - "build:module:compact:debug": "node task/build RELEASE=compact.module DEBUG=true SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=false SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=true POLYFILL=false FORMATTING=PRETTY_PRINT", - "build:module:light": "node task/build RELEASE=light.module DEBUG=false SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=false SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false", - "build:module:light:debug": "node task/build RELEASE=light.module DEBUG=true SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=false SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=false SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false FORMATTING=PRETTY_PRINT", - "build:all": "npm version --no-git-tag-version patch && npm run build:bundle && npm run build:bundle:debug && npm run build:light && npm run build:light:debug && npm run build:compact && npm run build:compact:debug && npm run build:es5 && npm run build:es5:debug && npm run build:module && npm run build:module:debug && npm run build:module:min && npm run build:module:bundle && npm run build:module:bundle:debug && npm run build:module:light && npm run build:module:light:debug && npm run build:module:compact && npm run build:module:compact:debug", + "build:module:bundle": "node task/build RELEASE=bundle.module DEBUG=false PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false", + "build:module:bundle:debug": "node task/build RELEASE=bundle.module DEBUG=true PROFILER=false SUPPORT_WORKER=true SUPPORT_ENCODER=true SUPPORT_CACHE=true SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=true SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=true SUPPORT_RESOLVER=true SUPPORT_KEYSTORE=true SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT", + "build:module:compact": "node task/build RELEASE=compact.module DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=false SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false", + "build:module:compact:debug": "node task/build RELEASE=compact.module DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=true SUPPORT_CACHE=false SUPPORT_ASYNC=true SUPPORT_STORE=true SUPPORT_TAGS=false SUPPORT_SUGGESTION=true SUPPORT_SERIALIZE=true SUPPORT_DOCUMENT=true POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT", + "build:module:light": "node task/build RELEASE=light.module DEBUG=false PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=true SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=false SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false", + "build:module:light:debug": "node task/build RELEASE=light.module DEBUG=true PROFILER=false SUPPORT_WORKER=false SUPPORT_ENCODER=false SUPPORT_CACHE=true SUPPORT_ASYNC=false SUPPORT_STORE=false SUPPORT_TAGS=false SUPPORT_SUGGESTION=false SUPPORT_SERIALIZE=false SUPPORT_DOCUMENT=false POLYFILL=false SUPPORT_PERSISTENT=false SUPPORT_RESOLVER=false SUPPORT_KEYSTORE=false SUPPORT_COMPRESSION=false FORMATTING=PRETTY_PRINT", + "build:all": "npm version --no-git-tag-version patch && npm run build:bundle && npm run build:bundle:debug && npm run build:light && npm run build:light:debug && npm run build:compact && npm run build:compact:debug && npm run build:module && npm run build:module:debug && npm run build:module:min && npm run build:module:bundle && npm run build:module:bundle:debug && npm run build:module:light && npm run build:module:light:debug && npm run build:module:compact && npm run build:module:compact:debug && npm run build:es5 && npm run build:es5:debug", "test": "cd test && npm install && npm run test" }, "files": [ @@ -86,6 +87,6 @@ "babel-plugin-transform-property-literals": "^6.9.4", "babel-plugin-transform-simplify-comparison-operators": "^6.9.4", "babel-plugin-transform-undefined-to-void": "^6.9.4", - "google-closure-compiler": "^20230802.0.0" + "google-closure-compiler": "^20240317.0.0" } } diff --git a/src/async.js b/src/async.js index 31f1c08..ed88091 100644 --- a/src/async.js +++ b/src/async.js @@ -1,51 +1,71 @@ -import { IndexInterface, DocumentInterface } from "./type.js"; -//import { promise as Promise } from "./polyfill.js"; -import { is_function, is_object, is_string } from "./common.js"; +import Document from "./document.js"; +import Index from "./index.js"; export default function(prototype){ - - register(prototype, "add"); - register(prototype, "append"); - register(prototype, "search"); - register(prototype, "update"); - register(prototype, "remove"); + register.call(prototype, "add"); + register.call(prototype, "append"); + register.call(prototype, "search"); + register.call(prototype, "update"); + register.call(prototype, "remove"); } -function register(prototype, key){ +// let cycle; +// let budget = 0; +// +// function tick(resolve){ +// cycle = null; +// budget = 0; +// resolve(); +// } - prototype[key + "Async"] = function(){ +/** + * @param {!string} key + * @this {Index|Document} + */ + +function register(key){ + this[key + "Async"] = function(){ + + // // prevent stack overflow of adding too much tasks to the same event loop + // // actually limit stack to 1,000,000 tasks every ~4ms + // cycle || ( + // cycle = new Promise(resolve => setTimeout(tick, 0, resolve)) + // ); + // + // // apply different performance budgets + // if(key === "update" || key === "remove" && this.fastupdate === false){ + // budget += 1000 * this.resolution; + // if(this.depth) + // budget += 1000 * this.resolution_ctx; + // } + // else if(key === "search"){ + // budget++; + // } + // else{ + // budget += 20 * this.resolution; + // if(this.depth) + // budget += 20 * this.resolution_ctx; + // } + // + // // wait for the event loop cycle + // if(budget >= 1e6){ + // await cycle; + // } - /** @type {IndexInterface|DocumentInterface} */ - const self = this; const args = /*[].slice.call*/(arguments); const arg = args[args.length - 1]; let callback; - if(is_function(arg)){ - + if(typeof arg === "function"){ callback = arg; delete args[args.length - 1]; } - const promise = new Promise(function(resolve){ - - setTimeout(function(){ - - self.async = true; - const res = self[key].apply(self, args); - self.async = false; - resolve(res); - }); - }); - - if(callback){ - - promise.then(callback); - return this; - } - else{ - - return promise; - } + this.async = true; + const res = this[key].apply(this, args); + this.async = false; + res.then ? res.then(callback) + : callback(res); + return res; }; } diff --git a/src/cache.js b/src/cache.js index 5cd7407..65cc890 100644 --- a/src/cache.js +++ b/src/cache.js @@ -1,169 +1,79 @@ -import { IndexInterface, DocumentInterface } from "./type.js"; -import { create_object, is_object } from "./common.js"; +import Index from "./index.js"; +import Document from "./document.js"; + +/** + * @param {string|Object} query + * @param {number|Object=} limit + * @param {Object=} options + * @this {Index|Document} + * @returns {Array|Promise} + */ + +export function searchCache(query, limit, options){ + + query = (typeof query === "object" + ? "" + query.query + : "" + query + ).toLowerCase(); + + //let encoded = this.encoder.encode(query).join(" "); + let cache = this.cache.get(query); + if(!cache){ + cache = this.search(query, limit, options); + if(cache instanceof Promise){ + const self = this; + cache.then(function(cache){ + self.cache.set(query, cache); + }); + } + this.cache.set(query, cache); + } + return cache; +} /** * @param {boolean|number=} limit * @constructor */ -function CacheClass(limit){ - +export default function CacheClass(limit){ /** @private */ - this.limit = (limit !== true) && limit; - + this.limit = (!limit || limit === true) ? 1000 : limit; /** @private */ - this.cache = create_object(); - + this.cache = new Map(); /** @private */ - this.queue = []; - - //this.clear(); + this.last = ""; } -export default CacheClass; - -/** - * @param {string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @this {IndexInterface} - * @returns {Array} - */ - -export function searchCache(query, limit, options){ - - if(is_object(query)){ - - query = query["query"]; - } - - let cache = this.cache.get(query); - - if(!cache){ - - cache = this.search(query, limit, options); - this.cache.set(query, cache); - } - - return cache; -} - -// CacheClass.prototype.clear = function(){ -// -// /** @private */ -// this.cache = create_object(); -// -// /** @private */ -// this.queue = []; -// }; - CacheClass.prototype.set = function(key, value){ - - if(!this.cache[key]){ - - // it is just a shame that native function array.shift() performs so bad - - // const length = this.queue.length; - // - // this.queue[length] = key; - // - // if(length === this.limit){ - // - // delete this.cache[this.queue.shift()]; - // } - - // the same bad performance - - // this.queue.unshift(key); - // - // if(this.queue.length === this.limit){ - // - // this.queue.pop(); - // } - - // fast implementation variant - - // let length = this.queue.length; - // - // if(length === this.limit){ - // - // length--; - // - // delete this.cache[this.queue[0]]; - // - // for(let x = 0; x < length; x++){ - // - // this.queue[x] = this.queue[x + 1]; - // } - // } - // - // this.queue[length] = key; - - // current fastest implementation variant - // theoretically that should not perform better compared to the example above - - let length = this.queue.length; - - if(length === this.limit){ - - delete this.cache[this.queue[length - 1]]; + if(!this.cache.has(key)){ + this.cache.set(this.last = key, value); + if(this.limit && this.cache.size > this.limit){ + this.cache.delete(this.cache.keys().next().value); } - else{ - - length++; - } - - for(let x = length - 1; x > 0; x--){ - - this.queue[x] = this.queue[x - 1]; - } - - this.queue[0] = key; } - - this.cache[key] = value; }; CacheClass.prototype.get = function(key){ - - const cache = this.cache[key]; - - if(this.limit && cache){ - - // probably the indexOf() method performs faster when matched content is on front (left-to-right) - // using lastIndexOf() does not help, it performs almost slower - - const pos = this.queue.indexOf(key); - - // if(pos < this.queue.length - 1){ - // - // const tmp = this.queue[pos]; - // this.queue[pos] = this.queue[pos + 1]; - // this.queue[pos + 1] = tmp; - // } - - if(pos){ - - const tmp = this.queue[pos - 1]; - this.queue[pos - 1] = this.queue[pos]; - this.queue[pos] = tmp; - } + const cache = this.cache.get(key); + if(cache && this.limit && this.last !== key){ + this.cache.delete(key); + this.cache.set(this.last = key, cache); } - return cache; }; -CacheClass.prototype.del = function(id){ - - for(let i = 0, item, key; i < this.queue.length; i++){ - - key = this.queue[i]; - item = this.cache[key]; - - if(item.includes(id)){ - - this.queue.splice(i--, 1); - delete this.cache[key]; +CacheClass.prototype.remove = function(id){ + for(const item of this.cache){ + const key = item[0]; + const value = item[1]; + if(value.includes(id)){ + this.cache.delete(key); } } }; + +CacheClass.prototype.clear = function(){ + this.cache.clear(); + this.last = ""; +}; diff --git a/src/common.js b/src/common.js index 878eb1f..f09fe66 100644 --- a/src/common.js +++ b/src/common.js @@ -1,6 +1,67 @@ -export function parse_option(value, default_value){ +/** + * @param {*} value + * @param {*} default_value + * @param {*=} merge_value + * @return {*} + */ - return typeof value !== "undefined" ? value : default_value; +export function parse_option(value, default_value, merge_value){ + + const type_merge = typeof merge_value; + const type_value = typeof value; + + if(type_merge !== "undefined"){ + if(type_value !== "undefined"){ + + if(merge_value){ + if(type_value === "function" && + type_merge === type_value){ + return function(str){ + return /** @type Function */ (value)( + /** @type Function */ (merge_value)(str) + ); + } + } + + const constructor_value = value.constructor; + const constructor_merge = merge_value.constructor; + + if(constructor_value === constructor_merge){ + + if(constructor_value === Array){ + return merge_value.concat(value); + } + + if(constructor_value === Map){ + const map = new Map(/** @type !Map */ (merge_value)); + for(const item of /** @type !Map */ (value)){ + const key = item[0]; + const val = item[1]; + map.set(key, val); + } + return map; + } + + if(constructor_value === Set){ + const set = new Set(/** @type !Set */ (merge_value)); + for(const val of /** @type !Set */ (value).values()){ + set.add(val); + } + return set; + } + } + } + + return value; + } + else{ + return merge_value; + } + } + + return type_value === "undefined" + ? default_value + : value; } /** @@ -13,19 +74,33 @@ export function create_object_array(count){ const array = new Array(count); for(let i = 0; i < count; i++){ - array[i] = create_object(); } return array; } +/** + * @param {!number} count + * @returns {Array} + */ + +export function create_map_array(count){ + + const array = new Array(count); + + for(let i = 0; i < count; i++){ + array[i] = new Map(); + } + + return array; +} + export function create_arrays(count){ const array = new Array(count); for(let i = 0; i < count; i++){ - array[i] = []; } @@ -38,41 +113,76 @@ export function create_arrays(count){ */ export function get_keys(obj){ - return Object.keys(obj); } export function create_object(){ - return Object.create(null); } export function concat(arrays){ - return [].concat.apply([], arrays); } export function sort_by_length_down(a, b){ - return b.length - a.length; } -export function is_array(val){ +export function sort_by_length_up(a, b){ + return a.length - b.length; +} +export function is_array(val){ return val.constructor === Array; } export function is_string(val){ - return typeof val === "string"; } export function is_object(val){ - return typeof val === "object"; } export function is_function(val){ - return typeof val === "function"; } + +/** + * @param {Map|Set} val + * @param {boolean=} stringify + * @return {Array} + */ + +export function toArray(val, stringify){ + const result = []; + for(const key of val.keys()){ + result.push(stringify ? "" + key : key); + } + return result; +} + +// TODO support generic function created from string when tree depth > 1 +export function parse_simple(obj, tree){ + + if(is_string(tree)){ + obj = obj[tree]; + } + else for(let i = 0; obj && (i < tree.length); i++){ + obj = obj[tree[i]]; + } + + return obj; +} + +export function get_max_len(arr){ + let len = 0; + for(let i = 0, res; i < arr.length; i++){ + if((res = arr[i])){ + if(len < res.length){ + len = res.length; + } + } + } + return len; +} \ No newline at end of file diff --git a/src/compress.js b/src/compress.js new file mode 100644 index 0000000..37fb20f --- /dev/null +++ b/src/compress.js @@ -0,0 +1,78 @@ +// COMPILER BLOCK --> +import { SUPPORT_CACHE } from "./config.js"; +// <-- COMPILER BLOCK + +let table;// = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz+/"; +let timer; +const cache = new Map(); + +function toRadix(number, radix = 255) { + + if(!table){ + table = new Array(radix); + // the char code 0 could be a special marker + for(let i = 0; i < radix; i++) table[i] = i + 1; + table = String.fromCharCode.apply(null, table); + } + + let rixit; + let residual = number; + let result = ""; + + while(true){ + rixit = residual % radix; + result = table.charAt(rixit) + result; + residual = residual / radix | 0; + if(!residual) + break; + } + + return result; +} + +export default function(str){ + + if(SUPPORT_CACHE){ + if(timer){ + if(cache.has(str)){ + return cache.get(str); + } + } + else{ + timer = setTimeout(clear); + } + } + + /* 2 ** ((level + 1.5) * 1.6 | 0) */ + + const result = toRadix( + typeof str == "number" + ? str + : lcg(str) + ); + + if(SUPPORT_CACHE){ + cache.size > 2e5 && cache.clear(); + cache.set(str, result); + } + + return result; +} + +function lcg(str) { + let range = 2 ** 32 - 1; + if(typeof str == "number"){ + return str & range; + } + let crc = 0, bit = 32 + 1; + for(let i = 0; i < str.length; i++) { + crc = (crc * bit ^ str.charCodeAt(i)) & range; + } + // shift up from Int32 to UInt32 range 0xFFFFFFFF + return crc + 2 ** 31; +} + +function clear(){ + timer = null; + cache.clear(); +} diff --git a/src/config.js b/src/config.js index 2a7ad21..844553d 100644 --- a/src/config.js +++ b/src/config.js @@ -2,7 +2,10 @@ export const RELEASE = "custom"; /** @define {boolean} */ -export const DEBUG = false; +export const DEBUG = true; + +/** @define {boolean} */ +export const PROFILER = true; /** @define {boolean} */ export const POLYFILL = true; @@ -22,9 +25,6 @@ export const SUPPORT_ASYNC = true; /** @define {boolean} */ export const SUPPORT_STORE = true; -/** @define {boolean} */ -export const SUPPORT_TAGS = true; - /** @define {boolean} */ export const SUPPORT_SUGGESTION = true; @@ -33,3 +33,18 @@ export const SUPPORT_SERIALIZE = true; /** @define {boolean} */ export const SUPPORT_DOCUMENT = true; + +/** @define {boolean} */ +export const SUPPORT_TAGS = true; + +/** @define {boolean} */ +export const SUPPORT_PERSISTENT = true; + +/** @define {boolean} */ +export const SUPPORT_KEYSTORE = true; + +/** @define {boolean} */ +export const SUPPORT_COMPRESSION = true; + +/** @define {boolean} */ +export const SUPPORT_RESOLVER = true; \ No newline at end of file diff --git a/src/db/clickhouse/README.md b/src/db/clickhouse/README.md new file mode 100644 index 0000000..deaf9bf --- /dev/null +++ b/src/db/clickhouse/README.md @@ -0,0 +1,112 @@ +# Clickhouse FlexSearch + +Clickhouse is highly optimized for raw I/O and write access. When your index needs to be frequently updated, this storage engine can help to reduce blocking write access. + +You'll need to install the npm package `clickhouse` into your project: +```bash +npm install clickhouse@2.6.0 +``` + +Create an index and assign a Clickhouse storage adapter to it by using `index.mount(db)`: + +```js +import Index from "./index.js"; +import Database from "./db/clickhouse/index.js"; + +// your database configuration +const config = { + host: "http://localhost", + port: "8123", + database: "default", + basicAuth: null, + debug: false +}; + +// create an index +const index = new Index(); +// create db instance with optional prefix +const db = new Database("my-store", config); +// mount and await before transfering data +await index.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +> Changes are automatically committed by default when you need to wait for the task completion, then you can use the `await index.commit()` method explicitely. You can disable the auto-commit feature optionally. + +## Configuration + +### Custom DB Instance + +Pass a valid `clickhouse` instance on creation: + +```js +import { ClickHouse } from "clickhouse"; +import Database from "./db/clickhouse/index.js"; +// assume you've created a custom database instance... +const database = new ClickHouse({/* config */}); +// pass database instance as option +const db = new Database("my-store", { + db: database +}); +``` + +For every instance of `DocumentIndex`, `WorkerIndex` or `Index` (as standalone) you'll need to create a `Database` instance having its own name. + +### ID Type + +The Clickhouse driver does not properly support upgrading a merge key by ALTER TABLE. Therefore, the default type for ID is `text`. + +You will save required disk space and also gain performance when define a numeric ID type explicitly. + +```js +import Database from "./db/clickhouse/index.js"; +// force integer type +const db = new Database("my-store", { + type: "integer" +}); +// .... +index.add("15712", "content..."); +// IDs will cast to integer +let result = await index.search("content..."); +// -> [15712] +``` + +BigInt: + +```js +const db = new Database("my-store", { + type: "bigint" +}); +``` + +To change the type later the index needs to be deleted by `db.destroy()` and re-created by `db.mount()`. + +### Table Structure + +FlexSearch is creating different `DATABASE` entries for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively. + +``` +DATABASE + |__TABLE map:field (FlexSearch Data) + |__TABLE ctx:field (FlexSearch Data) + |__TABLE tag:field (FlexSearch Data) + |__TABLE cfg:field (FlexSearch Data) + |__TABLE reg (FlexSearch Data) +``` + +The default DATABASE in Clickhouse is commonly named `default` and is required to make the first connection for creating the new databases. You can force using a specific default DATABASE by option: + +```js +const db = new Database({ + database: "public" +}); +``` diff --git a/src/db/clickhouse/index.js b/src/db/clickhouse/index.js new file mode 100644 index 0000000..224231a --- /dev/null +++ b/src/db/clickhouse/index.js @@ -0,0 +1,714 @@ +// COMPILER BLOCK --> +import { DEBUG } from "../../config.js"; +// <-- COMPILER BLOCK +import { ClickHouse } from "clickhouse"; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { concat, toArray } from "../../common.js"; +const defaults = { + host: "http://localhost", + port: "8123", + debug: false, + basicAuth: null, + isUseGzip: false, + trimQuery: false, + usePost: false, + format: "json", + raw: false, + config: { + output_format_json_quote_64bit_integers: 0, + enable_http_compression: 0, + database: "default" + } +}; + +const VERSION = 1; +const fields = ["map", "ctx", "tag", "reg", "cfg"]; +const types = { + "text": "String", + "char": "String", + "varchar": "String", + "string": "String", + "number": "Int32", + "numeric": "Int32", + "integer": "Int32", + "smallint": "Int16", + "tinyint": "Int8", + "mediumint": "Int32", + "int": "Int32", + "int8": "Int8", + "uint8": "UInt8", + "int16": "Int16", + "uint16": "UInt16", + "int32": "Int32", + "uint32": "UInt32", + "int64": "Int64", + "uint64": "UInt64", + "bigint": "Int64" +}; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +let DB; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function ClickhouseDB(name, config = {}){ + if(!(this instanceof ClickhouseDB)){ + return new ClickhouseDB(name, config); + } + if(typeof name === "object"){ + name = name.name; + config = name; + } + if(!name){ + console.info("Default storage space was used, because a name was not passed."); + } + //field = "Test-456"; + this.id = "flexsearch" + (name ? "_" + sanitize(name) : ""); + this.field = config.field ? "_" + sanitize(config.field) : ""; + // Clickhouse does not support ALTER TABLE to upgrade + // the type of the ID when it is a part of the merge key + this.type = config.type ? types[config.type.toLowerCase()] : "String"; + if(!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); + //this.trx = false; + this.support_tag_search = true; + this.db = DB || (DB = config.db || null); + Object.assign(defaults, config); + config.database && (defaults.config.database = config.database); + this.db && delete defaults.db; +}; + +ClickhouseDB.prototype.mount = function(flexsearch){ + if(flexsearch instanceof Document){ + return flexsearch.mount(this); + } + defaults.resolution = Math.max(flexsearch.resolution, flexsearch.resolution_ctx); + flexsearch.db = this; + return this.open(); +}; + +ClickhouseDB.prototype.open = async function(){ + + if(!this.db) { + this.db = DB || ( + DB = new ClickHouse(defaults) + ); + } + + const exists = await this.db.query(` + SELECT 1 FROM system.databases WHERE name = '${this.id}'; + `).toPromise(); + + if(!exists || !exists.length){ + await this.db.query(` + CREATE DATABASE IF NOT EXISTS ${this.id}; + `).toPromise(); + } + + for(let i = 0; i < fields.length; i++){ + switch(fields[i]){ + case "map": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.map${this.field}( + key String, + res ${defaults.resolution <= 255 ? "UInt8" : "UInt16"}, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (key)*/ + ORDER BY (key, id); + `, { params: { name: this.id + ".map" + this.field }}).toPromise(); + break; + + case "ctx": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.ctx${this.field}( + ctx String, + key String, + res ${defaults.resolution <= 255 ? "UInt8" : "UInt16"}, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (ctx, key)*/ + ORDER BY (ctx, key, id); + `).toPromise(); + break; + + case "tag": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.tag${this.field}( + tag String, + id ${this.type} + ) + ENGINE = MergeTree + /*PRIMARY KEY (ctx, key)*/ + ORDER BY (tag, id); + `).toPromise(); + break; + + case "reg": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.reg( + id ${this.type}, + doc Nullable(String) + ) + ENGINE = MergeTree + ORDER BY (id); + `).toPromise(); + break; + + case "cfg": + await this.db.query(` + CREATE TABLE IF NOT EXISTS ${this.id}.cfg${this.field}( + cfg String + ) + ENGINE = TinyLog; + `).toPromise(); + break; + } + } + + return this.db; +}; + +ClickhouseDB.prototype.close = function(){ + this.db.close(); + this.db = null; + return this; +}; + +ClickhouseDB.prototype.destroy = async function(){ + await Promise.all([ + this.db.query(`DROP TABLE ${this.id}.map${this.field};`).toPromise(), + this.db.query(`DROP TABLE ${this.id}.ctx${this.field};`).toPromise(), + this.db.query(`DROP TABLE ${this.id}.tag${this.field};`).toPromise(), + this.db.query(`DROP TABLE ${this.id}.cfg${this.field};`).toPromise(), + this.db.query(`DROP TABLE ${this.id}.reg;`).toPromise() + ]); + this.close(); +}; + +ClickhouseDB.prototype.clear = function(){ + return Promise.all([ + this.db.query(`TRUNCATE TABLE ${this.id}.map${this.field};`).toPromise(), + this.db.query(`TRUNCATE TABLE ${this.id}.ctx${this.field};`).toPromise(), + this.db.query(`TRUNCATE TABLE ${this.id}.tag${this.field};`).toPromise(), + this.db.query(`TRUNCATE TABLE ${this.id}.cfg${this.field};`).toPromise(), + this.db.query(`TRUNCATE TABLE ${this.id}.reg;`).toPromise() + ]); +}; + +function create_result(rows, resolve, enrich){ + if(resolve){ + for(let i = 0; i < rows.length; i++){ + if(enrich){ + if(rows[i].doc){ + rows[i].doc = JSON.parse(rows[i].doc); + } + } + else{ + rows[i] = rows[i].id; + } + } + return rows; + } + else{ + const arr = []; + for(let i = 0, row; i < rows.length; i++){ + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich + ? row + : row.id + ); + } + return arr; + } +} + +ClickhouseDB.prototype.get = function(key, ctx, limit = 0, offset = 0, resolve = true, enrich = false, tags){ + let rows; + let stmt = ''; + let params = ctx ? { ctx, key } : { key }; + let table = this.id + (ctx ? ".ctx" : ".map") + this.field; + if(tags){ + for(let i = 0, count = 1; i < tags.length; i+=2){ + stmt += ` AND ${ table }.id IN (SELECT id FROM ${ this.id }.tag_${ sanitize(tags[i]) } WHERE tag = {tag${ count }:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + if(ctx){ + rows = this.db.query(` + SELECT ${ table }.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT OUTER JOIN ${ this.id }.reg ON ${ this.id }.reg.id = ${ table }.id + ` : "" } + WHERE ctx = {ctx:String} AND key = {key:String} + ORDER BY res + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" }`, + { params } + ).toPromise(); + } + else{ + rows = this.db.query(` + SELECT ${ table }.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT OUTER JOIN ${ this.id }.reg ON ${ this.id }.reg.id = ${ table }.id + ` : "" } + WHERE key = {key:String} + ORDER BY res + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" }`, + { params } + ).toPromise(); + } + return rows.then(function(rows){ + return create_result(rows, resolve, enrich); + }); +}; + +ClickhouseDB.prototype.tag = function(tag, limit = 0, offset = 0, enrich = false){ + const table = this.id + ".tag" + this.field; + const promise = this.db.query(` + SELECT ${ table }.id + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT OUTER JOIN ${ this.id }.reg ON ${ this.id }.reg.id = ${ table }.id + ` : "" } + WHERE tag = {tag:String} + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" }`, + { params: { tag } } + ).toPromise(); + enrich || promise.then(function(rows){ + return create_result(rows, true, false); + }); + return promise; +} + +ClickhouseDB.prototype.enrich = async function(ids){ + let MAXIMUM_QUERY_VARS = 1e5; + let result = []; + if(typeof ids !== "object"){ + ids = [ids]; + } + for(let count = 0; count < ids.length;){ + const chunk = ids.length - count > MAXIMUM_QUERY_VARS + ? ids.slice(count, count + MAXIMUM_QUERY_VARS) + : count ? ids.slice(count) : ids; + count += chunk.length; + let params = {}; + let stmt = ""; + for(let i = 0; i < chunk.length; i++){ + stmt += (stmt ? "," : "") + "{id" + (i + 1) + ":String}"; + params["id" + (i + 1)] = chunk[i]; + } + const res = await this.db.query(` + SELECT id, doc + FROM ${ this.id }.reg + WHERE id IN (${ stmt })`, + { params } + ).toPromise(); + if(res && res.length){ + for(let i = 0, doc; i < res.length; i++){ + if((doc = res[i].doc)){ + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + return result.length === 1 + ? result[0] + : result.length > 1 + ? concat(result) + : result; +} + +ClickhouseDB.prototype.has = function(id){ + return this.db.query(` + SELECT EXISTS( + SELECT 1 + FROM ${this.id}.reg + WHERE id = {id:${this.type /*=== "number" ? "Int32" : "String"*/}} + LIMIT 1 + )`, + { params: { id }} + ).toPromise(); +}; + +ClickhouseDB.prototype.search = function(flexsearch, query, limit = 100, offset = 0, suggest = false, resolve = true, enrich = true, tags){ + let rows; + if(query.length > 1 && flexsearch.depth){ + + let where = ""; + let params = {}; + let keyword = query[0]; + let term; + + for(let i = 1; i < query.length; i++){ + term = query[i]; + const swap = flexsearch.bidirectional && (term > keyword); + where += (where ? " OR " : "") + `(ctx = {ctx${ i }:String} AND key = {key${ i }:String})` + params["ctx" + i] = swap ? term : keyword; + params["key" + i] = swap ? keyword : term; + keyword = term; + } + + if(tags){ + where = "(" + where + ")"; + for(let i = 0, count = 1; i < tags.length; i+=2){ + where += ` AND id IN (SELECT id FROM ${ this.id }.tag_${ sanitize(tags[i]) } WHERE tag = {tag${ count }:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + + rows = this.db.query(` + SELECT r.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ( + SELECT id, count(*) as count, + ${ suggest ? "SUM" : "MIN" }(res) as res + FROM ${ this.id }.ctx${ this.field } + WHERE ${ where } + GROUP BY id + ) as r + ${ enrich ? ` + LEFT OUTER JOIN ${ this.id }.reg ON ${ this.id }.reg.id = r.id + ` : "" } + ${ suggest ? "" : "WHERE count = " + (query.length - 1) } + ORDER BY ${ suggest ? "count DESC, res" : "res" } + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, { params }).toPromise(); + + // for(let i = 1; i < query.length; i++){ + // where += (where ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM ${this.id}.ctx${this.field} + // WHERE ctx = {ctx${i}:String} AND key = {key${i}:String} + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params["ctx" + i] = swap ? term : keyword; + // params["key" + i] = swap ? keyword : term; + // keyword = term; + // } + // + // rows = await this.db.query(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, { params }).toPromise(); + } + else{ + + let where = ""; + let params = {}; + + for(let i = 0; i < query.length; i++){ + where += (where ? "," : "") + `{key${i}:String}`; + params["key" + i] = query[i]; + } + where = "key " + (query.length > 1 ? "IN (" + where + ")" : "= " + where ); + + if(tags){ + where = "(" + where + ")"; + for(let i = 0, count = 1; i < tags.length; i+=2){ + where += ` AND id IN (SELECT id FROM ${ this.id }.tag_${ sanitize(tags[i]) } WHERE tag = {tag${ count }:String})`; + params["tag" + count] = tags[i + 1]; + count++; + } + } + + rows = this.db.query(` + SELECT r.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ( + SELECT id, count(*) as count, + ${ suggest ? "SUM" : "MIN" }(res) as res + FROM ${ this.id }.map${ this.field } + WHERE ${ where } + GROUP BY id + ) as r + ${ enrich ? ` + LEFT OUTER JOIN ${ this.id }.reg ON ${ this.id }.reg.id = r.id + ` : "" } + ${ suggest ? "" : "WHERE count = " + query.length } + ORDER BY ${ suggest ? "count DESC, res" : "res" } + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, { params }).toPromise(); + + // for(let i = 0; i < query.length; i++){ + // params["key" + i] = query[i]; + // where += (where ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM ${ this.id }.map${ this.field } + // WHERE key = {key${i}:String} + // `; + // } + // rows = await this.db.query(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, { params }).toPromise(); + } + return rows.then(function(rows){ + return create_result(rows, resolve, enrich); + }); +} + +ClickhouseDB.prototype.info = function(){ + // todo +}; + +ClickhouseDB.prototype.transaction = function(task){ + + // not supported + return task.call(this); +}; + +ClickhouseDB.prototype.commit = async function(flexsearch, _replace, _append){ + + // process cleanup tasks + if(_replace){ + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } + else{ + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for(let i = 0, task; i < tasks.length; i++){ + task = tasks[i]; + // there are just removals in the task queue + if(task.clear){ + await this.clear(); + _replace = true; + break; + } + else{ + tasks[i] = task.del; + } + } + if(!_replace){ + if(!_append){ + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && await this.remove(tasks); + } + } + if(!flexsearch.reg.size){ + return; + } + + if(flexsearch.map.size){ + let data = []; + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + //this.type || (this.type = typeof ids[0]); + for(let j = 0; j < ids.length; j++){ + data.push({ + key: key, + res: i, + id: /*this.type === "number" + ? parseInt(ids[j], 10) + :*/ ids[j] + }); + } + } + } + } + if(data.length){ + await this.db.insert( + `INSERT INTO ${ this.id }.map${ this.field } (key, res, id)`, data + ).toPromise(); + } + } + + if(flexsearch.ctx.size){ + let data = []; + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + for(let j = 0; j < ids.length; j++){ + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: /*this.type === "number" + ? parseInt(ids[j], 10) + :*/ ids[j] + }); + } + } + } + } + } + if(data.length){ + await this.db.insert( + `INSERT INTO ${ this.id }.ctx${ this.field } (ctx, key, res, id)`, data + ).toPromise(); + } + } + + if(flexsearch.tag){ + let data = []; + for(const item of flexsearch.tag){ + const tag = item[0]; + const ids = item[1]; + if(!ids.length) continue; + for(let j = 0; j < ids.length; j++){ + data.push({ tag, id: ids[j] }); + } + } + if(data.length){ + await this.db.insert( + `INSERT INTO ${this.id}.tag${ this.field } (tag, id)`, data + ).toPromise(); + } + } + + if(flexsearch.store){ + let data = []; + for(const item of flexsearch.store.entries()){ + const id = item[0]; + const doc = item[1]; + data.push({ id, doc: doc && JSON.stringify(doc) }); + } + if(data.length){ + await this.db.insert( + `INSERT INTO ${this.id}.reg (id, doc)`, data + ).toPromise(); + } + } + else if(!flexsearch.bypass){ + let data = toArray(flexsearch.reg); + for(let i = 0; i < data.length; i++){ + data[i] = { id: data[i] }; + } + if(data.length){ + await this.db.insert( + `INSERT INTO ${this.id}.reg (id)`, data + ).toPromise(); + } + } + + // TODO + // await this.db.insert(`INSERT INTO ${this.id}.cfg${this.field} (cfg)`, [{ + // cfg: JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }) + // }]).toPromise(); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && + flexsearch.tag.clear(); + flexsearch.store && + flexsearch.store.clear(); + flexsearch.document || + flexsearch.reg.clear(); + + await Promise.all([ + this.db.query(`OPTIMIZE TABLE ${this.id}.map${this.field} FINAL`).toPromise(), + this.db.query(`OPTIMIZE TABLE ${this.id}.ctx${this.field} FINAL`).toPromise(), + this.db.query(`OPTIMIZE TABLE ${this.id}.tag${this.field} FINAL`).toPromise(), + this.db.query(`OPTIMIZE TABLE ${this.id}.reg FINAL`).toPromise() + ]); +}; + +ClickhouseDB.prototype.remove = async function(ids){ + + if(typeof ids !== "object"){ + ids = [ids]; + } + + while(ids.length){ + + let chunk = ids.slice(0, 1e5); + ids = ids.slice(1e5); + chunk = this.type === "String" + ? "'" + chunk.join("','") + "'" + : chunk.join(","); + + await Promise.all([ + this.db.query(` + ALTER TABLE ${this.id}.map${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;` + ).toPromise(), + + this.db.query(` + ALTER TABLE ${this.id}.ctx${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;` + ).toPromise(), + + this.db.query(` + ALTER TABLE ${this.id}.tag${this.field} + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;` + ).toPromise(), + + this.db.query(` + ALTER TABLE ${this.id}.reg + DELETE WHERE id IN (${chunk}) + SETTINGS mutations_sync = 1;` + ).toPromise() + ]); + } +}; diff --git a/src/db/clickhouse/package.json b/src/db/clickhouse/package.json new file mode 100644 index 0000000..3bbde1b --- /dev/null +++ b/src/db/clickhouse/package.json @@ -0,0 +1,16 @@ +{ + "public": true, + "preferGlobal": false, + "name": "flexsearch-clickhouse", + "version": "0.1.0", + "main": "index.js", + "scripts": {}, + "files": [ + "index.js", + "README.md", + "LICENSE" + ], + "dependencies": { + "clickhouse": "^2.6.0" + } +} diff --git a/src/db/indexeddb/README.md b/src/db/indexeddb/README.md new file mode 100644 index 0000000..9677e6c --- /dev/null +++ b/src/db/indexeddb/README.md @@ -0,0 +1,41 @@ +# IndexedDB FlexSearch + +IndexedDB is a persistent storage supported by all major browsers. + +Create an index and assign a IndexedDB storage adapter to it by using `index.mount(db)`: + +```js +import Index from "./index.js"; +import Database from "./db/indexeddb/index.js"; +// create an index +const index = new Index(); +// create db instance with optional namespace +const db = new Database("my-store"); +// mount and await before transfering data +await index.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +> Changes are automatically committed by default when you need to wait for the task completion, then you can use the `await index.commit()` method explicitely. You can disable the auto-commit feature optionally. + +### Table Structure + +FlexSearch is creating different `DATABASE` entries for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively. + +``` +DATABASE + |__OBJECTSTORE map:field (FlexSearch Data) + |__OBJECTSTORE ctx:field (FlexSearch Data) + |__OBJECTSTORE tag:field (FlexSearch Data) + |__OBJECTSTORE cfg:field (FlexSearch Data) + |__OBJECTSTORE reg (FlexSearch Data) +``` diff --git a/src/db/indexeddb/index.js b/src/db/indexeddb/index.js new file mode 100644 index 0000000..22abc63 --- /dev/null +++ b/src/db/indexeddb/index.js @@ -0,0 +1,631 @@ +// COMPILER BLOCK --> +import { + DEBUG, + SUPPORT_STORE, + SUPPORT_TAGS +} from "../../config.js"; +// <-- COMPILER BLOCK +import Document from "../../document.js"; + +const VERSION = 1; +const IndexedDB = typeof window !== "undefined" && ( + window.indexedDB || + window.mozIndexedDB || + window.webkitIndexedDB || + window.msIndexedDB +); +const IDBTransaction = typeof window !== "undefined" && ( + window.IDBTransaction || + window.webkitIDBTransaction || + window.msIDBTransaction +); +const IDBKeyRange = typeof window !== "undefined" && ( + window.IDBKeyRange || + window.webkitIDBKeyRange || + window.msIDBKeyRange +); +const fields = ["map", "ctx", "tag", "reg", "cfg"]; +import StorageInterface from "../interface.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +/** + * @constructor + * @implements StorageInterface + */ + +export default function IdxDB(name, config = {}){ + if(!(this instanceof IdxDB)){ + return new IdxDB(name, config); + } + if(typeof name === "object"){ + name = name.name; + config = name; + } + if(!name){ + console.info("Default storage space was used, because a name was not passed."); + } + this.id = "flexsearch" + (name ? ":" + sanitize(name) : ""); + this.field = config.field ? sanitize(config.field) : ""; + this.support_tag_search = false; + this.db = null; + this.trx = {}; +}; + +IdxDB.prototype.mount = function(flexsearch){ + if(flexsearch instanceof Document){ + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +IdxDB.prototype.open = function(){ + + let self = this; + + navigator.storage && + navigator.storage.persist(); + + return this.db || new Promise(function(resolve, reject){ + + const req = IndexedDB.open(self.id + (self.field ? ":" + self.field : ""), VERSION); + + req.onupgradeneeded = function(event){ + + const db = self.db = this.result; + + // Using Indexes + IDBKeyRange on schema map => [key, res, id] performs + // too bad and blows up amazingly in size + // The schema map:key => [res][id] is currently used instead + // In fact that bypass the idea of a storage solution, + // IndexedDB is such a poor contribution :( + + fields.forEach(ref => { + db.objectStoreNames.contains(ref) || + db.createObjectStore(ref);//{ autoIncrement: true /*keyPath: "id"*/ } + //.createIndex("idx", "ids", { multiEntry: true, unique: false }); + }); + + // switch(event.oldVersion){ // existing db version + // case 0: + // // version 0 means that the client had no database + // // perform initialization + // case 1: + // // client had version 1 + // // update + // } + }; + + req.onblocked = function(event) { + // this event shouldn't trigger if we handle onversionchange correctly + // it means that there's another open connection to the same database + // and it wasn't closed after db.onversionchange triggered for it + console.error("blocked", event); + reject(); + }; + + req.onerror = function(event){ + console.error(this.error, event); + reject(); + }; + + req.onsuccess = function(event){ + self.db = this.result; //event.target.result; + self.db.onversionchange = function(){ + //database is outdated + self.close(); + }; + resolve(self); + }; + }); +}; + +IdxDB.prototype.close = function(){ + this.db.close(); + this.db = null; +}; + +IdxDB.prototype.destroy = function(){ + this.db && this.close(); + return IndexedDB.deleteDatabase(this.id + (this.field ? ":" + this.field : "")); +}; + +// IdxDB.prototype.set = function(ref, key, ctx, data){ +// const transaction = this.db.transaction(ref, "readwrite"); +// const map = transaction.objectStore(ref); +// const req = map.put(data, ctx ? ctx + ":" + key : key); +// return transaction;//promisfy(req, callback); +// }; + +// IdxDB.prototype.delete = function(ref, key, ctx){ +// const transaction = this.db.transaction(ref, "readwrite"); +// const map = transaction.objectStore(ref); +// const req = map.delete(ctx ? ctx + ":" + key : key); +// return transaction;//promisfy(req, callback); +// }; + +IdxDB.prototype.clear = function(){ + const transaction = this.db.transaction(fields, "readwrite"); + for(let i = 0; i < fields.length; i++){ + transaction.objectStore(fields[i]).clear(); + } + return promisfy(transaction); +}; + +IdxDB.prototype.get = function(key, ctx, limit = 0, offset = 0, resolve = true, enrich = false){ + const transaction = this.db.transaction(ctx ? "ctx" : "map", "readonly"); + const map = transaction.objectStore(ctx ? "ctx" : "map"); + const req = map.get(ctx ? ctx + ":" + key : key); + const self = this; + return promisfy(req).then(function(res){ + let result = []; + if(!res || !res.length) return result; + if(resolve){ + if(!limit && !offset && res.length === 1){ + return res[0]; + } + for(let i = 0, arr; i < res.length; i++){ + if((arr = res[i]) && arr.length){ + if(offset >= arr.length){ + offset -= arr.length; + continue; + } + const end = limit + ? offset + Math.min(arr.length - offset, limit) + : arr.length; + for(let j = offset; j < end; j++){ + result.push(arr[j]); + } + offset = 0; + if(result.length === limit){ + break; + } + } + } + return SUPPORT_STORE && enrich + ? self.enrich(result) + : result; + } + else{ + return res; + } + }); +}; + +if(SUPPORT_TAGS){ + + IdxDB.prototype.tag = function(tag, limit = 0, offset = 0, enrich = false){ + const transaction = this.db.transaction("tag", "readonly"); + const map = transaction.objectStore("tag"); + const req = map.get(tag); + const self = this; + return promisfy(req).then(function(ids){ + if(!ids || !ids.length || offset >= ids.length) return []; + if(!limit && !offset) return ids; + const result = ids.slice(offset, offset + limit); + return SUPPORT_STORE && enrich + ? self.enrich(result) + : result; + }); + }; +} + +if(SUPPORT_STORE){ + + IdxDB.prototype.enrich = function(ids){ + if(typeof ids !== "object"){ + ids = [ids]; + } + const transaction = this.db.transaction("reg", "readonly"); + const map = transaction.objectStore("reg"); + const promises = []; + for(let i = 0; i < ids.length; i++){ + promises[i] = promisfy(map.get(ids[i])); + } + return Promise.all(promises).then(function(docs){ + for(let i = 0; i < docs.length; i++){ + docs[i] = { + id: ids[i], + doc: docs[i] ? JSON.parse(docs[i]) : null + }; + } + return docs; + }); + }; +} + +IdxDB.prototype.has = function(id){ + const transaction = this.db.transaction("reg", "readonly"); + const map = transaction.objectStore("reg"); + const req = map.getKey(id); + return promisfy(req); +}; + +IdxDB.prototype.search = null; + +// IdxDB.prototype.has = function(ref, key, ctx){ +// const transaction = this.db.transaction(ref, "readonly"); +// const map = transaction.objectStore(ref); +// const req = map.getKey(ctx ? ctx + ":" + key : key); +// return promisfy(req); +// }; + +IdxDB.prototype.info = function(){ + // todo +}; + +/** + * @param {!string} ref + * @param {!string} modifier + * @param {!Function} task + */ + +IdxDB.prototype.transaction = function(ref, modifier, task){ + + let store = this.trx[ref + ":" + modifier]; + if(store) return task.call(this, store); + + let transaction = this.db.transaction(ref, modifier); + this.trx[ref + ":" + modifier] = store = transaction.objectStore(ref); + + return new Promise((resolve, reject) => { + transaction.onerror = (err) => { + this.trx[ref+ ":" + modifier] = null; + transaction.abort(); + transaction = store = null; + reject(err); + //db.close; + }; + transaction.oncomplete = (res) => { + this.trx[ref+ ":" + modifier] = null; + transaction = store = null; + resolve(res || true); + //db.close; + }; + return task.call(this, store); + }); +}; + +IdxDB.prototype.commit = async function(flexsearch, _replace, _append){ + + // process cleanup tasks + if(_replace){ + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } + else{ + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for(let i = 0, task; i < tasks.length; i++){ + task = tasks[i]; + // there are just removals in the task queue + if(task.clear){ + await this.clear(); + _replace = true; + break; + } + else{ + tasks[i] = task.del; + } + } + if(!_replace){ + if(!_append){ + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && await this.remove(tasks); + } + } + + if(!flexsearch.reg.size){ + return; + } + + await this.transaction("map", "readwrite", function(store){ + + for(const item of flexsearch.map){ + + const key = item[0]; + const value = item[1]; + if(!value.length) continue; + + if(_replace){ + store.put(value, key); + continue; + } + + store.get(key).onsuccess = function(){ + let result = this.result; + let changed; + if(result && result.length){ + const maxlen = Math.max(result.length, value.length); + for(let i = 0, res, val; i < maxlen; i++){ + val = value[i]; + if(val && val.length){ + res = result[i]; + if(res && res.length){ + for(let j = 0; j < val.length; j++){ + res.push(val[j]); + } + changed = 1; + //result[i] = res.concat(val); + //result[i] = new Set([...result[i], ...value[i]]); + //result[i] = result[i].union(new Set(value[i])); + } + else{ + result[i] = val; + changed = 1; + //result[i] = new Set(value[i]) + } + } + } + } + else{ + result = value; + changed = 1; + //result = []; + //for(let i = 0; i < value.length; i++){ + // if(value[i]) result[i] = new Set(value[i]); + //} + } + + changed && + store.put(result, key); + } + } + }); + + await this.transaction("ctx", "readwrite", function(store){ + + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + + for(const item of ctx_value){ + const key = item[0]; + const value = item[1]; + if(!value.length) continue; + + if(_replace){ + store.put(value, ctx_key + ":" + key); + continue; + } + + store.get(ctx_key + ":" + key).onsuccess = function(){ + let result = this.result; + let changed; + if(result && result.length){ + const maxlen = Math.max(result.length, value.length); + for(let i = 0, res, val; i < maxlen; i++){ + val = value[i]; + if(val && val.length){ + res = result[i]; + if(res && res.length){ + for(let j = 0; j < val.length; j++){ + res.push(val[j]); + } + //result[i] = res.concat(val); + changed = 1; + } + else{ + result[i] = val; + changed = 1; + } + } + } + } + else{ + result = value; + changed = 1; + } + + changed && + store.put(result, ctx_key + ":" + key); + } + } + } + }); + + if(SUPPORT_STORE && flexsearch.store){ + await this.transaction("reg", "readwrite", function(store){ + for(const item of flexsearch.store){ + const id = item[0]; + const doc = item[1]; + store.put(typeof doc === "object" + ? JSON.stringify(doc) + : 1 + , id); + } + }); + } + else if(!flexsearch.bypass){ + await this.transaction("reg", "readwrite", function(store){ + for(const id of flexsearch.reg.keys()){ + store.put(1, id); + } + }); + } + + if(SUPPORT_TAGS && flexsearch.tag){ + await this.transaction("tag", "readwrite", function(store){ + for(const item of flexsearch.tag){ + const tag = item[0]; + const ids = item[1]; + if(!ids.length) continue; + + store.get(tag).onsuccess = function(){ + let result = this.result; + result = result && result.length + ? result.concat(ids) + : ids; + store.put(result, tag); + } + } + }); + } + + // TODO + // await this.transaction("cfg", "readwrite", function(store){ + // store.put({ + // "charset": flexsearch.charset, + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "fastupdate": flexsearch.fastupdate, + // "compress": flexsearch.compress, + // "encoder": { + // "minlength": flexsearch.encoder.minlength + // }, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }, "current"); + // }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + if(SUPPORT_TAGS){ + flexsearch.tag && + flexsearch.tag.clear(); + } + if(SUPPORT_STORE){ + flexsearch.store && + flexsearch.store.clear(); + } + flexsearch.document || + flexsearch.reg.clear(); +}; + +/** + * @param {IDBCursorWithValue} cursor + * @param {Array} ids + * @param {boolean=} _tag + */ + +function handle(cursor, ids, _tag){ + + const arr = cursor.value; + let changed; + let parse; + let count = 0; + + for(let x = 0, result; x < arr.length; x++){ + // tags has no resolution layer + if((result = _tag ? arr : arr[x])){ + for(let i = 0, pos, id; i < ids.length; i++){ + id = ids[i]; + pos = result.indexOf(parse ? parseInt(id, 10) : id); + if(pos < 0 && !parse && typeof id === "string" && !isNaN(id)){ + pos = result.indexOf(parseInt(id, 10)); + pos && (parse = 1); + } + if(pos >= 0){ + changed = 1; + if(result.length > 1){ + result.splice(pos, 1); + } + else{ + arr[x] = []; + break; + } + } + } + + count += result.length; + } + if(_tag) break; + } + + if(!count){ + + cursor.delete(); + //store.delete(cursor.key); + } + else if(changed){ + + //await new Promise(resolve => { + cursor.update(arr);//.onsuccess = resolve; + //}); + } + + cursor.continue(); +} + +IdxDB.prototype.remove = function(ids){ + + if(typeof ids !== "object"){ + ids = [ids]; + } + + return /** @type {!Promise} */(Promise.all([ + this.transaction("map", "readwrite", function(store){ + store.openCursor().onsuccess = function(){ + const cursor = this.result; + cursor && handle(cursor, ids); + }; + }), + this.transaction("ctx", "readwrite", function(store){ + store.openCursor().onsuccess = function(){ + const cursor = this.result; + cursor && handle(cursor, ids); + }; + }), + SUPPORT_TAGS && this.transaction("tag", "readwrite", function(store){ + store.openCursor().onsuccess = function(){ + const cursor = this.result; + cursor && handle(cursor, ids, /* tag? */ true); + }; + }), + // let filtered = []; + this.transaction("reg", "readwrite", function(store){ + for(let i = 0; i < ids.length; i++){ + store.delete(ids[i]); + } + // return new Promise(resolve => { + // store.openCursor().onsuccess = function(){ + // const cursor = this.result; + // if(cursor){ + // const id = cursor.value; + // if(ids.includes(id)){ + // filtered.push(id); + // cursor.delete(); + // } + // cursor.continue(); + // } + // else{ + // resolve(); + // } + // }; + // }); + }) + // ids = filtered; + ])); +}; + +/** + * @param {IDBRequest} req + * @param {Function=} callback + * @return {!Promise} + */ + +function promisfy(req, callback){ + return new Promise((resolve, reject) => { + /** @this IDBRequest */ + req.onsuccess = function(){ + callback && callback(this.result); + resolve(this.result); + }; + /** @this IDBRequest */ + req.oncomplete = function(){ + callback && callback(this.result); + resolve(this.result); + }; + req.onerror = reject; + req = null; + }); +} diff --git a/src/db/interface.js b/src/db/interface.js new file mode 100644 index 0000000..ea35670 --- /dev/null +++ b/src/db/interface.js @@ -0,0 +1,47 @@ +/** + * @interface + */ +export default function StorageInterface(name, config){}; + +// Mandatory Initializer +// ------------------------------ + +// assign store to an index +StorageInterface.prototype.mount = async function(index){}; +// open connection +StorageInterface.prototype.open = async function(){}; +// close connection +StorageInterface.prototype.close = function(){}; +// drop the database (drop tables) +StorageInterface.prototype.destroy = async function(){}; + +// Mandatory Query Tasks +// ------------------------------ + +// transfer all changes of an index to the database +StorageInterface.prototype.commit = async function(index, _replace, _append){}; +// get results of a term "key" with optional context "ctx" +StorageInterface.prototype.get = async function(key, ctx, limit, offset, resolve, enrich){}; +// get documents stored in index (enrich result) +StorageInterface.prototype.enrich = async function(ids){}; +// check if id exists on a specific index +StorageInterface.prototype.has = async function(id){}; +// delete one id or multiple ids on a specific index +StorageInterface.prototype.remove = async function(ids){}; +// clear all data (truncate) +StorageInterface.prototype.clear = async function(){}; + +// Optional Methods +// ------------------------------ + +/** + * Perform the query intersection on database side + * @type {Function|null} + */ +StorageInterface.prototype.search = async function(index, query, limit, offset, suggest, resolve, enrich){}; + +/** + * Give some information about the storage + * @type {Function|null} + */ +StorageInterface.prototype.info = async function(){}; diff --git a/src/db/mongo/README.md b/src/db/mongo/README.md new file mode 100644 index 0000000..624dab7 --- /dev/null +++ b/src/db/mongo/README.md @@ -0,0 +1,67 @@ +# MongoDB FlexSearch + +MongoDB is a common NoSQL document store providing everything you might want from a solid FlexSearch storage solution. + +You'll need to install the npm package `mongodb` into your project: +```bash +npm install mongodb@6.13.0 +``` + +Create an index and assign a MongoDB storage adapter to it by using `index.mount(db)`: + +```js +import Index from "./index.js"; +import Database from "./db/mongodb/index.js"; +// create an index +const index = new Index(); +// create db instance with optional namespace +const db = new Database("my-store"); +// mount and await before transfering data +await index.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +> Changes are automatically committed by default when you need to wait for the task completion, then you can use the `await index.commit()` method explicitely. You can disable the auto-commit feature optionally. + +## Configuration + +### Custom DB Instance + +Pass a valid `mongodb` instance on creation: + +```js +import { MongoClient } from "mongodb"; +import Database from "./db/mongodb/index.js"; +// assume you've created a custom database instance... +const database = new MongoClient("mongodb://localhost:27017/"); +// connect and await +await database.connect(); +// pass database instance as option +const db = new Database("my-store", { + db: database +}); +``` + +For every instance of `DocumentIndex`, `WorkerIndex` or `Index` (as standalone) you'll need to create a `Database` instance having its own name. + +### Table Structure + +FlexSearch is creating different `DATABASE` entries for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively. + +``` +DATABASE + |__COLLECTION map:field (FlexSearch Data) + |__COLLECTION ctx:field (FlexSearch Data) + |__COLLECTION tag:field (FlexSearch Data) + |__COLLECTION cfg:field (FlexSearch Data) + |__COLLECTION reg (FlexSearch Data) +``` diff --git a/src/db/mongo/index.js b/src/db/mongo/index.js new file mode 100644 index 0000000..e119e07 --- /dev/null +++ b/src/db/mongo/index.js @@ -0,0 +1,695 @@ +import { MongoClient } from "mongodb"; +const defaults = { + host: "localhost", + port: "27017", + user: null, + pass: null +}; +const VERSION = 1; +const fields = ["map", "ctx", "tag", "reg", "cfg"]; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +let CLIENT; +let DB = Object.create(null); + +/** + * @constructor + * @implements StorageInterface + */ + +export default function MongoDB(name, config = {}){ + if(!(this instanceof MongoDB)){ + return new MongoDB(name, config); + } + if(typeof name === "object"){ + name = name.name; + config = name; + } + if(!name){ + console.info("Default storage space was used, because a name was not passed."); + } + this.id = "flexsearch" + (name ? "-" + sanitize(name) : ""); + this.field = config.field ? "-" + sanitize(config.field) : ""; + this.type = config.type || ""; + this.db = config.db || DB[this.id] || CLIENT || null; + this.trx = false; + this.support_tag_search = true; + Object.assign(defaults, config); + this.db && delete defaults.db; +}; + +// MongoDB.mount = function(flexsearch){ +// return new this().mount(flexsearch); +// }; + +MongoDB.prototype.mount = function(flexsearch){ + if(flexsearch instanceof Document){ + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +async function createCollection(db, ref, field){ + switch(ref){ + case "map": + await db.createCollection("map" + field); + await db.collection("map" + field).createIndex({ key: 1 }); + await db.collection("map" + field).createIndex({ id: 1 }); + break; + case "ctx": + await db.createCollection("ctx" + field); + await db.collection("ctx" + field).createIndex({ ctx: 1, key: 1 }); + await db.collection("ctx" + field).createIndex({ id: 1 }); + break; + case "tag": + await db.createCollection("tag" + field); + await db.collection("tag" + field).createIndex({ tag: 1 }); + await db.collection("tag" + field).createIndex({ id: 1 }); + break; + case "reg": + await db.createCollection("reg"); + await db.collection("reg").createIndex({ id: 1 }); + break; + case "cfg": + await db.createCollection("cfg" + field); + } +} + +MongoDB.prototype.open = async function(){ + + if(!this.db){ + if(!(this.db = DB[this.id])){ + if(!(this.db = CLIENT)){ + + let url = defaults.url; + if(!url){ + url = defaults.user + ? `mongodb://${ defaults.user }:${ defaults.pass }@${ defaults.host }:${ defaults.port }` + : `mongodb://${ defaults.host }:${ defaults.port }`; + } + this.db = CLIENT = new MongoClient(url); + await this.db.connect(); + } + } + } + + if(this.db.db){ + this.db = DB[this.id] = this.db.db(this.id); + } + + const collections = await this.db.listCollections().toArray(); + + for(let i = 0; i < fields.length; i++){ + let found = false; + for(let j = 0; j < collections.length; j++){ + if(collections[j].name === fields[i] + (fields[i] !== "reg" ? this.field : "")){ + found = true; + break; + } + } + if(!found){ + await createCollection(this.db, fields[i], this.field); + } + } + + return this.db; +}; + +MongoDB.prototype.close = function(){ + this.db.close(); + this.db = null; + return this; +}; + +MongoDB.prototype.destroy = async function(){ + await Promise.all([ + this.db.dropCollection("map" + this.field), + this.db.dropCollection("ctx" + this.field), + this.db.dropCollection("tag" + this.field), + this.db.dropCollection("cfg" + this.field), + this.db.dropCollection("reg") + ]); + this.close(); +}; + +async function clear(ref){ + await this.db.dropCollection(ref); + await createCollection(this.db, ref, this.field); +} + +MongoDB.prototype.clear = function(){ + return Promise.all([ + clear.call(this, "map" + this.field), + clear.call(this, "ctx" + this.field), + clear.call(this, "tag" + this.field), + clear.call(this, "cfg" + this.field), + clear.call(this, "reg") + ]); +}; + +function create_result(rows, resolve, enrich){ + if(resolve){ + if(!enrich) for(let i = 0; i < rows.length; i++){ + rows[i] = rows[i].id; + } + return rows; + } + else{ + const arr = []; + for(let i = 0, row; i < rows.length; i++){ + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich + ? row + : row.id + ); + } + return arr; + } +} + +MongoDB.prototype.get = async function(key, ctx, limit = 0, offset = 0, resolve = true, enrich = false, tags){ + let rows; + let params = ctx ? { ctx, key } : { key }; + if(!enrich && !tags){ + rows = await this.db.collection((ctx ? "ctx" : "map") + this.field) + .find(params, { projection: { _id: 0, res: 1, id: 1 }, limit, skip: offset }) + .toArray(); + } + else{ + + const project = { _id: 0, id: 1 }; + const stmt = [ + { $match: params } + ]; + + if(!resolve){ + project["res"] = 1; + } + + if(enrich){ + project["doc"] = "$doc.doc"; + stmt.push( + { $lookup: { + from: "reg", + localField: "id", + foreignField: "id", + as: "doc" + } }, + { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: true + } } + ); + } + + if(tags){ + + const match = {}; + for(let i = 0, count = 1; i < tags.length; i += 2){ + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push( + { $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "id", + foreignField: "id", + as: "tag" + count + } } + ); + count++; + } + + stmt.push( + { $project: project }, + { $match: match }, + { $project: { id: 1, doc: 1 } } + ); + } + else{ + stmt.push( + { $project: project } + ); + } + + stmt.push( + { $sort: { res: 1 } }, + { $skip: offset}, + { $limit: limit } + ); + + rows = []; + const result = await this.db.collection((ctx ? "ctx" : "map") + this.field).aggregate(stmt); + + while(true/*await rows.hasNext()*/){ + const row = await result.next(); + if(row) rows.push(row) + else break; + } + } + + return create_result(rows, resolve, enrich); +}; + +MongoDB.prototype.tag = async function(tag, limit = 0, offset = 0, enrich = false){ + + if(!enrich){ + + const rows = await this.db.collection("tag" + this.field) + .find({ tag }, { projection: { _id: 0, id: 1 }, limit, skip: offset }) + .toArray(); + return create_result(rows, true, false); + } + else{ + + const stmt = [ + { $match: { tag } }, + { $skip: offset}, + { $limit: limit }, + { $lookup: { + from: "reg", + localField: "id", + foreignField: "id", + as: "doc" + } }, + { $project: { _id: 0, id: 1, doc: "$doc.doc" } }, + { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: true + } } + ]; + + let rows = []; + const result = await this.db.collection("tag" + this.field).aggregate(stmt); + + while(true/*await rows.hasNext()*/){ + const row = await result.next(); + if(row) rows.push(row) + else break; + } + + return rows; + } +}; + +MongoDB.prototype.enrich = function(ids){ + if(typeof ids !== "object"){ + ids = [ids]; + } + return this.db.collection("reg") + .find({ id: { $in: ids } }, { projection: { _id: 0, id: 1, doc: 1 } }) + .toArray(); +}; + +MongoDB.prototype.has = function(id){ + return this.db.collection("reg").countDocuments({ id }, { limit: 1 }); +}; + +MongoDB.prototype.search = async function(flexsearch, query, limit = 100, offset = 0, suggest = false, resolve = true, enrich = false, tags){ + + let result = [], rows; + + if(query.length > 1 && flexsearch.depth){ + + let params = []; + let keyword = query[0]; + let term; + + for(let i = 1; i < query.length; i++){ + term = query[i]; + const swap = flexsearch.bidirectional && (term > keyword); + params.push({ + ctx: swap ? term : keyword, + key: swap ? keyword : term + }); + keyword = term; + } + + let project = resolve + ? { _id: 1 } + : { _id: 1, res: 1 }; + + const stmt = [ + { $match: { $or: params } }, + { $group: { + _id: "$id", + res: suggest ? { $sum: 1 } : { $min: 1 }, + count: { $sum: 1 } + } } + ]; + + suggest || stmt.push( + { $match: { count: query.length - 1 } } + ); + + if(enrich){ + project["doc"] = "$doc.doc"; + stmt.push( + { $lookup: { + from: "reg", + localField: "_id", + foreignField: "id", + as: "doc" + } }, + { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: true + } } + ); + } + + if(tags){ + + const match = {}; + for(let i = 0, count = 1; i < tags.length; i += 2){ + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push( + { $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "_id", + foreignField: "id", + as: "tag" + count + } } + ); + count++; + } + + stmt.push( + { $project: project }, + { $match: match } + ); + } + else{ + stmt.push( + { $project: project } + ); + } + + stmt.push( + { $sort: suggest + ? { count: -1, res: 1} + : { res: 1 } }, + { $skip: offset}, + { $limit: limit } + ); + + if(tags){ + project = { _id: 1 }; + if(!resolve) project["res"] = 1; + if(enrich) project["doc"] = 1; + + stmt.push( + { $project: project } + ); + } + + rows = await this.db.collection("ctx" + this.field).aggregate(stmt); + } + else{ + + let project = resolve + ? { _id: 1 } + : { _id: 1, res: 1 }; + + const stmt = [ + { $match: { + key: { $in: query } + } }, + { $group: { + _id: "$id", + res: suggest ? { $sum: 1 } : { $min: 1 }, + count: { $sum: 1 } + } } + ]; + + suggest || stmt.push( + { $match: { count: query.length } } + ); + + if(enrich){ + project["doc"] = "$doc.doc"; + stmt.push( + { $lookup: { + from: "reg", + localField: "_id", + foreignField: "id", + as: "doc" + } }, + { $unwind: { + path: "$doc", + preserveNullAndEmptyArrays: true + } } + ); + } + + if(tags){ + + const match = {}; + for(let i = 0, count = 1; i < tags.length; i += 2){ + project["tag" + count] = "$tag" + count + ".tag"; + match["tag" + count] = tags[i + 1]; + stmt.push( + { $lookup: { + from: "tag-" + sanitize(tags[i]), + localField: "_id", + foreignField: "id", + as: "tag" + count + } } + ); + count++; + } + + stmt.push( + { $project: project }, + { $match: match } + ); + } + else{ + stmt.push( + { $project: project } + ); + } + + stmt.push( + { $sort: suggest + ? { count: -1, res: 1 } + : { res: 1 } }, + { $skip: offset}, + { $limit: limit } + ) + + if(tags){ + project = { _id: 1 }; + if(!resolve) project["res"] = 1; + if(enrich) project["doc"] = 1; + + stmt.push( + { $project: project } + ); + } + + rows = await this.db.collection("map" + this.field).aggregate(stmt); + } + + while(true/*await rows.hasNext()*/) { + const row = await rows.next(); + if(row){ + if(resolve && !enrich){ + result.push(row._id); + } + else{ + row.id = row._id; + delete row._id; + result.push(row); + } + } + else break; + } + + if(resolve && !enrich){ + return result; + } + else{ + return create_result(result, resolve, enrich); + } +} + +MongoDB.prototype.info = function(){ + // todo +}; + +MongoDB.prototype.transaction = function(task){ + // not supported + return task.call(this); +}; + +MongoDB.prototype.commit = async function(flexsearch, _replace, _append){ + + // process cleanup tasks + if(_replace){ + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } + else{ + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for(let i = 0, task; i < tasks.length; i++){ + task = tasks[i]; + // there are just removals in the task queue + if(task.clear){ + await this.clear(); + _replace = true; + break; + } + else{ + tasks[i] = task.del; + } + } + if(!_replace){ + if(!_append){ + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && await this.remove(tasks); + } + } + + if(!flexsearch.reg.size){ + return; + } + + if(flexsearch.map.size){ + let data = []; + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + for(let j = 0; j < ids.length; j++){ + data.push({ + key: key, + res: i, + id: ids[j] + }); + } + } + } + } + if(data.length){ + await this.db.collection("map" + this.field).insertMany(data); + flexsearch.map.clear(); + } + } + + if(flexsearch.ctx.size){ + let data = []; + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + for(let j = 0; j < ids.length; j++){ + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: ids[j] + }); + } + } + } + } + } + if(data.length){ + await this.db.collection("ctx" + this.field).insertMany(data); + flexsearch.ctx.clear(); + } + } + + if(flexsearch.tag){ + let data = []; + if(flexsearch.tag){ + for(const item of flexsearch.tag){ + const tag = item[0]; + const ids = item[1]; + if(!ids.length) continue; + for(let j = 0; j < ids.length; j++){ + data.push({ tag, id: ids[j] }); + } + } + } + if(data.length){ + await this.db.collection("tag" + this.field).insertMany(data); + flexsearch.tag.clear(); + } + } + + let data = []; + if(flexsearch.store){ + for(const item of flexsearch.store.entries()){ + const id = item[0]; + const doc = item[1]; + data.push({ id, doc }); + } + } + else if(!flexsearch.bypass){ + for(const id of flexsearch.reg.keys()){ + data.push({ id }); + } + } + if(data.length){ + await this.db.collection("reg").insertMany(data); + flexsearch.store && + flexsearch.store.clear(); + flexsearch.document || + flexsearch.reg.clear(); + } + + // TODO + // await this.db.collection("cfg" + this.field).insertOne({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }); +}; + +MongoDB.prototype.remove = function(ids){ + + if(!ids && ids !== 0) return; + if(typeof ids !== "object"){ + ids = [ids]; + } + + // if(this.type !== "string" && typeof ids[0] !== "number"){ + // ids = ids.map(item => parseInt(item, 10)); + // } + + return Promise.all([ + this.db.collection("map" + this.field).deleteMany({ "id": { "$in": ids }}), + this.db.collection("ctx" + this.field).deleteMany({ "id": { "$in": ids }}), + this.db.collection("tag" + this.field).deleteMany({ "id": { "$in": ids }}), + this.db.collection("reg").deleteMany({ "id": { "$in": ids }}) + ]); +}; diff --git a/src/db/mongo/package.json b/src/db/mongo/package.json new file mode 100644 index 0000000..b8609c2 --- /dev/null +++ b/src/db/mongo/package.json @@ -0,0 +1,16 @@ +{ + "public": true, + "preferGlobal": false, + "name": "flexsearch-mongo", + "version": "0.1.0", + "main": "index.js", + "scripts": {}, + "files": [ + "index.js", + "README.md", + "LICENSE" + ], + "dependencies": { + "mongodb": "^6.13.0" + } +} diff --git a/src/db/postgres/README.md b/src/db/postgres/README.md new file mode 100644 index 0000000..2de2354 --- /dev/null +++ b/src/db/postgres/README.md @@ -0,0 +1,107 @@ +# PostgreSQL FlexSearch + +Postgres is widely used as the default database engine. It can hold large amounts of data and also shines in scaling capabilities. + +You'll need to install the npm package `pg-promise` into your project: +```bash +npm install pg-promise@11.10.2 +``` + +Create an index and assign a Postgres storage adapter to it by using `index.mount(db)`: + +```js +import Index from "./index.js"; +import Database from "./db/postgres/index.js"; + +// your database configuration +const config = { + user: "postgres", + pass: "postgres", + host: "localhost", + port: "5432", + // database name: + name: "postgres" +}; + +// create an index +const index = new Index(); +// create db instance with optional prefix +const db = new Database("my-store", config); +// mount and await before transfering data +await index.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +> Changes are automatically committed by default when you need to wait for the task completion, then you can use the `await index.commit()` method explicitely. You can disable the auto-commit feature optionally. + +## Configuration + +### Custom DB Instance + +Pass a valid `pg-promise` instance on creation: + +```js +import pg_promise from "pg-promise"; +import Database from "./db/postgres/index.js"; +const pgp = pg_promise(); +// assume you've created a custom database instance... +const database = pgp(`postgres://${user}:${pass}@${host}:${port}/${name}`); +// pass database instance as option +const db = new Database("my-store", { + db: database +}); +``` + +### ID Type + +The Postgres driver supports type upgrading by ALTER TABLE. The index will automatically upgrade to `bigint` or to `string`, starting from `integer`. + +Once the type was upgraded, you'll need to re-create the index to switch back. + +When you data content is including numeric strings (eg. for ID "15712") then defining the type will automatically cast into the right type: + +```js +import Database from "./db/postgres/index.js"; +// force integer type +const db = new Database("my-store", { + type: "integer" +}); +// .... +// the string will cast to integer +index.add("15712", "content..."); +``` + +> You will save required disk space and also gain the performance when using a numeric ID over a string type. + +### Table Structure + +FlexSearch is creating different `SCHEMA` for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively. + +``` +DATABASE_NAME + |__ SCHEMA + |__TABLE map:field (FlexSearch Data) + |__TABLE ctx:field (FlexSearch Data) + |__TABLE tag:field (FlexSearch Data) + |__TABLE cfg:field (FlexSearch Data) + |__TABLE reg (FlexSearch Data) +``` + +You can force using a specific SCHEMA by option without passing a name: + +```js +const db = new Database({ + schema: "public" +}); +``` + +Every Instance you made of `DocumentIndex`, `WorkerIndex` or `Index` (as standalone) you'll need to create a `Database` instance having its own schema. \ No newline at end of file diff --git a/src/db/postgres/index.js b/src/db/postgres/index.js new file mode 100644 index 0000000..77f4f12 --- /dev/null +++ b/src/db/postgres/index.js @@ -0,0 +1,1017 @@ +// COMPILER BLOCK --> +import { DEBUG } from "../../config.js"; +// <-- COMPILER BLOCK + +import pg_promise from "pg-promise"; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { concat, toArray } from "../../common.js"; +const defaults = { + schema: "flexsearch", + user: "postgres", + pass: "postgres", + name: "postgres", + host: "localhost", + port: "5432" +}; + +const pgp = pg_promise(/*{ noWarnings: true }*/); +const VERSION = 1; +const MAXIMUM_QUERY_VARS = 16000; +const fields = ["map", "ctx", "reg", "tag", "cfg"]; +const types = { + "text": "text", + "char": "text", + "varchar": "text", + "string": "text", + "number": "int", + "numeric": "int", + "integer": "int", + "smallint": "int", + "tinyint": "int", + "mediumint": "int", + "int": "int", + "int8": "int", + "uint8": "int", + "int16": "int", + "uint16": "int", + "int32": "int", + "uint32": "bigint", + "int64": "bigint", + "bigint": "bigint" +}; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +let DB, TRX; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function PostgresDB(name, config = {}){ + if(!(this instanceof PostgresDB)){ + return new PostgresDB(name, config); + } + if(typeof name === "object"){ + name = name.name; + config = name; + } + if(!name){ + console.info("Default storage space was used, because a name was not passed."); + } + this.id = (config.schema ? sanitize(config.schema) : defaults.schema) + (name ? "_" + sanitize(name) : ""); + this.field = config.field ? "_" + sanitize(config.field) : ""; + this.type = config.type ? types[config.type.toLowerCase()] : "text"; + this.support_tag_search = true; + if(!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); + this.db = DB || (DB = config.db || null); + Object.assign(defaults, config); + this.db && delete defaults.db; +}; + +PostgresDB.prototype.mount = function(flexsearch){ + if(flexsearch instanceof Document){ + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +PostgresDB.prototype.open = async function(){ + + if(!this.db) { + this.db = DB || ( + DB = pgp(`postgres://${defaults.user}:${encodeURIComponent(defaults.pass)}@${defaults.host}:${defaults.port}/${defaults.name}`) + ); + } + + const exist = await this.db.oneOrNone(` + SELECT EXISTS ( + SELECT 1 + FROM information_schema.schemata + WHERE schema_name = '${this.id}' + ); + `); + if(!exist || !exist.exists){ + await this.db.none(`CREATE SCHEMA IF NOT EXISTS ${ this.id };`); + } + + for(let i = 0; i < fields.length; i++){ + const exist = await this.db.oneOrNone(` + SELECT EXISTS ( + SELECT 1 FROM pg_tables + WHERE schemaname = '${this.id}' AND tablename = '${fields[i] + (fields[i] !== "reg" ? this.field : "")}' + ); + `); + if(exist && exist.exists) continue; + + const type = this.type === "text" + ? "varchar(128)" + : this.type; + + switch(fields[i]){ + case "map": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.map${this.field}( + key varchar(128) NOT NULL, + res smallint NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_map_index${this.field} + ON ${this.id}.map${this.field} (key); + CREATE INDEX IF NOT EXISTS ${this.id}_map_id${this.field} + ON ${this.id}.map${this.field} (id); + `); + break; + + case "ctx": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.ctx${this.field}( + ctx varchar(128) NOT NULL, + key varchar(128) NOT NULL, + res smallint NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_ctx_index${this.field} + ON ${this.id}.ctx${this.field} (ctx, key); + CREATE INDEX IF NOT EXISTS ${this.id}_ctx_id${this.field} + ON ${this.id}.ctx${this.field} (id); + `); + break; + + case "tag": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.tag${this.field}( + tag varchar(128) NOT NULL, + id ${type} NOT NULL + ); + CREATE INDEX IF NOT EXISTS ${this.id}_tag_index${this.field} + ON ${this.id}.tag${this.field} (tag); + CREATE INDEX IF NOT EXISTS ${this.id}_tag_id${this.field} + ON ${this.id}.tag${this.field} (id); + `); + break; + + case "reg": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.reg( + id ${type} NOT NULL + CONSTRAINT ${this.id}_reg_pk + PRIMARY KEY, + doc text DEFAULT NULL + ); + `).catch(e => { + // document indexes will try to create same registry table + // and the "IF NOT EXISTS" did not apply on parallel + }); + break; + + case "cfg": + await this.db.none(` + CREATE TABLE IF NOT EXISTS ${this.id}.cfg${this.field}( + cfg text NOT NULL + ); + `); + break; + } + } + + return this.db; +}; + +PostgresDB.prototype.close = function(){ + this.db.close && this.db.close(); + this.db = DB = null; + return this; +}; + +PostgresDB.prototype.destroy = async function(){ + await this.db.none(` + DROP TABLE IF EXISTS ${this.id}.map${this.field}; + DROP TABLE IF EXISTS ${this.id}.ctx${this.field}; + DROP TABLE IF EXISTS ${this.id}.tag${this.field}; + DROP TABLE IF EXISTS ${this.id}.cfg${this.field}; + DROP TABLE IF EXISTS ${this.id}.reg; + `); + this.close(); +}; + +PostgresDB.prototype.clear = function(){ + return this.db.none(` + TRUNCATE TABLE ${this.id}.map${ this.field }; + TRUNCATE TABLE ${this.id}.ctx${ this.field }; + TRUNCATE TABLE ${this.id}.tag${ this.field }; + TRUNCATE TABLE ${this.id}.cfg${ this.field }; + TRUNCATE TABLE ${this.id}.reg; + `); +}; + +function create_result(rows, resolve, enrich){ + if(resolve){ + for(let i = 0; i < rows.length; i++){ + if(enrich){ + if(rows[i].doc){ + rows[i].doc = JSON.parse(rows[i].doc); + } + } + else{ + rows[i] = rows[i].id; + } + } + return rows; + } + else{ + const arr = []; + for(let i = 0, row; i < rows.length; i++){ + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich + ? row + : row.id + ); + } + return arr; + } +} + +PostgresDB.prototype.get = function(key, ctx, limit = 0, offset = 0, resolve = true, enrich = false, tags){ + let rows; + let stmt = ''; + let params = ctx ? [ctx, key] : [key]; + let table = this.id + (ctx ? ".ctx" : ".map") + this.field; + if(tags){ + for(let i = 0, count = params.length + 1; i < tags.length; i+=2){ + stmt += ` AND ${ table }.id IN (SELECT id FROM ${ this.id }.tag_${ sanitize(tags[i]) } WHERE tag = $${ count++ })`; + params.push(tags[i + 1]); + } + } + if(ctx){ + rows = this.db.any(` + SELECT ${ table }.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT JOIN ${ this.id }.reg ON ${ this.id }.reg.id = ${ table }.id + ` : "" } + WHERE ctx = $1 AND key = $2 ${stmt} + ORDER BY res + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" }`, + params + ); + } + else{ + rows = this.db.any(` + SELECT ${ table }.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT JOIN ${ this.id }.reg ON ${ this.id }.reg.id = ${ table }.id + ` : "" } + WHERE key = $1 ${stmt} + ORDER BY res + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" }`, + params + ); + } + return rows.then(function(rows){ + return create_result(rows, resolve, enrich); + }); +}; + +PostgresDB.prototype.tag = function(tag, limit = 0, offset = 0, enrich = false){ + const table = this.id + ".tag" + this.field; + const promise = this.db.any(` + SELECT ${ table }.id + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT JOIN ${ this.id }.reg ON ${ this.id }.reg.id = ${ table }.id + ` : "" } + WHERE tag = $1 + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" }`, + [tag] + ); + enrich || promise.then(function(rows){ + return create_result(rows, true, false); + }); + return promise; +}; + +PostgresDB.prototype.enrich = async function(ids){ + let result = []; + if(typeof ids !== "object"){ + ids = [ids]; + } + for(let count = 0; count < ids.length;){ + const chunk = ids.length - count > MAXIMUM_QUERY_VARS + ? ids.slice(count, count + MAXIMUM_QUERY_VARS) + : count ? ids.slice(count) : ids; + count += chunk.length; + let stmt = ""; + for(let i = 1; i <= chunk.length; i++){ + stmt += (stmt ? "," : "") + "$" + i; + } + const res = await this.db.any(` + SELECT id, doc + FROM ${ this.id }.reg + WHERE id IN (${ stmt })`, + ids + ); + if(res && res.length){ + for(let i = 0, doc; i < res.length; i++){ + if((doc = res[i].doc)){ + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + return result.length === 1 + ? result[0] + : result.length > 1 + ? concat(result) + : result; +}; + +PostgresDB.prototype.has = function(id){ + return this.db.oneOrNone("SELECT EXISTS(SELECT 1 FROM " + this.id + ".reg WHERE id = $1)", [id]); +}; + +PostgresDB.prototype.search = function(flexsearch, query, limit = 100, offset = 0, suggest = false, resolve = true, enrich = false, tags){ + + let rows; + + if(query.length > 1 && flexsearch.depth){ + + let where = ""; + let params = []; + let keyword = query[0]; + let term; + let count = 1; + // variant new + + for(let i = 1; i < query.length; i++){ + term = query[i]; + const swap = flexsearch.bidirectional && (term > keyword); + where += (where ? " OR " : "") + `(ctx = $${ count++ } AND key = $${ count++ })` + params.push(swap ? term : keyword, swap ? keyword : term); + keyword = term; + } + + if(tags){ + where = "(" + where + ")"; + for(let i = 0; i < tags.length; i+=2){ + where += ` AND id IN (SELECT id FROM ${ this.id }.tag_${ sanitize(tags[i]) } WHERE tag = $${ count++ })`; + params.push(tags[i + 1]); + } + } + + rows = this.db.any(` + SELECT r.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ( + SELECT id, count(*) as count, + ${ suggest ? "SUM" : "MIN" }(res) as res + FROM ${ this.id }.ctx${ this.field } + WHERE ${ where } + GROUP BY id + ) as r + ${ enrich ? ` + LEFT JOIN ${ this.id }.reg ON ${ this.id }.reg.id = r.id + ` : "" } + ${ suggest ? "" : "WHERE count = " + (query.length - 1) } + ORDER BY ${ suggest ? "count DESC, res" : "res" } + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, params); + + // variant 1 + + // for(let i = 1, count = 1; i < query.length; i++){ + // where += (where ? " UNION " : "") + ` + // SELECT id, res + // FROM ${this.id}.ctx${this.field} + // WHERE ctx = $${count++} AND key = $${count++} + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params.push( + // swap ? term : keyword, + // swap ? keyword : term + // ); + // keyword = term; + // } + // + // rows = await db.any(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, params); + + } + else{ + + let where = ""; + let count = 1; + let query_length = query.length; + for(let i = 0; i < query_length; i++){ + where += (where ? "," : "") + "$" + count++; + } + where = "key " + (query_length > 1 ? "IN (" + where + ")" : "= " + where ); + + if(tags){ + where = "(" + where + ")"; + for(let i = 0; i < tags.length; i+=2){ + where += ` AND id IN (SELECT id FROM ${ this.id }.tag_${ sanitize(tags[i]) } WHERE tag = $${ count++ })`; + query.push(tags[i + 1]); + } + } + + rows = this.db.any(` + SELECT r.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ( + SELECT id, count(*) as count, + ${ suggest ? "SUM" : "MIN" }(res) as res + FROM ${ this.id }.map${ this.field } + WHERE ${ where } + GROUP BY id + ) as r + ${ enrich ? ` + LEFT JOIN ${ this.id }.reg ON ${ this.id }.reg.id = r.id + ` : "" } + ${ suggest ? "" : "WHERE count = " + query_length } + ORDER BY ${ suggest ? "count DESC, res" : "res" } + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, query); + + // variant 1 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " UNION " : "") + ` + // SELECT id, res + // FROM ${ this.id }.map${ this.field } + // WHERE key = $${i} + // `; + // } + // rows = await db.any(` + // SELECT id, res + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${where}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, query); + + // variant 2 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " AND EXISTS " : "") + `(SELECT FROM ${this.id}.map${this.field} as d WHERE d.id = t.id AND d.key = $` + i + ")"; + // } + // rows = await db.any(` + // SELECT t.id, min(t.res) + // FROM ${this.id}.map${this.field} AS t + // WHERE EXISTS ${where} + // GROUP BY t.id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + + // variant 3 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " INTERSECT " : "") + `SELECT id FROM ${this.id}.map${this.field} WHERE key = $` + i; + // } + // rows = await db.any(` + // WITH filtering (id) AS(${where}) + // SELECT t.id, min(t.res) + // FROM ${this.id}.map${this.field} AS t + // JOIN filtering USING (id) + // GROUP BY t.id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + + // variant 4 + // for(let i = 1; i <= query.length; i++){ + // where += (where ? " INTERSECT " : "") + `SELECT id FROM ${this.id}.map${this.field} WHERE key = $` + i; + // } + // rows = await db.any(` + // SELECT id, min(res) + // FROM ${this.id}.map${this.field} + // WHERE id IN (${where}) + // GROUP BY id + // LIMIT ${limit || 100} + // OFFSET ${offset || 0} + // `, query); + } + + return rows.then(function(rows){ + return create_result(rows, resolve, enrich); + }); +}; + +PostgresDB.prototype.info = function(){ + // todo +}; + +// PostgresDB.prototype.transaction = async function(task){ +// const self = this; +// if(TRX){ +// return TRX.then(function(){ +// return self.transaction(task); +// //task.call(self, TRX); +// }); +// } +// TRX = await this.db.tx(async function(trx){ +// await task.call(self, trx); +// }); +// TRX = null; +// }; + +PostgresDB.prototype.transaction = function(task){ + if(TRX){ + return task.call(this, TRX); + } + const self = this; + return (TRX || this.db).tx(function(trx){ + return task.call(self, TRX = trx); + }).finally(function(){ + TRX = null; + }) +}; + +PostgresDB.prototype.commit = async function(flexsearch, _replace, _append){ + + // process cleanup tasks + if(_replace){ + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } + else{ + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for(let i = 0, task; i < tasks.length; i++){ + task = tasks[i]; + // there are just removals in the task queue + if(task.clear){ + await this.clear(); + _replace = true; + break; + } + else{ + tasks[i] = task.del; + } + } + if(!_replace){ + if(!_append){ + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && await this.remove(tasks); + } + + //console.log("tasks:", tasks) + } + + if(!flexsearch.reg.size){ + return; + } + + await this.transaction(function(trx){ + + const batch = []; + + // Datastore + Registry + + if(flexsearch.store){ + let data = []; + let stmt = new pgp.helpers.ColumnSet(["id", "doc"],{ + table: this.id + ".reg" + }); + for(const item of flexsearch.store.entries()){ + const id = item[0]; + const doc = item[1]; + // const migration = checkMigration.call(this, id); + // migration && await migration; + data.push({ id, doc: doc && JSON.stringify(doc) }); + if(data.length === MAXIMUM_QUERY_VARS){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + data = []; + } + } + if(data.length){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + } + // while(data.length){ + // let next; + // if(data.length > MAXIMUM_QUERY_VARS){ + // next = data.slice(MAXIMUM_QUERY_VARS); + // data = data.slice(0, MAXIMUM_QUERY_VARS); + // } + // let insert = pgp.helpers.insert(data, stmt); + // trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')); + // if(next) data = next; + // else break; + // } + } + + // Registry Only + + else if(!flexsearch.bypass){ + let data = []; + let stmt = new pgp.helpers.ColumnSet(["id"],{ + table: this.id + ".reg" + }); + for(const id of flexsearch.reg.keys()){ + // const migration = checkMigration.call(this, id); + // migration && await migration; + data.push({ id }); + if(data.length === MAXIMUM_QUERY_VARS){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + data = []; + } + } + if(data.length){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + } + } + + // Default Index + + if(flexsearch.map.size){ + let data = []; + let stmt = new pgp.helpers.ColumnSet(["key", "res", "id"], { + table: this.id + ".map" + this.field + }); + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + //this.type || (this.type = typeof ids[0]); + // let stmt = "($1,$2,$3)"; + // let params = [key, i, ids[0]]; + for(let j = 0; j < ids.length; j++){ + // stmt += ",($1,$2,$3)"; + // params.push(key, i, ids[j]); + //trx.none(`INSERT INTO ${config.schema}.map${self.field} (key, res, id) VALUES ($1,$2,$3)`, [key, i, ids[j]]); + data.push({ + key: key, + res: i, + id: ids[j] + }); + if(data.length === MAXIMUM_QUERY_VARS){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + data = []; + } + } + } + } + } + if(data.length){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + } + } + + // Context Index + + if(flexsearch.ctx.size){ + let data = []; + let stmt = new pgp.helpers.ColumnSet(["ctx", "key", "res", "id"], { + table: this.id + ".ctx" + this.field + }); + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + // let stmt = "(?,?,?)"; + // let params = [ctx_key + ":" + key, i, ids[0]]; + for(let j = 0; j < ids.length; j++){ + // stmt += ",(?,?,?)"; + // params.push(ctx_key + ":" + key, i, ids[j]); + //trx.none("INSERT INTO " + config.schema + ".ctx" + self.field + " (ctx, key, res, id) VALUES ($1,$2,$3,$4)", [ctx_key, key, i, ids[j]]); + data.push({ + ctx: ctx_key, + key: key, + res: i, + id: ids[j] + }); + if(data.length === MAXIMUM_QUERY_VARS){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + data = []; + } + } + } + } + } + } + if(data.length){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + } + } + + // Tag Index + + if(flexsearch.tag){ + let data = []; + let stmt = new pgp.helpers.ColumnSet(["tag", "id"],{ + table: this.id + ".tag" + this.field + }); + for(const item of flexsearch.tag){ + const tag = item[0]; + const ids = item[1]; + if(!ids.length) continue; + for(let j = 0; j < ids.length; j++){ + data.push({ tag, id: ids[j] }); + if(data.length === MAXIMUM_QUERY_VARS){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + data = []; + } + } + } + if(data.length){ + let insert = pgp.helpers.insert(data, stmt); + batch.push( + trx.none(insert.replace(/^(insert into )"([^"]+)"/, '$1 $2')) + ); + } + } + + // Field Configuration + + // TODO + // trx.none("INSERT INTO " + this.id + ".cfg" + this.field + " (cfg) VALUES ($1)", [ + // JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // }) + // ]); + + //return Promise.all(batch); + + if(batch.length){ + return trx.batch(batch); + } + }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && + flexsearch.tag.clear(); + flexsearch.store && + flexsearch.store.clear(); + flexsearch.document || + flexsearch.reg.clear(); +}; + +PostgresDB.prototype.remove = function(ids){ + + if(!ids && ids !== 0){ + return; + } + if(typeof ids !== "object"){ + ids = [ids]; + } + if(!ids.length){ + return; + } + + // ids = [ids]; + // return this.db.none( + // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1);", [ids] + // ); + + // ids = [ids]; + // return Promise.all([ + // this.db.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + // this.db.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids) + // ]); + + return this.transaction(function(trx){ + + //console.log("remove:", ids); + + // ids = [ids]; + // trx.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids); + // trx.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids); + + // ids = [ids]; + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", ids); + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", ids); + // return; + + // return trx.none( + // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1);" + + // "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1);", [ids] + // ); + + // while(ids.length){ + // let next; + // let stmt = ""; + // let chunk = 0; + // let migration; + // for(let i = 0; i < ids.length; i++){ + // stmt += (stmt ? "," : "") + "$" + (i + 1); // + "::text"; + // // maximum count of variables supported + // if(++chunk === MAXIMUM_QUERY_VARS){ + // next = ids.slice(MAXIMUM_QUERY_VARS); + // ids = ids.slice(0, MAXIMUM_QUERY_VARS); + // break; + // } + // } + // + // trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN (" + stmt + ")", ids), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id IN (" + stmt + ")", ids) + // ]); + // + // // trx.none( + // // "DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN (" + stmt + ");" + + // // "DELETE FROM " + this.id + ".reg WHERE id IN (" + stmt + ");", ids + // // ); + // + // if(next) ids = next; + // else break; + // } + + ids = [ids]; + return trx.batch([ + trx.none({ text: "DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + trx.none({ text: "DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + trx.none({ text: "DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", rowMode: "array" }, ids), + trx.none({ text: "DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", rowMode: "array" }, ids) + ]); + + // ids = [ids]; + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id = ANY ($1)", ids), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY ($1)", ids) + // ]); + + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".tag" + this.field + " WHERE id IN ($1:csv)", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id IN ($1:csv)", [ids]) + // ]); + + // const type = self.type === "text" ? "text" : "int"; + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + self.field + " WHERE id = ANY($1::" + type + "[])", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + self.field + " WHERE id = ANY($1::" + type + "[])", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY($1::" + type + "[])", [ids]) + // ]); + + // return trx.batch([ + // trx.none("DELETE FROM " + this.id + ".map" + self.field + " WHERE id = ANY($1)", [ids]), + // trx.none("DELETE FROM " + this.id + ".ctx" + self.field + " WHERE id = ANY($1)", [ids]), + // trx.none("DELETE FROM " + this.id + ".reg WHERE id = ANY($1)", [ids]) + // ]); + }); +}; + +// if(this.type === "int" && typeof ids[0] === "string"){ +// ids = ids.map(item => parseInt(item, 10)); +// } +// if(this.type === "bigint" && typeof ids[0] === "string"){ +// ids = ids.map(item => BigInt(item)); +// } +// if(this.type === "string" && typeof ids[0] === "number"){ +// ids = ids.map(item => item + ""); +// } +// this.type !== "string" && checkMigration.call(this, ids[0]); + +// function checkMigration(id){ +// if(this.type !== "string"){ +// if(typeof id === "object"){ +// id = id[0]; +// } +// if(typeof id === "string"){ +// this.type = "string"; +// return upgradeTextId.call(this); +// } +// if(this.type !== "bigint"){ +// if(id > 2 ** 31 - 1){ +// this.type = "bigint"; +// return upgradeBigIntId.call(this); +// } +// } +// } +// } +// +// function upgradeTextId(){ +// return this.db.none(` +// ALTER TABLE ${this.id}.map${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.ctx${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.tag${this.field} +// ALTER COLUMN id type varchar(128) +// USING id::text; +// ALTER TABLE ${this.id}.reg +// ALTER COLUMN id type varchar(128) +// USING id::text; +// `); +// } +// +// function upgradeBigIntId(){ +// return this.db.none(` +// ALTER TABLE ${this.id}.map${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.ctx${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.tag${this.field} +// ALTER COLUMN id type bigint +// USING id::bigint; +// ALTER TABLE ${this.id}.reg +// ALTER COLUMN id type bigint +// USING id::bigint; +// `); +// } diff --git a/src/db/postgres/package.json b/src/db/postgres/package.json new file mode 100644 index 0000000..9219e39 --- /dev/null +++ b/src/db/postgres/package.json @@ -0,0 +1,16 @@ +{ + "public": true, + "preferGlobal": false, + "name": "flexsearch-postgres", + "version": "0.1.0", + "main": "index.js", + "scripts": {}, + "files": [ + "index.js", + "README.md", + "LICENSE" + ], + "dependencies": { + "pg-promise": "^11.10.2" + } +} diff --git a/src/db/redis-json/index.js b/src/db/redis-json/index.js new file mode 100644 index 0000000..e8ccabd --- /dev/null +++ b/src/db/redis-json/index.js @@ -0,0 +1,551 @@ +import { createClient } from "redis"; +import { create_object } from "../../common.js"; +const config = { + url: "redis://localhost:6379", + // host: "localhost", + // port: "6379", + // password: void 0, + // prefix: "flexsearch" +}; +const VERSION = 1; +const fields = ["map", "ctx", "reg", "cfg"]; +import StorageInterface from "../interface.js"; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function RedisDB(sid, field){ + //field = "Test-456"; + this.id = "flexsearch" + (sid ? "-" + sid : ""); + // prevent malicious sql injection + this.field = field ? "-" - field.toLowerCase().replace(/[^a-z_-]/g, "") : ""; + this.type = ""; + this.resolution = 0; + this.resolution_ctx = 0; + this.fastupdate = true; + this.db = null; + this.trx = false; +}; + +RedisDB.mount = function(flexsearch){ + return new this().mount(flexsearch); +}; + +RedisDB.prototype.mount = function(flexsearch){ + flexsearch.db = this; + this.resolution = flexsearch.resolution; + this.resolution_ctx = flexsearch.resolution_ctx; + this.fastupdate = flexsearch.fastupdate; + return this.open(); +}; + +RedisDB.prototype.open = async function(){ + + if(this.db) return this.db; + config.prefix = this.id; + + return this.db = + await createClient(config) + .on("error", err => console.error(err)) + .connect(); +}; + +RedisDB.prototype.close = async function(){ + await this.db.disconnect(); // this.db.client.disconnect(); + this.db = null; + return this; +}; + +RedisDB.prototype.destroy = async function(){ + await this.db.flushAll(); + return this.close(); +}; + +RedisDB.prototype.clear = function(ref){ + return fields.includes(ref) && this.db./*json.*/del(ref + this.field); +}; + +RedisDB.prototype.reset = function(callback){ + return this.db.flushAll(); +}; + +RedisDB.prototype.get = function(ref, key, ctx){ + let result; + switch(ref){ + case "map": + result = this.db.json.get(ref + this.field, { path: "$." + key + "[*]" }); + // fallthrough + case "ctx": + result = result || this.db.json.get(ref + this.field, { path: "$." + ctx + "." + key + "[*]" }); + return this.type === "number" + ? result.then( + res => res.map( + arr => arr + ? arr.map(id => parseInt(id, 10)) + : arr + ) + ) + : result; + case "reg": + return this.db.sIsMember("reg", "" + key); + case "cfg": + return this.db.get("cfg" + this.field); + } +}; + +RedisDB.prototype.has = function(ref, key, ctx){ + switch(ref){ + case "map": + return this.db.json.arrLen("map" + self.field, "$." + key); + case "ctx": + return this.db.json.arrLen("ctx" + self.field, "$." + ctx + "." + key); + case "reg": + return this.db.sIsMember("reg", "" + key); + case "cfg": + return this.db.exists("cfg" + self.field); + } +}; + +RedisDB.prototype.info = function(){ + // todo +}; + +RedisDB.prototype.transaction = async function(task, callback){ + + if(this.trx){ + return task.call(this, this.trx); + } + + this.trx = this.db.multi(); + await task.call(this, this.trx); + const promise = this.trx.exec(); + this.trx = null; + callback && promise.then(callback); + await promise; +}; + +function not_empty(arr){ + for(let i = 0; i < arr.length; i++){ + if(arr[i] && arr[i].length){ + return true; + } + } +} + +RedisDB.prototype.commit = async function(flexsearch, _replace, _append){ + + _replace + ? await this.db.flushAll() + : _append || await this.remove([...flexsearch.reg.keys()]); + + await this.transaction(async function(trx){ + + // the biggest issue with RedisJSON is the need of an existing + // structure before using JSON.ARRAPPEND on it. + // the most cost comes about of checking for this structure. + + // todo: + // instead of JSON.SET map $.ctx.term[*][*] the json could be split into + // term scopes like JSON.SET map:ctx:term $[*][*]. + + let exist = await this.db.json.objLen("map" + this.field); + if(!exist){ + + trx.json.set("map" + this.field, "$", Object.fromEntries(flexsearch.map)); + + if(this.fastupdate){ + const reg = new Map(); + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const ref = "m:" + /*"$." +*/ key + "[" + i + "]"; + for(let j = 0, arr, id; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + //trx.sAdd("ref" + this.field + ":" + ids[j], "map:" + ref); + } + } + } + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + } + else{ + + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + + let i = 0; + let failed; + + // optimistic try to add to an existing slot + for(let ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const ref = /*"$." +*/ key + "[" + i + "]"; + // transaction can't be used here because it needs some result immediately + const res = await this.db.json.arrAppend("map" + this.field, "$." + ref, ...ids); + // when not found break and continue with structure checking + if(!res.length) { + failed = true; + break; + } + if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + trx.sAdd("ref" + this.field + ":" + ids[j], "m:" + ref); + } + } + } + if(!failed) continue; + + exist = (await this.db.json.arrLen("map" + this.field, "$." + key))[0]; + if(!exist /*|| getsize(exist, 0) === 0*/){ + trx.json.set("map" + this.field, "$." + key, arr); + + if(this.fastupdate){ + const reg = new Map(); + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const ref = "m:" + /*"$." +*/ key + "[" + i + "]"; + for(let j = 0, id, arr; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + //trx.sAdd("ref" + this.field + ":" + ids[j], "map:" + ref); + } + } + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + + continue; + } + + for(let ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + + exist = (await this.db.json.arrLen("map" + this.field, "$." + key + "[" + i + "]"))[0]; + if(!exist){ + trx.json.set("map" + this.field, "$." + key + "[" + i + "]", ids); + + if(this.fastupdate){ + const reg = new Map(); + const ref = "m:" + /*"$." +*/ key + "[" + i + "]"; + for(let j = 0, id, arr; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + //trx.sAdd("ref" + this.field + ":" + ids[j], "map:" + ref); + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + + continue; + } + + const ref = /*"$." +*/ key + "[" + i + "]"; + trx.json.arrAppend("map" + this.field, "$." + ref, ...ids); + if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + trx.sAdd("ref" + this.field + ":" + ids[j], "m:" + ref); + } + } + } + } + } + + exist = await this.db.json.objLen("ctx" + this.field); + if(!exist){ + const obj = Object.create(null); + for(const item of flexsearch.ctx){ + const key = item[0]; + const value = item[1]; + obj[key] = Object.fromEntries(value); + // obj[ctx_key] = Object.create(null); + // for(let [key, value] of ctx_value){ + // obj[ctx_key][key] = value; + // } + } + trx.json.set("ctx" + this.field, "$", obj); + + if(this.fastupdate){ + const reg = new Map(); + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const ref = "c:" + /*"$." +*/ ctx_key + ":" + key + "[" + i + "]"; + for(let j = 0, id, arr; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + //trx.sAdd("ref" + this.field + ":" + ids[j], "map:" + ref); + } + } + } + } + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + } + else{ + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + + let exist = (await this.db.json.objLen("ctx" + this.field, "$." + ctx_key))[0]; + if(!exist /*|| getsize(exist) === 0*/){ + trx.json.set("ctx" + this.field, "$." + ctx_key, Object.fromEntries(ctx_value)); + + if(this.fastupdate){ + const reg = new Map(); + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const ref = "c:" + /*"$." +*/ ctx_key + ":" + key + "[" + i + "]"; + for(let j = 0, id, arr; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + } + } + } + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + + continue; + } + + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + + let i = 0; + let failed; + + // optimistic try to add to an existing slot + for(let ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const res = await this.db.json.arrAppend("ctx" + this.field, "$." + ctx_key + "." + key + "[" + i + "]", ...ids); + if(!res.length){ + failed = true; + break; + } + } + } + if(!failed) continue; + + //let exist = await this.db.json.get("ctx" + this.field, { path: "$." + ctx_key + ":" + key + "[*]" }); + let exist = (await this.db.json.arrLen("ctx" + this.field, "$." + ctx_key + "." + key))[0]; + if(!exist /*|| getsize(exist) === 0*/){ + trx.json.set("ctx" + this.field, "$." + ctx_key + "." + key, arr); + + if(this.fastupdate){ + const reg = new Map(); + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + const ref = "c:" + /*"$." +*/ ctx_key + ":" + key + "[" + i + "]"; + for(let j = 0, id, arr; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + } + } + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + + continue; + } + + for(let ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + this.type || (this.type = typeof ids[0]); + exist = (await this.db.json.arrLen("ctx" + this.field, "$." + ctx_key + "." + key + "[" + i + "]"))[0]; + if(!exist){ + trx.json.set("ctx" + this.field, "$." + ctx_key + "." + key + "[" + i + "]", ids); + + if(this.fastupdate){ + const reg = new Map(); + const ref = "c:" + /*"$." +*/ ctx_key + ":" + key + "[" + i + "]"; + for(let j = 0, id, arr; j < ids.length; j++){ + id = ids[j]; + arr = reg.get(id); + arr || reg.set(id, arr = []); + arr.push(ref); + } + for(const item of reg){ + const key = item[0]; + const value = item[1]; + trx.sAdd("ref" + this.field + ":" + key, value); + } + } + + continue; + } + + const ref = ctx_key + "." + key + "[" + i + "]"; + trx.json.arrAppend("ctx" + this.field, "$." + ref, ...ids); + if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + trx.sAdd("ref" + this.field + ":" + ids[j], "c:" + ref); + } + // for(let j = 1; j < ids.length; j++){ + // exist = (await this.db.json.arrLen("ctx" + self.field, "$." + ctx_key + "." + key + "[" + i + "]"))[0]; + // if(exist /*&& exist.length*/){ + // trx.json.arrAppend("ctx" + self.field, "$." + ctx_key + "." + key + "[" + i + "]", ids[j]); + // } + // else{ + // trx.json.set("ctx" + this.field, "$." + ctx_key + "." + key + "[" + i + "]", ids); + // } + // } + } + } + } + } + } + + let ids = [...flexsearch.reg.keys()]; + if(ids.length){ + if(typeof ids[0] === "number"){ + ids = ids.map(id => "" + id); + } + trx.sAdd("reg", ids); + } + + trx.set("cfg" + this.field, JSON.stringify({ + "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + "tokenize": flexsearch.tokenize, + "resolution": flexsearch.resolution, + "minlength": flexsearch.minlength, + "optimize": flexsearch.optimize, + "fastupdate": flexsearch.fastupdate, + "encoder": flexsearch.encoder, + "context": { + "depth": flexsearch.depth, + "bidirectional": flexsearch.bidirectional, + "resolution": flexsearch.resolution_ctx + } + })); + }); + + //flexsearch.map = new Map(); + //flexsearch.ctx = new Map(); + //flexsearch.reg = flexsearch.bidirectional ? new Map() : new Set(); + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.reg.clear(); +}; + +RedisDB.prototype.remove = async function(ids){ + + if(typeof ids !== "object"){ + ids = [ids]; + } + if(!ids.length){ + return; + } + if(typeof ids[0] === "number"){ + ids = ids.map(item => "" + item); + } + + const check = await this.db.smIsMember("reg", ids); + + return this.transaction(async function(trx){ + //let stmt = ""; + for(let i = 0, id; i < ids.length; i++){ + if(!check[i]) continue; + id = ids[i]; + + if(this.fastupdate){ + const refs = await this.db.sMembers("ref" + this.field + ":" + id); + if(refs){ + for(let j = 0; j < refs.length; j++){ + const ref = refs[j][0] === "m" ? "map" : "ctx"; + const path = refs[j].substring(2); + trx.json.del(ref + this.field, "$." + path + "[?@==" + id + "]"); + //trx.json.del("ctx" + this.field, refs[j] + "[?@==" + id + "]"); + } + trx.del("ref" + this.field + ":" + id); + } + } + else{ + + trx.json.del("map" + this.field, "$.*[*][?@==" + id + "]"); + trx.json.del("ctx" + this.field, "$.*.*[*][?@==" + id + "]"); + } + + //stmt += (stmt ? "||" : "") + "@==" + ids[i]; + //const result = await this.db.json.arrIndex("map" + this.field, "$.*.* ", ids[i]) + //trx.json.del("map" + this.field, "$.*[*][?@==" + ids[i] + "]"); + //trx.json.del("ctx" + this.field, "$.*.*[*][?@==" + ids[i] + "]"); + trx.sRem("reg", ids[i]); + } + //stmt += "||@==null"; + + //trx.json.del("map" + this.field, "$.*[*][?" + stmt + "]"); + //trx.json.del("ctx" + this.field, "$.*.*[*][?" + stmt + "]"); + + // trx.json.del("map" + this.field, "$..[?@==null]"); + // trx.json.del("ctx" + this.field, "$..[?@==null]"); + }); +}; + +// RedisDB.prototype.promisfy = function(opt){ +// const db = this.db; +// return new Promise(function(resolve, reject){ +// db[opt.method](opt.stmt, opt.params || [], function(err, rows){ +// opt.callback && opt.callback(rows); +// err ? reject(err) +// : resolve(rows); +// }); +// }); +// }; \ No newline at end of file diff --git a/src/db/redis/README.md b/src/db/redis/README.md new file mode 100644 index 0000000..b4fe5c9 --- /dev/null +++ b/src/db/redis/README.md @@ -0,0 +1,105 @@ +# Redis FlexSearch + +Redis is a standard storage engine in almost every modern application stack. It combines the advantages of both worlds: the performance of an InMemory engine by also keeping data persistent. The downside is that all the data has to keep in RAM. + +You'll need to install the npm package `redis` into your project: +```bash +npm install redis@4.7.0 +``` + +Create an index and assign a Redis storage adapter to it by using `index.mount(db)`: + +```js +import Index from "./index.js"; +import Database from "./db/redis/index.js"; + +// Redis Connection +const config = { + host: "localhost", + port: "6379", + user: null, + pass: null +}; + +// create an index +const index = new Index(); +// create db instance with optional prefix +const db = new Database("my-store", config); +// mount and await before transfering data +await index.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +> Changes are automatically committed by default when you need to wait for the task completion, then you can use the `await index.commit()` method explicitely. You can disable the auto-commit feature optionally. + +## Configuration + +### Custom DB Instance + +Pass a valid `redis` instance on creation: + +```js +import { createClient } from "redis"; +import Database from "./db/redis/index.js"; +// assume you've created a custom redis instance... +const redis = await createClient("redis://localhost:6379").connect(); +// pass instance as option +const db = new Database("my-store", { + db: redis +}); +``` + +### ID Type + +Since Redis stores everything as Strings, you'll need to define the type of ID. Otherwise, you'll get back stringify ID results by default. +Also when your data content includes numeric strings (eg. "15712"), defining a type will automatically cast IDs into your choice: + +```js +import Database from "./db/redis/index.js"; +// force integer type +const db = new Database("my-store", { + type: "integer" +}); +// .... +index.add("15712", "content..."); +// IDs will cast to integer +let result = await index.search("content..."); +// -> [15712] +``` + +### Table Structure + +FlexSearch is using a prefix style to prevent collision with other existing keys. + +Prefix Schema when no name was set: + +``` +flexsearch|map:field +flexsearch|ctx:field +flexsearch|tag:field +flexsearch|cfg:field +flexsearch|reg +flexsearch|doc +``` + +Prefix Schema when name was set e.g. "my-store": + +``` +my-store|map:field +my-store|ctx:field +my-store|tag:field +my-store|cfg:field +my-store|reg +my-store|doc +``` + +For every instance of `DocumentIndex`, `WorkerIndex` or `Index` (as standalone) you'll need to create a `Database` instance having its own name. \ No newline at end of file diff --git a/src/db/redis/index.js b/src/db/redis/index.js new file mode 100644 index 0000000..4e7199c --- /dev/null +++ b/src/db/redis/index.js @@ -0,0 +1,560 @@ +// COMPILER BLOCK --> +import { DEBUG } from "../../config.js"; +// <-- COMPILER BLOCK + +import { createClient } from "redis"; +const defaults = { + host: "localhost", + port: "6379", + user: null, + pass: null +}; +const VERSION = 1; +const fields = ["map", "ctx", "reg", "tag", "doc", "cfg"]; +import StorageInterface from "../interface.js"; +import Document from "../../document.js"; +import { toArray } from "../../common.js"; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_\-]/g, ""); +} + +let DB, TRX; + +/** + * @constructor + * @implements StorageInterface + */ + +export default function RedisDB(name, config = {}){ + if(!(this instanceof RedisDB)){ + return new RedisDB(name, config); + } + if(typeof name === "object"){ + name = name.name; + config = name; + } + if(!name){ + console.info("Default storage space was used, because a name was not passed."); + } + this.id = (name ? sanitize(name) : "flexsearch") + "|"; + this.field = config.field ? "-" + sanitize(config.field) : ""; + this.type = config.type || ""; + this.fastupdate = true; + this.db = config.db || DB || null; + this.support_tag_search = true; + //this.trx = false; + Object.assign(defaults, config); + this.db && delete defaults.db; +}; + +// RedisDB.mount = function(flexsearch){ +// return new this().mount(flexsearch); +// }; + +RedisDB.prototype.mount = function(flexsearch){ + if(flexsearch instanceof Document){ + return flexsearch.mount(this); + } + flexsearch.db = this; + // todo support + //this.fastupdate = flexsearch.fastupdate; + return this.open(); +}; + +RedisDB.prototype.open = async function(){ + if(this.db){ + return this.db + } + if(DB){ + return this.db = DB; + } + let url = defaults.url; + if(!url){ + url = defaults.user + ? `redis://${defaults.user}:${defaults.pass}@${defaults.host}:${defaults.port}` + : `redis://${defaults.host}:${defaults.port}`; + } + return this.db = + await createClient(url) + .on("error", err => console.error(err)) + .connect(); +}; + +RedisDB.prototype.close = async function(){ + await this.db.disconnect(); // this.db.client.disconnect(); + this.db = null; + return this; +}; + +RedisDB.prototype.clear = function(){ + return this.db.unlink([ + this.id + "map" + this.field, + this.id + "ctx" + this.field, + this.id + "tag" + this.field, + this.id + "cfg" + this.field, + this.id + "doc", + this.id + "reg" + ]); +}; + +function create_result(range, type, resolve, enrich){ + if(resolve){ + for(let i = 0, tmp, id; i < range.length; i++){ + tmp = range[i]; + id = type === "number" + ? parseInt(tmp.value || tmp, 10) + : tmp.value || tmp; + range[i] = /*enrich + ? { id, doc: tmp.doc } + :*/ id; + } + return range; + } + else{ + let result = []; + for(let i = 0, tmp, id, score; i < range.length; i++){ + tmp = range[i]; + id = type === "number" + ? parseInt(tmp.value, 10) + : tmp.value + score = tmp.score; + result[score] || (result[score] = []); + result[score].push( + enrich + ? { id, doc: tmp.doc } + : id + ); + } + return result; + } +} + +RedisDB.prototype.get = function(key, ctx, limit = 0, offset = 0, resolve = true, enrich = false, tags){ + + if(tags){ + // flexsearch dummy + const flexsearch = { depth: !!ctx }; + const query = ctx ? [ctx, key] : [key]; // keyword first + return this.search(flexsearch, query, limit, offset, /* suggest */ false, resolve, enrich, tags); + } + + const type = this.type; + const self = this; + let result; + + if(ctx){ + result = this.db[resolve ? "zRange" : "zRangeWithScores"]( + this.id + "ctx" + this.field + ":" + ctx + ":" + key, + "" + offset, + "" + (offset + limit - 1), + { REV: true } + ); + } + else{ + result = this.db[resolve ? "zRange" : "zRangeWithScores"]( + this.id + "map" + this.field + ":" + key, + "" + offset, + "" + (offset + limit - 1), + { REV: true } + ); + } + + return result.then(async function(range){ + if(!range.length) return range; + if(enrich) range = await self.enrich(range); + return create_result(range, type, resolve, enrich); + }); +}; + +RedisDB.prototype.tag = function(tag, limit = 0, offset = 0, enrich = false){ + const self = this; + return this.db.sMembers(this.id + "tag" + this.field + ":" + tag).then(function(ids){ + if(!ids || !ids.length || offset >= ids.length) return []; + if(!limit && !offset) return ids; + const result = ids.slice(offset, offset + limit); + return enrich + ? self.enrich(result) + : result; + }); +}; + +RedisDB.prototype.enrich = function(ids){ + if(typeof ids !== "object"){ + ids = [ids]; + } + return this.db.hmGet(this.id + "doc", ids).then(function(res){ + for(let i = 0; i < res.length; i++){ + res[i] = { + id: ids[i], + doc: res[i] && JSON.parse(res[i]) + }; + } + return res; + }); +}; + +RedisDB.prototype.has = function(id){ + return this.db.sIsMember(this.id + "reg", "" + id); +}; + +RedisDB.prototype.search = function(flexsearch, query, limit = 100, offset = 0, suggest = false, resolve = true, enrich = false, tags){ + + let result; + + if(query.length > 1 && flexsearch.depth){ + + const key = this.id + "ctx" + this.field + ":"; + let params = []; + let keyword = query[0]; + let term; + + for(let i = 1, swap; i < query.length; i++){ + term = query[i]; + swap = flexsearch.bidirectional && (term > keyword); + params.push(key + (swap ? term : keyword) + ":" + (swap ? keyword : term)); + keyword = term; + } + query = params; + } + else{ + + const key = this.id + "map" + this.field + ":"; + for(let i = 0; i < query.length; i++){ + query[i] = key + query[i]; + } + } + + const type = this.type; + let key = this.id + "tmp:" + Math.random(); + const strict_tag_intersection = false; + + if(suggest){ + if(!strict_tag_intersection){ + if(tags) for(let i = 0; i < tags.length; i += 2){ + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + } + const multi = this.db.multi().zUnionStore(key, query, { AGGREGATE: "SUM" }); + // Strict Tag Intersection: it does not put tags into union, instead it calculates the + // intersection against the term match union. This was the default behavior + // of Tag-Search. But putting everything into union will also provide suggestions derived + // from tags when no term was matched. + if(strict_tag_intersection){ + if(tags){ + // copy over zInterStore into the same destination surprisingly works fine + // const key2 = key + ":2"; + query = [key]; + for(let i = 0; i < tags.length; i += 2){ + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + multi.zInterStore(key, query, { AGGREGATE: "SUM" }); + // .unlink(key) + // key = key2; + } + } + result = multi + [(resolve ? "zRange" : "zRangeWithScores")](key, "" + offset, "" + (offset + limit - 1), { REV: true }) + .unlink(key) + .exec(); + } + else{ + if(tags) for(let i = 0; i < tags.length; i+=2){ + query.push(this.id + "tag-" + sanitize(tags[i]) + ":" + tags[i + 1]); + } + result = this.db.multi() + .zInterStore(key, query, { AGGREGATE: "MIN" }) + [(resolve ? "zRange" : "zRangeWithScores")](key, "" + offset, "" + (offset + limit - 1), { REV: true }) + .unlink(key) + .exec(); + } + + const self = this; + return result.then(async function(range){ + range = suggest && tags + // take the 3rd result from batch return + ? range[2] + // take the 2nd result from batch return + : range[1]; + if(!range.length) return range; + if(enrich) range = await self.enrich(range); + return create_result(range, type, resolve, enrich); + }); +}; + +RedisDB.prototype.info = function(){ + // todo +}; + +RedisDB.prototype.transaction = async function(task, callback){ + + if(TRX){ + return task.call(this, TRX); + } + + TRX = this.db.multi(); + let promise1 = /*await*/ task.call(this, TRX) + let promise2 = TRX.exec(); + TRX = null; + callback && promise.then(callback); + await Promise.all([promise1, promise2]); +}; + +RedisDB.prototype.commit = async function(flexsearch, _replace, _append){ + + // process cleanup tasks + if(_replace){ + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } + else{ + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for(let i = 0, task; i < tasks.length; i++){ + task = tasks[i]; + // there are just removals in the task queue + if(task.clear){ + await this.clear(); + _replace = true; + break; + } + else{ + tasks[i] = "" + task.del; + } + } + if(!_replace){ + if(!_append){ + tasks = tasks.concat(toArray(flexsearch.reg, /* stringify */ true)); + } + tasks.length && await this.remove(tasks); + } + } + + if(!flexsearch.reg.size){ + return; + } + + await this.transaction(function(trx){ + + let refs = new Map(); + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + + let result = []; + for(let j = 0; j < ids.length; j++){ + result.push({ + score: i, + value: "" + ids[j] + }); + } + if(typeof ids[0] === "number"){ + this.type = "number"; + } + + const ref = this.id + "map" + this.field + ":" + key; + trx.zAdd(ref, result); + // if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + // trx.sAdd("ref" + this.field + ":" + ids[j], ref); + // } + if(this.fastupdate) for(let j = 0, id; j < ids.length; j++){ + // Map performs bad when pushing numeric-like values as key + // id = ids[j]; + // let tmp = refs.get(id); + // tmp || refs.set(id, tmp = []); + // tmp.push(ref); + id = ids[j]; + let tmp = refs.get(id); + tmp || refs.set(id, tmp = []); + tmp.push(ref); + } + } + } + } + // if(this.fastupdate) for(let item of refs){ + // const key = item[0]; + // const value = item[1]; + // trx.sAdd("ref" + this.field + ":" + key, value); + // } + if(this.fastupdate) for(const item of refs){ + const key = item[0]; + const value = item[1]; + trx.sAdd(this.id + "ref" + this.field + ":" + key, value); + } + + refs = new Map(); + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + let result = []; + for(let j = 0; j < ids.length; j++){ + result.push({ score: i, value: "" + ids[j] }); + } + if(typeof ids[0] === "number"){ + this.type = "number"; + } + const ref = this.id + "ctx" + this.field + ":" + ctx_key + ":" + key; + trx.zAdd(ref, result); + // if(this.fastupdate) for(let j = 0; j < ids.length; j++){ + // trx.sAdd("ref" + this.field + ":" + ids[j], ref); + // } + if(this.fastupdate) for(let j = 0, id; j < ids.length; j++){ + // Map performs bad when pushing numeric-like values as key + // id = ids[j]; + // let tmp = refs.get(id); + // tmp || refs.set(id, tmp = []); + // tmp.push(ref); + id = ids[j]; + let tmp = refs.get(id); + tmp || refs.set(id, tmp = []); + tmp.push(ref); + } + } + } + } + } + + if(this.fastupdate) for(const item of refs){ + const key = item[0]; + const value = item[1]; + trx.sAdd(this.id + "ref" + this.field + ":" + key, value); + } + + if(flexsearch.store){ + for(const item of flexsearch.store.entries()){ + const id = item[0]; + const doc = item[1]; + doc && trx.hSet(this.id + "doc", "" + id, JSON.stringify(doc)); + } + } + if(!flexsearch.bypass){ + let ids = toArray(flexsearch.reg, /* stringify */ true); + if(ids.length){ + trx.sAdd(this.id + "reg", ids); + } + } + + if(flexsearch.tag){ + for(const item of flexsearch.tag){ + const tag = item[0]; + const ids = item[1]; + if(!ids.length) continue; + let result = []; + // for(let i = 0; i < ids.length; i++){ + // result.push({ + // score: 0, + // value: "" + ids[i] + // }); + // } + if(typeof ids[0] === "number"){ + for(let i = 0; i < ids.length; i++){ + result[i] = "" + ids[i]; + } + } + else{ + result = ids; + } + trx.sAdd(this.id + "tag" + this.field + ":" + tag, result); + } + } + + // TODO + // trx.set(this.id + "cfg" + this.field, JSON.stringify({ + // "encode": typeof flexsearch.encode === "string" ? flexsearch.encode : "", + // "charset": typeof flexsearch.charset === "string" ? flexsearch.charset : "", + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "minlength": flexsearch.minlength, + // "optimize": flexsearch.optimize, + // "fastupdate": flexsearch.fastupdate, + // "encoder": flexsearch.encoder, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // })); + }); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && + flexsearch.tag.clear(); + flexsearch.store && + flexsearch.store.clear(); + flexsearch.document || + flexsearch.reg.clear(); +}; + +RedisDB.prototype.remove = function(ids){ + + if(!ids && ids !== 0){ + return; + } + if(typeof ids !== "object"){ + ids = [ids]; + } + if(!ids.length){ + return; + } + + return this.transaction(async function(trx){ + + while(ids.length){ + let next; + if(ids.length > 10000){ + next = ids.slice(10000); + ids = ids.slice(0, 10000); + } + + if(typeof ids[0] === "number"){ + for(let i = 0; i < ids.length; i++){ + ids[i] = "" + ids[i]; + } + } + + const check = await this.db.smIsMember(this.id + "reg", ids); + + for(let i = 0, id; i < ids.length; i++){ + if(!check[i]) continue; + id = "" + ids[i]; + + if(this.fastupdate){ + // const refs = new Map(); + const ref = await this.db.sMembers(this.id + "ref" + this.field + ":" + id); + if(ref){ + for(let j = 0; j < ref.length; j++){ + // let tmp = refs.get(ref[j]); + // tmp || refs.set(ref[j], tmp = []); + // tmp.push(id); + trx.zRem(ref[j], id); + } + trx.unlink(this.id + "ref" + this.field + ":" + id); + } + // for(let item of refs){ + // //console.log(item[0], item[1]) + // trx.zRem(item[0], item[1]); + // } + } + else{ + + // todo scan + } + + trx.hDel(this.id + "doc", id); + trx.sRem(this.id + "reg", id); + } + + if(next) ids = next; + else break; + } + }); +}; diff --git a/src/db/redis/package.json b/src/db/redis/package.json new file mode 100644 index 0000000..134d836 --- /dev/null +++ b/src/db/redis/package.json @@ -0,0 +1,16 @@ +{ + "public": true, + "preferGlobal": false, + "name": "flexsearch-redis", + "version": "0.1.0", + "main": "index.js", + "scripts": {}, + "files": [ + "index.js", + "README.md", + "LICENSE" + ], + "dependencies": { + "redis": "^4.7.0" + } +} diff --git a/src/db/sqlite/README.md b/src/db/sqlite/README.md new file mode 100644 index 0000000..c4ea44a --- /dev/null +++ b/src/db/sqlite/README.md @@ -0,0 +1,115 @@ +# SQLite FlexSearch + +SQLite offers you a compact and resource-friendly database which stores right into a portable db file on your filesystem. + +You'll need to install the npm package `sqlite3` into your project: +```bash +npm install sqlite3@5.1.7 +``` + +Create an index and assign a SQLite storage adapter to it by using `index.mount(db)`: + +```js +import Index from "./index.js"; +import Database from "./db/sqlite/index.js"; + +// create an index +const index = new Index(); +// create db instance with optional prefix +const db = new Database("my-store"); +// mount and await before transfering data +await index.mount(db); + +// update the index as usual +index.add(1, "content..."); +index.update(2, "content..."); +index.remove(3); + +// changes are automatically committed by default +// when you need to wait for the task completion, then you +// can use the commit method explicitely: +await index.commit(); +``` + +> Changes are automatically committed by default when you need to wait for the task completion, then you can use the `await index.commit()` method explicitely. You can disable the auto-commit feature optionally. + +## Configuration + +### Custom Filepath + +```js +// set the filepath without passing a name: +const db = new Database({ + path: "./path-to-db/main.db" +}); + +// ... +await index.mount(db); +``` + +For any instance you made from type `DocumentIndex`, `WorkerIndex` or `Index` (as standalone) you'll need to create a `Database` instance having its own path. + +### Custom DB Instance + +Pass a valid `sqlite3` instance on creation: + +```js +import sqlite3 from "sqlite3"; +import Database from "./db/sqlite/index.js"; + +// assume you've created a custom database instance... +const database = new sqlite3.Database("./path-to-db/main.db"); + +// pass database instance as option +const db = new Database("my-store", { + db: database +}); + +// ... +await index.mount(db); +``` + +### ID Type + +The SQLite driver does not properly support upgrading a key field type by ALTER TABLE. Therefore, the default type for ID is `text`. + +You can save required disk space and also gain performance when defining a numeric ID type expicitely. + +```js +// pass type in options +const db = new Database("my-store", { + type: "integer" +}); +``` + +BigInt Range: + +```js +const db = new Database("my-store", { + type: "bigint" +}); +``` + +To change the ID type later you'll need to delete and re-create the index by calling `index.db.destroy()`. + +### In-Memory Engine + +You can switch to SQLite native In-Memory store by passing `:memory:` as a name: + +```js +const db = new Database(":memory:"); +``` + +## Table Structure + +FlexSearch is creating different files for each index name e.g. "my-store". That is how you can use different stores for different indexes at the same time without getting collision of naming inheritance. Document Indexes will map their field names into table names respectively. + +``` +FILENAME + |__ MAIN + |__TABLES map:field (FlexSearch Data) + |__TABLES ctx:field (FlexSearch Data) + |__TABLES tag:field (FlexSearch Data) + |__TABLES cfg:field (FlexSearch Data) + |__TABLES reg (FlexSearch Data) +``` diff --git a/src/db/sqlite/index.js b/src/db/sqlite/index.js new file mode 100644 index 0000000..2785ca2 --- /dev/null +++ b/src/db/sqlite/index.js @@ -0,0 +1,817 @@ +// COMPILER BLOCK --> +import { DEBUG } from "../../config.js"; +// <-- COMPILER BLOCK + +//const sqlite3 = require("sqlite3").verbose(); +import sqlite3 from "sqlite3"; +import path from "path"; +import StorageInterface from "../interface.js"; +import { concat, toArray } from "../../common.js"; +import Document from "../../document.js"; + +const VERSION = 1; +const MAXIMUM_QUERY_VARS = 16000; +const fields = ["map", "ctx", "reg", "tag", "cfg"]; +const types = { + "text": "text", + "char": "text", + "varchar": "text", + "string": "text", + "number": "int", + "numeric": "int", + "integer": "int", + "smallint": "int", + "tinyint": "int", + "mediumint": "int", + "int": "int", + "int8": "int", + "uint8": "int", + "int16": "int", + "uint16": "int", + "int32": "int", + "uint32": "bigint", + "int64": "bigint", + "bigint": "bigint" +}; + +function sanitize(str) { + return str.toLowerCase().replace(/[^a-z0-9_]/g, ""); +} + +// global transaction to keep track of database lock +const TRX = Object.create(null); +const DB = Object.create(null); + +/** + * @constructor + * @implements StorageInterface + */ + +export default function SqliteDB(name, config = {}){ + if(!(this instanceof SqliteDB)){ + return new SqliteDB(name, config); + } + if(typeof name === "object"){ + name = name.name; + config = name; + } + if(!name){ + console.info("Default storage space was used, because a name was not passed."); + } + //field = "Test-456"; + this.id = config.path || ( + name === ":memory:" + ? name + : "flexsearch" + (name ? "-" + sanitize(name) : "") + ".sqlite" + ); + this.field = config.field ? "_" + sanitize(config.field) : ""; + this.support_tag_search = true; + this.db = config.db || DB[this.id] || null; + this.type = config.type ? types[config.type.toLowerCase()] : "string"; + if(!this.type) throw new Error("Unknown type of ID '" + config.type + "'"); +}; + +SqliteDB.prototype.mount = function(flexsearch){ + if(flexsearch instanceof Document){ + return flexsearch.mount(this); + } + flexsearch.db = this; + return this.open(); +}; + +SqliteDB.prototype.open = async function(){ + + if(!this.db){ + + if(!(this.db = DB[this.id])){ + + let filepath = this.id; + if(filepath !== ":memory:"){ + // skip absolute path + if(filepath[0] !== "/" && filepath[0] !== "\\"){ + // current working directory + const dir = process.cwd(); + filepath = path.join(dir, this.id); + } + } + + this.db = DB[this.id] = new sqlite3.Database(filepath); + } + } + + const db = this.db; + + for(let i = 0; i < fields.length; i++){ + const exist = await this.promisfy({ + method: "get", + stmt: "SELECT EXISTS(SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?) as exist", + params: [fields[i] + (fields[i] === "reg" ? "" : this.field)] + }); + if(!exist || !exist.exist){ + let stmt, stmt_index; + switch(fields[i]){ + case "map": + stmt = ` + CREATE TABLE IF NOT EXISTS main.map${this.field}( + key TEXT NOT NULL, + res INTEGER NOT NULL, + id ${this.type} NOT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS map_key_index${this.field} + ON map${this.field} (key); + CREATE INDEX IF NOT EXISTS map_id_index${this.field} + ON map${this.field} (id); + `; + break; + + case "ctx": + stmt = ` + CREATE TABLE IF NOT EXISTS main.ctx${this.field}( + ctx TEXT NOT NULL, + key TEXT NOT NULL, + res INTEGER NOT NULL, + id ${this.type} NOT NULL + ); + + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS ctx_key_index${this.field} + ON ctx${this.field} (ctx, key); + CREATE INDEX IF NOT EXISTS ctx_id_index${this.field} + ON ctx${this.field} (id); + `; + break; + + case "tag": + stmt = ` + CREATE TABLE IF NOT EXISTS main.tag${this.field}( + tag TEXT NOT NULL, + id ${this.type} NOT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS tag_index${this.field} + ON tag${this.field} (tag); + CREATE INDEX IF NOT EXISTS tag_id_index${this.field} + ON tag${this.field} (id); + `; + break; + + case "reg": + stmt = ` + CREATE TABLE IF NOT EXISTS main.reg( + id ${this.type} NOT NULL + CONSTRAINT reg_pk${this.field} + PRIMARY KEY, + doc TEXT DEFAULT NULL + ); + `; + stmt_index = ` + CREATE INDEX IF NOT EXISTS reg_index + ON reg (id); + `; + break; + + case "cfg": + stmt = ` + CREATE TABLE IF NOT EXISTS main.cfg${this.field} ( + cfg TEXT NOT NULL + ); + `; + break; + } + + await new Promise(function(resolve, reject){ + db.exec(stmt, function(err, rows){ + if(err) return reject(err); + stmt_index + ? db.exec(stmt_index, function(err, rows){ + if(err) return reject(err); + resolve(rows); + }) + : resolve(rows); + }); + }); + } + } + + db.exec("PRAGMA optimize = 0x10002"); + + return db; +}; + +SqliteDB.prototype.close = function(){ + this.db.close(); + this.db = null; + return this; +}; + +SqliteDB.prototype.destroy = async function(){ + await this.transaction(function(){ + this.db.run("DROP TABLE IF EXISTS main.map" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.ctx" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.tag" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.cfg" + this.field + ";"); + this.db.run("DROP TABLE IF EXISTS main.reg;"); + }); + this.close(); +}; + +SqliteDB.prototype.clear = function(){ + return this.transaction(function(){ + this.db.run("DELETE FROM main.map" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.ctx" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.tag" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.cfg" + this.field + " WHERE 1;"); + this.db.run("DELETE FROM main.reg WHERE 1;"); + }); +}; + +function create_result(rows, resolve, enrich){ + if(resolve){ + for(let i = 0; i < rows.length; i++){ + if(enrich){ + if(rows[i].doc){ + rows[i].doc = JSON.parse(rows[i].doc); + } + } + else{ + rows[i] = rows[i].id; + } + } + return rows; + } + else{ + const arr = []; + for(let i = 0, row; i < rows.length; i++){ + row = rows[i]; + arr[row.res] || (arr[row.res] = []); + arr[row.res].push(enrich + ? row + : row.id + ); + } + return arr; + } +} + +SqliteDB.prototype.get = function(key, ctx, limit = 0, offset = 0, resolve = true, enrich = false, tags){ + let result; + let stmt = ''; + let params = ctx ? [ctx, key] : [key]; + let table = "main." + (ctx ? "ctx" : "map") + this.field; + if(tags){ + for(let i = 0; i < tags.length; i+=2){ + stmt += ` AND ${ table }.id IN (SELECT id FROM main.tag_${ sanitize(tags[i]) } WHERE tag = ?)`; + params.push(tags[i + 1]); + } + } + if(ctx){ + result = this.promisfy({ + method: "all", + stmt: ` + SELECT ${ table }.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${ table }.id + ` : "" } + WHERE ctx = ? AND key = ? ${stmt} + ORDER BY res + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, + params + }); + } + else{ + result = this.promisfy({ + method: "all", + stmt: ` + SELECT ${ table }.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${ table }.id + ` : "" } + WHERE key = ? ${stmt} + ORDER BY res + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, + params + }); + } + return result.then(function(rows){ + return create_result(rows, resolve, enrich); + }); +}; + +SqliteDB.prototype.tag = function(tag, limit = 0, offset = 0, enrich = false){ + const table = "main.tag" + this.field; + const promise = this.promisfy({ + method: "all", + stmt: ` + SELECT ${ table }.id + ${ enrich ? ", doc" : "" } + FROM ${ table } + ${ enrich ? ` + LEFT JOIN main.reg ON main.reg.id = ${ table }.id + ` : "" } + WHERE tag = ? + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, + params: [tag] + }); + enrich || promise.then(function(rows){ + return create_result(rows, true, false); + }); + return promise; +}; + +function build_params(length){ + + let stmt = ",?"; + for(let i = 1; i < length;){ + if(i <= (length - i)){ + stmt += stmt; + i *= 2; + } + else{ + stmt += stmt.substring(0, (length - i) * 2); + break; + } + } + return stmt.substring(1); +} + +SqliteDB.prototype.enrich = function(ids){ + + const result = []; + const promises = []; + if(typeof ids !== "object"){ + ids = [ids]; + } + + for(let count = 0; count < ids.length;){ + + const chunk = ids.length - count > MAXIMUM_QUERY_VARS + ? ids.slice(count, count + MAXIMUM_QUERY_VARS) + : count ? ids.slice(count) : ids; + count += chunk.length; + + // let stmt = "?"; + // for(let i = 1; i < chunk.length; i++){ + // stmt += ",?"; + // } + + // 10x faster string concatenation + let stmt = build_params(chunk.length); + + promises.push(this.promisfy({ + method: "all", + stmt: `SELECT id, doc FROM main.reg WHERE id IN (${stmt})`, + params: chunk + })); + } + + return Promise.all(promises).then(function(promises){ + + for(let i = 0, res; i < promises.length; i++){ + res = promises[i]; + if(res && res.length){ + for(let i = 0, doc; i < res.length; i++){ + if((doc = res[i].doc)){ + res[i].doc = JSON.parse(doc); + } + } + result.push(res); + } + } + + return result.length === 1 + ? result[0] + : result.length > 1 + ? concat(result) + : result; + }); +}; + +SqliteDB.prototype.has = function(id){ + return this.promisfy({ + method: "get", + stmt: `SELECT EXISTS(SELECT 1 FROM main.reg WHERE id = ?) as exist`, + params: [id] + }).then(function(result){ + return result && result.exist; + }); +}; + +SqliteDB.prototype.search = function(flexsearch, query, limit = 100, offset = 0, suggest = false, resolve = true, enrich = false, tags){ + + let rows; + + if(query.length > 1 && flexsearch.depth){ + + let stmt = ""; + let params = []; + let keyword = query[0]; + let term; + + for(let i = 1; i < query.length; i++){ + term = query[i]; + const swap = flexsearch.bidirectional && (term > keyword); + stmt += (stmt ? " OR " : "") + `(ctx = ? AND key = ?)` + params.push(swap ? term : keyword, swap ? keyword : term); + keyword = term; + } + + if(tags){ + stmt = "(" + stmt + ")"; + for(let i = 0; i < tags.length; i+=2){ + stmt += ` AND id IN (SELECT id FROM main.tag_${ sanitize(tags[i]) } WHERE tag = ?)`; + params.push(tags[i + 1]); + } + } + + rows = this.promisfy({ + method: "all", + stmt: ` + SELECT r.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ( + SELECT id, count(*) as count, + ${ suggest ? "SUM" : "MIN" }(res) as res + FROM main.ctx${ this.field } + WHERE ${ stmt } + GROUP BY id + ) as r + ${ enrich ? ` + LEFT JOIN main.reg ON main.reg.id = r.id + ` : "" } + ${ suggest ? "" : "WHERE count = " + (query.length - 1) } + ORDER BY ${ suggest ? "count DESC, res" : "res" } + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, + params + }); + + // variant 1 + // for(let i = 1; i < query.length; i++){ + // stmt += (stmt ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM main.ctx${this.field} + // WHERE ctx = ? AND key = ? + // `; + // term = query[i]; + // const swap = flexsearch.bidirectional && (term > keyword); + // params.push(swap ? term : keyword, swap ? keyword : term); + // keyword = term; + // } + // + // rows = await this.promisfy({ + // method: "all", + // stmt: ` + // SELECT id/*, res */ + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${stmt}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${suggest ? "" : "WHERE count = " + (query.length - 1)} + // `, + // params + // }); + } + else{ + + let stmt = ""; + let query_length = query.length; + for(let i = 0; i < query_length; i++){ + stmt += (stmt ? " OR " : "") + `key = ?` + } + + if(tags){ + stmt = "(" + stmt + ")"; + for(let i = 0; i < tags.length; i+=2){ + stmt += ` AND id IN (SELECT id FROM main.tag_${sanitize(tags[i])} WHERE tag = ?)`; + query.push(tags[i + 1]); + } + } + + rows = this.promisfy({ + method: "all", + stmt: ` + SELECT r.id + ${ resolve ? "" : ", res" } + ${ enrich ? ", doc" : "" } + FROM ( + SELECT id, count(*) as count, + ${ suggest ? "SUM" : "MIN" }(res) as res + FROM main.map${ this.field } + WHERE ${ stmt } + GROUP BY id + ) as r + ${ enrich ? ` + LEFT JOIN main.reg ON main.reg.id = r.id + ` : "" } + ${ suggest ? "" : "WHERE count = " + query_length } + ORDER BY ${ suggest ? "count DESC, res" : "res" } + ${ limit ? "LIMIT " + limit : "" } + ${ offset ? "OFFSET " + offset : "" } + `, + params: query + }); + + // variant 1 + // for(let i = 0; i < query.length; i++){ + // stmt += (stmt ? " UNION ALL " : "") + ` + // SELECT id, res + // FROM main.map${ this.field } + // WHERE key = ? + // `; + // } + // + // rows = await this.promisfy({ + // method: "all", + // stmt: ` + // SELECT id/*, res*/ + // FROM ( + // SELECT id, ${suggest ? "SUM" : "MIN"}(res) as res, count(*) as count + // FROM (${stmt}) as t + // GROUP BY id + // ORDER BY ${suggest ? "count DESC, res" : "res"} + // LIMIT ${limit} + // OFFSET ${offset} + // ) as r + // ${ suggest ? "" : "WHERE count = " + query.length } + // `, + // params: query + // }); + } + + return rows.then(function(rows){ + return create_result(rows, resolve, enrich); + }); +}; + +SqliteDB.prototype.info = function(){ + // todo +}; + +SqliteDB.prototype.transaction = function(task, callback){ + + const self = this; + + if(TRX[this.id]){ + return TRX[this.id].then(function(){ + return self.transaction(task, callback); + }); + } + + const db = this.db; + + return TRX[this.id] = new Promise(function(resolve, reject){ + db.exec("PRAGMA optimize"); + db.exec('PRAGMA busy_timeout = 5000'); + db.exec("BEGIN"); + db.parallelize(function(){ + task.call(self); + }); + db.exec("COMMIT", function(err, rows){ + TRX[self.id] = null; + if(err) return reject(err); + callback && callback(rows); + resolve(rows); + db.exec("PRAGMA shrink_memory"); + }); + }); +}; + +SqliteDB.prototype.commit = async function(flexsearch, _replace, _append){ + + // process cleanup tasks + if(_replace){ + await this.clear(); + // there are just removals in the task queue + flexsearch.commit_task = []; + } + else{ + let tasks = flexsearch.commit_task; + flexsearch.commit_task = []; + for(let i = 0, task; i < tasks.length; i++){ + task = tasks[i]; + // there are just removals in the task queue + if(task.clear){ + await this.clear(); + _replace = true; + break; + } + else{ + tasks[i] = task.del; + } + } + if(!_replace){ + if(!_append){ + tasks = tasks.concat(toArray(flexsearch.reg)); + } + tasks.length && await this.remove(tasks); + } + } + + if(!flexsearch.reg.size){ + return; + } + + await this.transaction(function(){ + + for(const item of flexsearch.map){ + const key = item[0]; + const arr = item[1]; + + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + let stmt = ""; + let params = []; + + for(let j = 0; j < ids.length; j++){ + stmt += (stmt ? "," : "") + "(?,?,?)"; + params.push(key, i, ids[j]); + // maximum count of variables supported + if((j === ids.length - 1) || (params.length + 3 > MAXIMUM_QUERY_VARS)){ + this.db.run("INSERT INTO main.map" + this.field + " (key, res, id) VALUES " + stmt, params); + stmt = ""; + params = []; + } + //this.db.run("INSERT INTO map (key, res, id) VALUES (?, ?, ?) ON CONFLICT DO NOTHING", [key, i, ids[j]]); + } + } + } + } + //}); + //await this.transaction(function(){ + + for(const ctx of flexsearch.ctx){ + const ctx_key = ctx[0]; + const ctx_value = ctx[1]; + + for(const item of ctx_value){ + const key = item[0]; + const arr = item[1]; + + for(let i = 0, ids; i < arr.length; i++){ + if((ids = arr[i]) && ids.length){ + let stmt = ""; + let params = []; + + for(let j = 0; j < ids.length; j++){ + stmt += (stmt ? "," : "") + "(?,?,?,?)"; + params.push(ctx_key, key, i, ids[j]); + // maximum count of variables supported + if((j === ids.length - 1) || (params.length + 4 > MAXIMUM_QUERY_VARS)){ + this.db.run("INSERT INTO main.ctx" + this.field + " (ctx, key, res, id) VALUES " + stmt, params); + stmt = ""; + params = []; + } + } + } + } + } + } + //}); + //await this.transaction(function(){ + + if(flexsearch.store){ + let stmt = ""; + let chunk = []; + for(const item of flexsearch.store.entries()){ + const id = item[0]; + const doc = item[1]; + stmt += (stmt ? "," : "") + "(?,?)"; + chunk.push(id, typeof doc === "object" + ? JSON.stringify(doc) + : doc || null + ); + if(chunk.length + 2 > MAXIMUM_QUERY_VARS){ + this.db.run("INSERT INTO main.reg (id, doc) VALUES " + stmt, chunk); + stmt = ""; + chunk = []; + } + } + if(chunk.length){ + this.db.run("INSERT INTO main.reg (id, doc) VALUES " + stmt, chunk); + } + } + else if(!flexsearch.bypass){ + let ids = toArray(flexsearch.reg); + for(let count = 0; count < ids.length;){ + const chunk = ids.length - count > MAXIMUM_QUERY_VARS + ? ids.slice(count, count + MAXIMUM_QUERY_VARS) + : count ? ids.slice(count) : ids; + count += chunk.length; + const stmt = build_params(chunk.length); + this.db.run("INSERT INTO main.reg (id) VALUES (" + stmt + ")", chunk); + } + } + //}); + //await this.transaction(function(){ + + if(flexsearch.tag){ + let stmt = ""; + let chunk = []; + for(const item of flexsearch.tag){ + const tag = item[0]; + const ids = item[1]; + if(!ids.length) continue; + for(let i = 0; i < ids.length; i++){ + stmt += (stmt ? "," : "") + "(?,?)"; + chunk.push(tag, ids[i]); + } + if(chunk.length + 2 > MAXIMUM_QUERY_VARS){ + this.db.run("INSERT INTO main.tag" + this.field + " (tag, id) VALUES " + stmt, chunk); + stmt = ""; + chunk = []; + } + } + if(chunk.length){ + this.db.run("INSERT INTO main.tag" + this.field + " (tag, id) VALUES " + stmt, chunk); + } + } + }); + + // TODO + //await this.transaction(function(){ + // this.db.run("INSERT INTO main.cfg" + this.field + " (cfg) VALUES (?)", [JSON.stringify({ + // "charset": flexsearch.charset, + // "tokenize": flexsearch.tokenize, + // "resolution": flexsearch.resolution, + // "fastupdate": flexsearch.fastupdate, + // "compress": flexsearch.compress, + // "encoder": { + // "minlength": flexsearch.encoder.minlength + // }, + // "context": { + // "depth": flexsearch.depth, + // "bidirectional": flexsearch.bidirectional, + // "resolution": flexsearch.resolution_ctx + // } + // })]); + //}); + + flexsearch.map.clear(); + flexsearch.ctx.clear(); + flexsearch.tag && + flexsearch.tag.clear(); + flexsearch.store && + flexsearch.store.clear(); + flexsearch.document || + flexsearch.reg.clear(); +}; + +SqliteDB.prototype.remove = function(ids){ + + if(typeof ids !== "object"){ + ids = [ids]; + } + + let next; + // maximum count of variables supported + if(ids.length > MAXIMUM_QUERY_VARS){ + next = ids.slice(MAXIMUM_QUERY_VARS); + ids = ids.slice(0, MAXIMUM_QUERY_VARS); + } + + const self = this; + return this.transaction(function(){ + const stmt = build_params(ids.length); + this.db.run("DELETE FROM main.map" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.ctx" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.tag" + self.field + " WHERE id IN (" + stmt + ")", ids); + this.db.run("DELETE FROM main.reg WHERE id IN (" + stmt + ")", ids); + }).then(function(res){ + return next + ? self.remove(next) + : res; + }); +}; + +SqliteDB.prototype.promisfy = function(opt){ + const db = this.db; + return new Promise(function(resolve, reject){ + db[opt.method](opt.stmt, opt.params || [], function(err, rows){ + opt.callback && opt.callback(rows); + err ? reject(err) + : resolve(rows); + }); + }); +}; diff --git a/src/db/sqlite/package.json b/src/db/sqlite/package.json new file mode 100644 index 0000000..314ed53 --- /dev/null +++ b/src/db/sqlite/package.json @@ -0,0 +1,16 @@ +{ + "public": true, + "preferGlobal": false, + "name": "flexsearch-sqlite", + "version": "0.1.0", + "main": "index.js", + "scripts": {}, + "files": [ + "index.js", + "README.md", + "LICENSE" + ], + "dependencies": { + "sqlite3": "^5.1.7" + } +} diff --git a/src/document.js b/src/document.js index 9bb16c8..94735dc 100644 --- a/src/document.js +++ b/src/document.js @@ -8,90 +8,202 @@ // COMPILER BLOCK --> import { - + DEBUG, SUPPORT_ASYNC, SUPPORT_CACHE, + SUPPORT_KEYSTORE, + SUPPORT_PERSISTENT, SUPPORT_SERIALIZE, SUPPORT_STORE, SUPPORT_TAGS, SUPPORT_WORKER - } from "./config.js"; // <-- COMPILER BLOCK +import { DocumentOptions } from "./type.js"; import Index from "./index.js"; -import { DocumentInterface } from "./type.js"; -import Cache, { searchCache } from "./cache.js"; -import { create_object, is_array, is_string, is_object, parse_option, get_keys } from "./common.js"; -import apply_async from "./async.js"; -import { intersect, intersect_union } from "./intersect.js"; -import { exportDocument, importDocument } from "./serialize.js"; import WorkerIndex from "./worker/index.js"; +import Cache, { searchCache } from "./cache.js"; +import { is_string, is_object, parse_simple } from "./common.js"; +import apply_async from "./async.js"; +import { exportDocument, importDocument } from "./serialize.js"; +import { KeystoreMap, KeystoreSet } from "./keystore.js"; +import "./document/add.js"; +import "./document/search.js"; /** * @constructor - * @implements {DocumentInterface} - * @param {Object=} options - * @return {Document} + * @param {DocumentOptions=} options */ -function Document(options){ +export default function Document(options){ if(!(this instanceof Document)) { - return new Document(options); } - const document = options["document"] || options["doc"] || options; - let opt; + const document = options.document || options.doc || options; + let tmp, keystore; this.tree = []; this.field = []; this.marker = []; - this.register = create_object(); - this.key = ((opt = document["key"] || document["id"]) && parse_tree(opt, this.marker)) || "id"; - this.fastupdate = parse_option(options["fastupdate"], true); + this.key = ((tmp = document.key || document.id) && parse_tree(tmp, this.marker)) || "id"; + + keystore = SUPPORT_KEYSTORE && (options.keystore || 0); + keystore && (this.keystore = keystore); + this.fastupdate = !!options.fastupdate; + this.reg = this.fastupdate + ? (keystore ? new KeystoreMap(keystore) : new Map()) + : (keystore ? new KeystoreSet(keystore) : new Set()); if(SUPPORT_STORE){ - - this.storetree = (opt = document["store"]) && (opt !== true) && []; - this.store = opt && create_object(); - } - - if(SUPPORT_TAGS){ - - // TODO case-insensitive tags - - this.tag = ((opt = document["tag"]) && parse_tree(opt, this.marker)); - this.tagindex = opt && create_object(); + // todo support custom filter function + this.storetree = (tmp = document.store || null) && tmp !== true && []; + this.store = tmp && ( + keystore + ? new KeystoreMap(keystore) + : new Map() + ); } if(SUPPORT_CACHE){ - - this.cache = (opt = options["cache"]) && new Cache(opt); - - // do not apply cache again for the indexes - - options["cache"] = false; + this.cache = (tmp = options.cache || null) && new Cache(tmp); + // do not apply cache again for the indexes since .searchCache() + // is just a wrapper over .search() + options.cache = false; } if(SUPPORT_WORKER){ - - this.worker = options["worker"]; + this.worker = options.worker; } if(SUPPORT_ASYNC){ - // this switch is used by recall of promise callbacks - this.async = false; } /** @export */ this.index = parse_descriptor.call(this, options, document); + + if(SUPPORT_TAGS){ + this.tag = null; + // TODO case-insensitive tags? + if((tmp = document.tag)){ + if(typeof tmp === "string"){ + tmp = [tmp]; + } + if(tmp.length){ + this.tag = new Map(); + this.tagtree = []; + this.tagfield = []; + for(let i = 0, params, field; i < tmp.length; i++){ + params = tmp[i]; + field = params.field || params; + if(!field){ + throw new Error("The tag field from the document descriptor is undefined."); + } + if(params.custom){ + this.tagtree[i] = params.custom; + } + else{ + this.tagtree[i] = parse_tree(field, this.marker); + if(params.filter){ + if(typeof this.tagtree[i] === "string"){ + // it needs an object to put a property to it + this.tagtree[i] = new String(this.tagtree[i]); + } + this.tagtree[i]._filter = params.filter; + } + } + // the tag fields needs to be hold by indices + this.tagfield[i] = field; + this.tag.set(field, new Map()); + } + } + } + } + + if(SUPPORT_PERSISTENT){ + options.db && this.mount(options.db); + } } -export default Document; +if(SUPPORT_PERSISTENT){ + + Document.prototype.mount = function(db){ + + let fields = this.field; + + if(SUPPORT_TAGS && this.tag){ + // tag indexes are referenced by field + // move tags to their field indexes respectively + for(let i = 0, field; i < this.tagfield.length; i++){ + field = this.tagfield[i]; + let index = this.index.get(field); + if(!index){ + // create raw index when not exists + this.index.set(field, index = new Index({}, this.reg)); + // copy and push to the field selection + if(fields === this.field){ + fields = fields.slice(0); + } + // tag indexes also needs to be upgraded to db instances + fields.push(field); + } + // assign reference + index.tag = this.tag.get(field); + } + } + + const promises = []; + const config = { + db: db.db, + type: db.type, + fastupdate: db.fastupdate + }; + + // upgrade all indexes to db instances + for(let i = 0, index, field; i < fields.length; i++){ + config.field = field = fields[i]; + index = this.index.get(field); + const dbi = new db.constructor(db.id, config); + // take over the storage id + dbi.id = db.id; + promises[i] = dbi.mount(index); + // add an identification property + index.document = true; + if(i){ + // the register has to export just one time + // also it's needed by the index for ID contain check + index.bypass = true; + } + else if(SUPPORT_STORE){ + // the datastore has to export one time + index.store = this.store; + } + } + + this.async = true; + this.db = true; + return Promise.all(promises); + }; + + Document.prototype.commit = async function(replace, append){ + // parallel: + const promises = []; + for(const index of this.index.values()){ + promises.push(index.db.commit(index, replace, append)); + } + await Promise.all(promises); + this.reg.clear(); + // queued: + // for(const index of this.index.values()){ + // await index.db.commit(index, replace, append); + // } + // this.reg.clear(); + }; +} /** * @this Document @@ -99,11 +211,10 @@ export default Document; function parse_descriptor(options, document){ - const index = create_object(); - let field = document["index"] || document["field"] || document; + const index = new Map(); + let field = document.index || document.field || document; if(is_string(field)){ - field = [field]; } @@ -112,44 +223,64 @@ function parse_descriptor(options, document){ key = field[i]; if(!is_string(key)){ - opt = key; - key = key["field"]; + key = key.field; } opt = is_object(opt) ? Object.assign({}, options, opt) : options; if(SUPPORT_WORKER && this.worker){ - - index[key] = new WorkerIndex(opt); - - if(!index[key].worker){ - + const worker = new WorkerIndex(opt); + index.set(key, worker); + if(!worker.worker){ + // fallback when not supported this.worker = false; } } - if(!this.worker){ - - index[key] = new Index(opt, this.register); + if(!SUPPORT_WORKER || !this.worker){ + index.set(key, new Index(opt, this.reg)); + } + + if(opt.custom){ + this.tree[i] = opt.custom; + } + else{ + this.tree[i] = parse_tree(key, this.marker); + if(opt.filter){ + if(typeof this.tree[i] === "string"){ + // it needs an object to put a property to it + this.tree[i] = new String(this.tree[i]); + } + this.tree[i]._filter = opt.filter; + } } - this.tree[i] = parse_tree(key, this.marker); this.field[i] = key; } if(SUPPORT_STORE && this.storetree){ - let store = document["store"]; + let stores = document.store; + if(is_string(stores)) stores = [stores]; - if(is_string(store)){ - - store = [store]; - } - - for(let i = 0; i < store.length; i++){ - - this.storetree[i] = parse_tree(store[i], this.marker); + for(let i = 0, store, field; i < stores.length; i++){ + store = stores[i]; + field = store.field || store; + if(store.custom){ + this.storetree[i] = store.custom; + store.custom._field = field; + } + else{ + this.storetree[i] = parse_tree(field, this.marker); + if(store.filter){ + if(typeof this.storetree[i] === "string"){ + // it needs an object to put a property to it + this.storetree[i] = new String(this.storetree[i]); + } + this.storetree[i]._filter = store.filter; + } + } } } @@ -162,298 +293,58 @@ function parse_tree(key, marker){ let count = 0; for(let i = 0; i < tree.length; i++){ - key = tree[i]; - - if(key.indexOf("[]") >= 0){ - + // todo apply some indexes e.g. [0], [-1], [0-2] + if(key[key.length - 1] === "]"){ key = key.substring(0, key.length - 2); - if(key){ - marker[count] = true; } } - if(key){ - tree[count++] = key; } } if(count < tree.length){ - tree.length = count; } return count > 1 ? tree : tree[0]; } -// TODO support generic function created from string when tree depth > 1 - -function parse_simple(obj, tree){ - - if(is_string(tree)){ - - obj = obj[tree]; - } - else{ - - for(let i = 0; obj && (i < tree.length); i++){ - - obj = obj[tree[i]]; - } - } - - return obj; -} - -// TODO support generic function created from string when tree depth > 1 - -function store_value(obj, store, tree, pos, key){ - - obj = obj[key]; - - // reached target field - - if(pos === (tree.length - 1)){ - - // store target value - - store[key] = obj; - } - else if(obj){ - - if(is_array(obj)){ - - store = store[key] = new Array(obj.length); - - for(let i = 0; i < obj.length; i++){ - - // do not increase pos (an array is not a field) - store_value(obj, store, tree, pos, i); - } - } - else{ - - store = store[key] || (store[key] = create_object()); - key = tree[++pos]; - - store_value(obj, store, tree, pos, key); - } - } -} - -function add_index(obj, tree, marker, pos, index, id, key, _append){ - - obj = obj[key]; - - if(obj){ - - // reached target field - - if(pos === (tree.length - 1)){ - - // handle target value - - if(is_array(obj)){ - - // append array contents so each entry gets a new scoring context - - if(marker[pos]){ - - for(let i = 0; i < obj.length; i++){ - - index.add(id, obj[i], /* append: */ true, /* skip update: */ true); - } - - return; - } - - // or join array contents and use one scoring context - - obj = obj.join(" "); - } - - index.add(id, obj, _append, /* skip_update: */ true); - } - else{ - - if(is_array(obj)){ - - for(let i = 0; i < obj.length; i++){ - - // do not increase index, an array is not a field - - add_index(obj, tree, marker, pos, index, id, i, _append); - } - } - else{ - - key = tree[++pos]; - - add_index(obj, tree, marker, pos, index, id, key, _append); - } - } - } -} - -/** - * - * @param id - * @param content - * @param {boolean=} _append - * @returns {Document|Promise} - */ - -Document.prototype.add = function(id, content, _append){ - - if(is_object(id)){ - - content = id; - id = parse_simple(content, this.key); - } - - if(content && (id || (id === 0))){ - - if(!_append && this.register[id]){ - - return this.update(id, content); - } - - for(let i = 0, tree, field; i < this.field.length; i++){ - - field = this.field[i]; - tree = this.tree[i]; - - if(is_string(tree)){ - - tree = [tree]; - } - - add_index(content, tree, this.marker, 0, this.index[field], id, tree[0], _append); - } - - if(SUPPORT_TAGS && this.tag){ - - let tag = parse_simple(content, this.tag); - let dupes = create_object(); - - if(is_string(tag)){ - - tag = [tag]; - } - - for(let i = 0, key, arr; i < tag.length; i++){ - - key = tag[i]; - - if(!dupes[key]){ - - dupes[key] = 1; - arr = this.tagindex[key] || (this.tagindex[key] = []); - - if(!_append || !arr.includes(id)){ - - arr[arr.length] = id; - - // add a reference to the register for fast updates - - if(this.fastupdate){ - - const tmp = this.register[id] || (this.register[id] = []); - tmp[tmp.length] = arr; - } - } - } - } - } - - // TODO: how to handle store when appending contents? - - if(SUPPORT_STORE && this.store && (!_append || !this.store[id])){ - - let store; - - if(this.storetree){ - - store = create_object(); - - for(let i = 0, tree; i < this.storetree.length; i++){ - - tree = this.storetree[i]; - - if(is_string(tree)){ - - store[tree] = content[tree]; - } - else{ - - store_value(content, store, tree, 0, tree[0]); - } - } - } - - this.store[id] = store || content; - } - } - - return this; -}; - Document.prototype.append = function(id, content){ - return this.add(id, content, true); }; Document.prototype.update = function(id, content){ - return this.remove(id).add(id, content); }; Document.prototype.remove = function(id){ if(is_object(id)){ - id = parse_simple(id, this.key); } - if(this.register[id]){ + for(const index of this.index.values()){ + index.remove(id, /* skip deletion */ true); + } - for(let i = 0; i < this.field.length; i++){ - - // workers does not share the register - - this.index[this.field[i]].remove(id, !this.worker); - - if(this.fastupdate){ - - // when fastupdate was enabled all ids are removed - - break; - } - } + if(this.reg.has(id)){ if(SUPPORT_TAGS && this.tag){ - // when fastupdate was enabled all ids are already removed - if(!this.fastupdate){ - - for(let key in this.tagindex){ - - const tag = this.tagindex[key]; - const pos = tag.indexOf(id); - - if(pos !== -1){ - - if(tag.length > 1){ - - tag.splice(pos, 1); - } - else{ - - delete this.tagindex[key]; + for(let field of this.tag.values()){ + for(let item of field){ + const tag = item[0]; + const ids = item[1]; + const pos = ids.indexOf(id); + if(pos > -1){ + ids.length > 1 + ? ids.splice(pos, 1) + : field.delete(tag); } } } @@ -461,320 +352,88 @@ Document.prototype.remove = function(id){ } if(SUPPORT_STORE && this.store){ - - delete this.store[id]; + this.store.delete(id); } - delete this.register[id]; + this.reg.delete(id); + } + + // the cache could be used outside the InMemory store + if(SUPPORT_CACHE && this.cache){ + this.cache.remove(id); } return this; }; -/** - * @param {!string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @param {Array=} _resolve For internal use only. - * @returns {Promise|Array} - */ +Document.prototype.clear = function(){ -Document.prototype.search = function(query, limit, options, _resolve){ + //const promises = []; - if(!options){ + for(const index of this.index.values()){ + // db index will add clear task + index.clear(); + // const promise = index.clear(); + // if(promise instanceof Promise){ + // promises.push(promise); + // } + } - if(!limit && is_object(query)){ - - options = /** @type {Object} */ (query); - query = ""; - } - else if(is_object(limit)){ - - options = /** @type {Object} */ (limit); - limit = 0; + if(SUPPORT_TAGS && this.tag){ + for(const tags of this.tag.values()){ + tags.clear(); } } - let result = [], result_field = []; - let pluck, enrich; - let field, tag, bool, offset, count = 0; - - if(options){ - - if(is_array(options)){ - - field = options; - options = null; - } - else{ - - query = options["query"] || query; - pluck = options["pluck"]; - field = pluck || options["index"] || options["field"] /*|| (is_string(options) && [options])*/; - tag = SUPPORT_TAGS && options["tag"]; - enrich = SUPPORT_STORE && this.store && options["enrich"]; - bool = options["bool"] === "and"; - limit = options["limit"] || limit || 100; - offset = options["offset"] || 0; - - if(tag){ - - if(is_string(tag)){ - - tag = [tag]; - } - - // when tags is used and no query was set, - // then just return the tag indexes - - if(!query){ - - for(let i = 0, res; i < tag.length; i++){ - - res = get_tag.call(this, tag[i], limit, offset, enrich); - - if(res){ - - result[result.length] = res; - count++; - } - } - - return count ? result : []; - } - } - - if(is_string(field)){ - - field = [field]; - } - } + if(SUPPORT_STORE && this.store){ + this.store.clear(); } - field || (field = this.field); - bool = bool && ((field.length > 1) || (tag && (tag.length > 1))); - - const promises = !_resolve && (this.worker || this.async) && []; - - // TODO solve this in one loop below - - for(let i = 0, res, key, len; i < field.length; i++){ - - let field_options; - - key = field[i]; - - if(!is_string(key)){ - - field_options = key; - key = field_options["field"]; - query = field_options["query"] || query; - limit = field_options["limit"] || limit; - enrich = field_options["enrich"] || enrich; - } - - if(promises){ - - promises[i] = this.index[key].searchAsync(query, limit, field_options || options); - - // just collect and continue - - continue; - } - else if(_resolve){ - - res = _resolve[i]; - } - else{ - - // inherit options also when search? it is just for laziness, Object.assign() has a cost - - res = this.index[key].search(query, limit, field_options || options); - } - - len = res && res.length; - - if(tag && len){ - - const arr = []; - let count = 0; - - if(bool){ - - // prepare for intersection - - arr[0] = [res]; - } - - for(let y = 0, key, res; y < tag.length; y++){ - - key = tag[y]; - res = this.tagindex[key]; - len = res && res.length; - - if(len){ - - count++; - arr[arr.length] = bool ? [res] : res; - } - } - - if(count){ - - if(bool){ - - res = intersect(arr, limit || 100, offset || 0); - } - else{ - - res = intersect_union(res, arr); - } - - len = res.length; - } - } - - if(len){ - - result_field[count] = key; - result[count++] = res; - } - else if(bool){ - - return []; - } - } - - if(promises){ - - const self = this; - - // anyone knows a better workaround of optionally having async promises? - // the promise.all() needs to be wrapped into additional promise, - // otherwise the recursive callback wouldn't run before return - - return new Promise(function(resolve){ - - Promise.all(promises).then(function(result){ - - resolve(self.search(query, limit, options, result)); - }); - }); - } - - if(!count){ - - // fast path "not found" - - return []; - } - - if(pluck && (!enrich || !this.store)){ - - // fast path optimization - - return result[0]; - } - - for(let i = 0, res; i < result_field.length; i++){ - - res = result[i]; - - if(res.length){ - - if(enrich){ - - res = apply_enrich.call(this, res); - } - } - - if(pluck){ - - return res; - } - - result[i] = { - - "field": result_field[i], - "result": res - }; - } - - return result; + return this; /*promises.length + ? Promise.all(promises) + :*/ }; -/** - * @this Document - */ - -function get_tag(key, limit, offset, enrich){ - - let res = this.tagindex[key]; - let len = res && (res.length - offset); - - if(len && (len > 0)){ - - if((len > limit) || offset){ - - res = res.slice(offset, offset + limit); - } - - if(enrich){ - - res = apply_enrich.call(this, res); - } - - return { - - "tag": key, - "result": res - }; - } -} - -/** - * @this Document - */ - -function apply_enrich(res){ - - const arr = new Array(res.length); - - for(let x = 0, id; x < res.length; x++){ - - id = res[x]; - - arr[x] = { - - "id": id, - "doc": this.store[id] - }; - } - - return arr; -} - Document.prototype.contain = function(id){ - return !!this.register[id]; + if(SUPPORT_PERSISTENT && this.db){ + return this.index.get(this.field[0]).db.has(id); + } + + return this.reg.has(id); +}; + +Document.prototype.cleanup = function(){ + + for(const index of this.index.values()){ + index.cleanup(); + } + + return this; }; if(SUPPORT_STORE){ Document.prototype.get = function(id){ - return this.store[id]; + if(SUPPORT_PERSISTENT && this.db){ + return this.index.get(this.field[0]).db.enrich(id).then(function(result){ + return result[0] && result[0].doc; + }); + } + + return this.store.get(id); }; - Document.prototype.set = function(id, data){ + Document.prototype.set = function(id, store){ - this.store[id] = data; + this.store.set(id, store); return this; }; } if(SUPPORT_CACHE){ - + // todo mo Document.prototype.searchCache = searchCache; } diff --git a/src/document/add.js b/src/document/add.js new file mode 100644 index 0000000..aef0fb0 --- /dev/null +++ b/src/document/add.js @@ -0,0 +1,263 @@ +// COMPILER BLOCK --> +import { + DEBUG, + SUPPORT_KEYSTORE, + SUPPORT_PERSISTENT, + SUPPORT_STORE, + SUPPORT_TAGS +} from "../config.js"; +// <-- COMPILER BLOCK +import { create_object, is_array, is_object, is_string, parse_simple } from "../common.js"; +import { KeystoreArray } from "../keystore.js"; +import Document from "../document.js"; + +/** + * + * @param id + * @param content + * @param {boolean=} _append + * @returns {Document|Promise} + */ + +Document.prototype.add = function(id, content, _append){ + + if(is_object(id)){ + + content = id; + id = parse_simple(content, this.key); + } + + if(content && (id || (id === 0))){ + + if(!_append && this.reg.has(id)){ + return this.update(id, content); + } + + // this.field does not include db tag indexes + for(let i = 0, tree; i < this.field.length; i++){ + + tree = this.tree[i]; + + const index = this.index.get(this.field[i]); + if(typeof tree === "function"){ + const tmp = tree(content); + if(tmp){ + index.add(id, tmp, /* append: */ false, /* skip update: */ true); + } + } + else{ + const filter = tree._filter; + if(filter && !filter(content)){ + continue; + } + if(tree instanceof String){ + tree = ["" + tree]; + } + else if(is_string(tree)){ + tree = [tree]; + } + add_index(content, tree, this.marker, 0, index, id, tree[0], _append); + } + } + + if(SUPPORT_TAGS && this.tag){ + + //console.log(this.tag, this.tagtree) + + for(let x = 0; x < this.tagtree.length; x++){ + + let tree = this.tagtree[x]; + let field = this.tagfield[x]; + let ref = this.tag.get(field); + let dupes = create_object(); + let tags; + + if(typeof tree === "function"){ + tags = tree(content); + if(!tags) continue; + } + else{ + const filter = tree._filter; + if(filter && !filter(content)){ + continue; + } + if(tree instanceof String){ + tree = "" + tree; + } + tags = parse_simple(content, tree); + } + + if(!ref || !tags){ + ref || (DEBUG && console.warn("Tag '" + field + "' was not found")); + continue; + } + + if(is_string(tags)){ + tags = [tags]; + } + + for(let i = 0, tag, arr; i < tags.length; i++){ + + tag = tags[i]; + //console.log(this.tag, tag, key, field) + + if(!dupes[tag]){ + dupes[tag] = 1; + + let tmp; + tmp = ref.get(tag); + tmp ? arr = tmp : ref.set(tag, arr = []); + + if(!_append || ! /** @type {!Array|KeystoreArray} */(arr).includes(id)){ + + // auto-upgrade to keystore array if max size exceeded + if(SUPPORT_KEYSTORE){ + if(arr.length === 2**31-1 /*|| !(arr instanceof KeystoreArray)*/){ + const keystore = new KeystoreArray(arr); + if(this.fastupdate){ + for(let value of this.reg.values()){ + if(value.includes(arr)){ + value[value.indexOf(arr)] = keystore; + } + } + } + ref.set(tag, arr = keystore); + } + } + + arr.push(id); + + // add a reference to the register for fast updates + if(this.fastupdate){ + const tmp = this.reg.get(id); + tmp ? tmp.push(arr) + : this.reg.set(id, [arr]); + } + } + } + } + } + } + + if(SUPPORT_STORE && this.store && (!_append || !this.store.has(id))){ + + let payload; + + if(this.storetree){ + + payload = create_object(); + + for(let i = 0, tree; i < this.storetree.length; i++){ + tree = this.storetree[i]; + + const filter = tree._filter; + if(filter && !filter(content)){ + continue; + } + let custom; + if(typeof tree === "function"){ + custom = tree(content); + if(!custom) continue; + tree = [tree._field]; + } + else if(is_string(tree) || tree instanceof String){ + payload[tree] = content[tree]; + continue; + } + + store_value(content, payload, tree, 0, tree[0], custom); + } + } + + this.store.set(id, payload || content); + } + } + + return this; +}; + +// TODO support generic function created from string when tree depth > 1 + +/** + * @param obj + * @param store + * @param tree + * @param pos + * @param key + * @param {*=} custom + */ + +function store_value(obj, store, tree, pos, key, custom){ + + obj = obj[key]; + + // reached target field + if(pos === (tree.length - 1)){ + + // store target value + store[key] = custom || obj; + } + else if(obj){ + + if(is_array(obj)){ + + store = store[key] = new Array(obj.length); + + for(let i = 0; i < obj.length; i++){ + // do not increase pos (an array is not a field) + store_value(obj, store, tree, pos, i); + } + } + else{ + + store = store[key] || (store[key] = create_object()); + key = tree[++pos]; + store_value(obj, store, tree, pos, key); + } + } +} + +function add_index(obj, tree, marker, pos, index, id, key, _append){ + + if((obj = obj[key])){ + + // reached target field + if(pos === (tree.length - 1)){ + + // handle target value + if(is_array(obj)){ + + // append array contents so each entry gets a new scoring context + if(marker[pos]){ + for(let i = 0; i < obj.length; i++){ + index.add(id, obj[i], /* append: */ true, /* skip update: */ true); + } + return; + } + + // or join array contents and use one scoring context + obj = obj.join(" "); + } + + index.add(id, obj, _append, /* skip_update: */ true); + } + else{ + + if(is_array(obj)){ + for(let i = 0; i < obj.length; i++){ + // do not increase index, an array is not a field + add_index(obj, tree, marker, pos, index, id, i, _append); + } + } + else{ + key = tree[++pos]; + add_index(obj, tree, marker, pos, index, id, key, _append); + } + } + } + else{ + if(SUPPORT_PERSISTENT && index.db){ + index.remove(id); + } + } +} diff --git a/src/document/search.js b/src/document/search.js new file mode 100644 index 0000000..0be4fc7 --- /dev/null +++ b/src/document/search.js @@ -0,0 +1,485 @@ +// COMPILER BLOCK --> +import { + DEBUG, + SUPPORT_PERSISTENT, + SUPPORT_STORE, + SUPPORT_SUGGESTION, + SUPPORT_TAGS +} from "../config.js"; +// <-- COMPILER BLOCK + +import { DocumentSearchOptions } from "../type.js"; +import { create_object, is_array, is_object, is_string } from "../common.js"; +import { intersect_union } from "../intersect.js"; +import Document from "../document.js"; + +let debug = false; + +/** + * @param {!string|DocumentSearchOptions} query + * @param {number|DocumentSearchOptions=} limit + * @param {DocumentSearchOptions=} options + * @param {Array=} _resolve For internal use only. + * @returns {Promise|Array} + */ + +Document.prototype.search = function(query, limit, options, _resolve){ + + debug && console.log("checkoint:search", !!_resolve); + + if(!options){ + if(!limit && is_object(query)){ + options = /** @type {DocumentSearchOptions} */ (query); + query = ""; + } + else if(is_object(limit)){ + options = /** @type {DocumentSearchOptions} */ (limit); + limit = 0; + } + } + + let result = [], result_field = []; + let pluck, enrich, merge, suggest; + let field, tag, offset, count = 0; + + if(options){ + + // todo remove support? + if(is_array(options)){ + field = options; + options = null; + } + else{ + + query = options.query || query; + pluck = options.pluck; + merge = options.merge; + field = pluck || options.field || options.index; + tag = SUPPORT_TAGS && this.tag && options.tag; + enrich = SUPPORT_STORE && this.store && options.enrich; + suggest = SUPPORT_SUGGESTION && options.suggest; + limit = options.limit || limit; + offset = options.offset || 0; + limit || (limit = 100); + + if(tag && (!SUPPORT_PERSISTENT || !this.db || !_resolve)){ + + // Tag-Search + // ----------------------------- + + debug && console.log("checkoint:search:tag"); + + if(tag.constructor !== Array){ + tag = [tag]; + } + + let pairs = []; + + for(let i = 0, field; i < tag.length; i++){ + field = tag[i]; + if(DEBUG && is_string(field)){ + throw new Error("A tag option can't be a string, instead it needs a { field: tag } format."); + } + // default array notation + if(field.field && field.tag){ + const value = field.tag; + if(value.constructor === Array){ + for(let k = 0; k < value.length; k++){ + pairs.push(field.field, value[k]); + } + } + else{ + pairs.push(field.field, value); + } + } + // shorter object notation + else{ + const keys = Object.keys(field); + for(let j = 0, key, value; j < keys.length; j++){ + key = keys[j]; + value = field[key]; + if(value.constructor === Array){ + for(let k = 0; k < value.length; k++){ + pairs.push(key, value[k]); + } + } + else{ + pairs.push(key, value); + } + } + } + } + + if(DEBUG && !pairs.length){ + throw new Error("Your tag definition within the search options is probably wrong. No valid tags found."); + } + + // tag used as pairs from this point + tag = pairs; + + // when tags is used and no query was set, + // then just return the tag indexes + if(!query){ + + let promises = []; + if(pairs.length) for(let j = 0; j < pairs.length; j+=2){ + let ids; + if(SUPPORT_PERSISTENT && this.db){ + const index = this.index.get(pairs[j]); + if(!index){ + if(DEBUG){ + console.warn("Tag '" + pairs[j] + ":" + pairs[j + 1] + "' will be skipped because there is no field '" + pairs[j] + "'."); + } + continue; + } + debug && console.log("checkoint:search:tag:get", pairs[j + 1]); + promises.push(ids = index.db.tag(pairs[j + 1], limit, offset, enrich)); + } + else{ + debug && console.log("checkoint:search:tag:get", pairs[j + 1]); + ids = get_tag.call(this, pairs[j], pairs[j + 1], limit, offset, enrich); + } + result.push({ + "field": pairs[j], + "tag": pairs[j + 1], + "result": ids + }); + } + + if(promises.length){ + return Promise.all(promises).then(function(promises){ + for(let j = 0; j < promises.length; j++){ + result[j].result = promises[j]; + } + return result; + }); + } + + return result; + } + } + + // extend to multi field search by default + if(is_string(field)){ + field = [field]; + } + } + } + + field || (field = this.field); + let promises = !_resolve && (this.worker || this.async) && []; + let db_tag_search; + + // multi field search + // field could be a custom set of selected fields by this query + // db tag indexes are also included in this field list + for(let i = 0, res, key, len; i < field.length; i++){ + + key = field[i]; + + if(SUPPORT_PERSISTENT && SUPPORT_TAGS && this.db && this.tag){ + // tree is missing when it is a tag-only index (db) + if(!this.tree[i]){ + continue; + } + } + + let field_options; + + if(!is_string(key)){ + field_options = key; + key = field_options.field; + query = field_options.query || query; + limit = field_options.limit || limit; + //offset = field_options.offset || offset; + suggest = SUPPORT_SUGGESTION && (field_options.suggest || suggest); + //enrich = SUPPORT_STORE && this.store && (field_options.enrich || enrich); + } + + if(_resolve){ + res = _resolve[i]; + } + else{ + debug && console.log("checkoint:search:get", key); + let opt = field_options || options; + let index = this.index.get(key); + + if(tag){ + if(SUPPORT_PERSISTENT && this.db){ + opt.tag = tag; + db_tag_search = index.db.support_tag_search; + opt.field = field; + } + if(!db_tag_search){ + opt.enrich = false; + } + } + if(promises){ + promises[i] = index.searchAsync(query, limit, opt); + // restore enrich state + opt && enrich && (opt.enrich = enrich); + // just collect and continue + continue; + } + else{ + res = index.search(query, limit, opt); + // restore enrich state + opt && enrich && (opt.enrich = enrich); + } + } + + len = res && res.length; + + // todo when no term was matched but tag was retrieved extend suggestion to tags + // every field has to intersect against all selected tag fields + if(tag && len){ + + const arr = []; + let count = 0; + + // tags are only applied in resolve phase when it's a db + if(SUPPORT_PERSISTENT && this.db && _resolve){ + if(!db_tag_search){ + + // retrieve tag results assigned to it's field + for(let y = field.length; y < _resolve.length; y++){ + + let ids = _resolve[y]; + let len = ids && ids.length; + + if(len){ + count++; + arr.push(ids); + } + else if(!suggest){ + // no tags found + return result; + } + } + } + } + else{ + + // tag[] are pairs at this line + for(let y = 0, ids, len; y < tag.length; y+=2){ + + debug && console.log("checkoint:search:tag:get", tag[y + 1]); + ids = this.tag.get(tag[y]); + + if(!ids){ + if(DEBUG){ + console.warn("Tag '" + tag[y] + ":" + tag[y + 1] + "' will be skipped because there is no field '" + tag[y] + "'."); + } + if(suggest){ + continue; + } + else{ + return result; + } + } + + ids = ids && ids.get(tag[y + 1]); + len = ids && ids.length; + + if(len){ + count++; + arr.push(ids); + } + else if(!suggest){ + // no tags found + return result; + } + } + } + + if(count){ + debug && console.log("checkoint:search:tag:intersect"); + res = intersect_union(res, arr); // intersect(arr, limit, offset) + len = res.length; + if(!len && !suggest){ + // nothing matched + return result; + } + // move counter back by 1 + count--; + } + } + + if(len){ + result_field[count] = key; + result.push(res); + count++; + } + else if(field.length === 1){ + // fast path: nothing matched + return result; + } + } + + if(promises){ + if(SUPPORT_PERSISTENT && SUPPORT_TAGS && this.db){ + // todo when a tag index is never a search index this could be extracted + // push tag promises to the end + if(tag && tag.length && !db_tag_search){ + for(let y = 0; y < tag.length; y += 2){ + // it needs to retrieve data from tag pairs + const index = this.index.get(tag[y]); + if(!index){ + if(DEBUG){ + console.warn("Tag '" + tag[y] + ":" + tag[y + 1] + "' was not found because there is no field '" + tag[y] + "'."); + } + if(suggest){ + continue; + } + else{ + return result; + } + } + debug && console.log("checkoint:search:tag:get", tag[y + 1]); + promises.push(index.db.tag(tag[y + 1], limit, offset, /* enrich */ false)); + } + } + } + + const self = this; + + // TODO unroll this recursion + return Promise.all(promises).then(function(result){ + return result.length + ? self.search(query, limit, options, /* resolve: */ result) + : result; + }); + } + + if(!count){ + return result; + } + if(pluck && (!enrich || !this.store)){ + return result[0]; + } + + promises = []; + + for(let i = 0, res; i < result_field.length; i++){ + + res = result[i]; + + if(enrich && res.length && !res[0].doc){ + if(!SUPPORT_PERSISTENT || !this.db){ + if(res.length){ + res = apply_enrich.call(this, res); + } + } + else{ + debug && console.log("checkoint:search:doc:get"); + promises.push(res = this.index.get(this.field[0]).db.enrich(res)); + } + } + + if(pluck){ + return res; + } + + result[i] = { + "field": result_field[i], + "result": res + }; + } + + if(enrich && SUPPORT_PERSISTENT && this.db && promises.length){ + return Promise.all(promises).then(function(promises){ + for(let j = 0; j < promises.length; j++){ + result[j].result = promises[j]; + } + return merge + ? merge_fields(result, limit, offset) + : result; + }); + } + + return merge + ? merge_fields(result, limit, offset) + : result; +}; + +// todo support Resolver +// todo when searching through multiple fields each term should +// be found at least by one field to get a valid match without +// using suggestion explicitly + +function merge_fields(fields, limit, offset){ + const final = []; + const set = create_object(); + for(let i = 0, field, res; i < fields.length; i++){ + field = fields[i]; + res = field.result; + for(let j = 0, id, entry, tmp; j < res.length; j++){ + entry = res[j]; + id = entry.id; + tmp = set[id]; + if(!tmp){ + // offset was already applied on field indexes + // if(offset){ + // offset--; + // continue; + // } + // apply limit from last round, because just fields could + // be pushed without adding new results + if(final.length === limit){ + return final; + } + entry.field = set[id] = [field.field]; + final.push(entry); + } + else{ + tmp.push(field.field); + } + } + } + return final; +} + +/** + * @this Document + */ + +function get_tag(tag, key, limit, offset, enrich){ + + debug && console.log("checkoint:search:tag:get", key); + let res = this.tag.get(tag); + if(!res){ + DEBUG && console.warn("Tag '" + tag + "' was not found"); + return []; + } + res = res && res.get(key); + let len = res && (res.length - offset); + + if(len && (len > 0)){ + if((len > limit) || offset){ + res = res.slice(offset, offset + limit); + } + if(enrich){ + res = apply_enrich.call(this, res); + } + return res; + } +} + +/** + * @this Document + */ + +function apply_enrich(res){ + + const arr = new Array(res.length); + + for(let x = 0, id; x < res.length; x++){ + id = res[x]; + arr[x] = { + "id": id, + "doc": this.store.get(id) + }; + } + + return arr; +} diff --git a/src/encoder.js b/src/encoder.js new file mode 100644 index 0000000..02f463e --- /dev/null +++ b/src/encoder.js @@ -0,0 +1,802 @@ +// COMPILER BLOCK --> +import { + SUPPORT_CACHE, + SUPPORT_COMPRESSION, + SUPPORT_ENCODER +} from "./config.js"; +// <-- COMPILER BLOCK +import { parse_option } from "./common.js"; + +/* + +Custom Encoder +---------------- + +// Split a passed string into an Array of words: +function englishEncoder(string){ + return string.toLowerCase().split(/[^a-z]+/) +} + +// For CJK split a passed string into an Array of chars: +function chineseEncoder(string){ + return string.replace(/\s+/, "").split("") +} + +// Alternatively do not split the input: +function fixedEncoder(string){ + return [string] +} + +Built-in Encoder (Workflow) +---------------------------- +Pipeline: + 1. apply this.normalize: charset normalization: + applied on the whole input string e.g. lowercase, + will also apply on: filter, matcher, stemmer, mapper + 2. apply this.split: split input into terms (includes/excludes) + 3. apply this.filter (pre-filter) + 4. apply this.matcher (replace terms) + 5. apply this.stemmer (replace term endings) + 6. apply this.filter (post-filter) + 7. apply this.mapper (replace chars) + 8. apply this.replacer (custom regex) + 9. apply this.dedupe (letter deduplication) + 10. apply this.finalize +*/ + +const whitespace = /[^\p{L}\p{N}]+/u; // /[\p{Z}\p{S}\p{P}\p{C}]+/u; +//const numeric_split = /(\d{3})/g; +const numeric_split_length = /(\d{3})/g; +const numeric_split_prev_char = /(\D)(\d{3})/g; +const numeric_split_next_char = /(\d{3})(\D)/g; +//.replace(/(\d{3})/g, "$1 ") +//.replace(/([^\d])([\d])/g, "$1 $2") +//.replace(/([\d])([^\d])/g, "$1 $2") +const normalize = "".normalize && /[\u0300-\u036f]/g; // '´`’ʼ., +const normalize_mapper = SUPPORT_ENCODER && !normalize && new Map([ + + // Charset Normalization + + ["ª","a"], + ["²","2"], + ["³","3"], + ["¹","1"], + ["º","o"], + ["¼","1⁄4"], + ["½","1⁄2"], + ["¾","3⁄4"], + ["à","a"], + ["á","a"], + ["â","a"], + ["ã","a"], + ["ä","a"], + ["å","a"], + ["ç","c"], + ["è","e"], + ["é","e"], + ["ê","e"], + ["ë","e"], + ["ì","i"], + ["í","i"], + ["î","i"], + ["ï","i"], + ["ñ","n"], + ["ò","o"], + ["ó","o"], + ["ô","o"], + ["õ","o"], + ["ö","o"], + ["ù","u"], + ["ú","u"], + ["û","u"], + ["ü","u"], + ["ý","y"], + ["ÿ","y"], + ["ā","a"], + ["ă","a"], + ["ą","a"], + ["ć","c"], + ["ĉ","c"], + ["ċ","c"], + ["č","c"], + ["ď","d"], + ["ē","e"], + ["ĕ","e"], + ["ė","e"], + ["ę","e"], + ["ě","e"], + ["ĝ","g"], + ["ğ","g"], + ["ġ","g"], + ["ģ","g"], + ["ĥ","h"], + ["ĩ","i"], + ["ī","i"], + ["ĭ","i"], + ["į","i"], + ["ij","ij"], + ["ĵ","j"], + ["ķ","k"], + ["ĺ","l"], + ["ļ","l"], + ["ľ","l"], + ["ŀ","l"], + ["ń","n"], + ["ņ","n"], + ["ň","n"], + ["ʼn","n"], + ["ō","o"], + ["ŏ","o"], + ["ő","o"], + ["ŕ","r"], + ["ŗ","r"], + ["ř","r"], + ["ś","s"], + ["ŝ","s"], + ["ş","s"], + ["š","s"], + ["ţ","t"], + ["ť","t"], + ["ũ","u"], + ["ū","u"], + ["ŭ","u"], + ["ů","u"], + ["ű","u"], + ["ų","u"], + ["ŵ","w"], + ["ŷ","y"], + ["ź","z"], + ["ż","z"], + ["ž","z"], + ["ſ","s"], + ["ơ","o"], + ["ư","u"], + ["dž","dz"], + ["lj","lj"], + ["nj","nj"], + ["ǎ","a"], + ["ǐ","i"], + ["ǒ","o"], + ["ǔ","u"], + ["ǖ","u"], + ["ǘ","u"], + ["ǚ","u"], + ["ǜ","u"], + ["ǟ","a"], + ["ǡ","a"], + ["ǣ","ae"], + ["æ","ae"], + ["ǽ","ae"], + ["ǧ","g"], + ["ǩ","k"], + ["ǫ","o"], + ["ǭ","o"], + ["ǯ","ʒ"], + ["ǰ","j"], + ["dz","dz"], + ["ǵ","g"], + ["ǹ","n"], + ["ǻ","a"], + ["ǿ","ø"], + ["ȁ","a"], + ["ȃ","a"], + ["ȅ","e"], + ["ȇ","e"], + ["ȉ","i"], + ["ȋ","i"], + ["ȍ","o"], + ["ȏ","o"], + ["ȑ","r"], + ["ȓ","r"], + ["ȕ","u"], + ["ȗ","u"], + ["ș","s"], + ["ț","t"], + ["ȟ","h"], + ["ȧ","a"], + ["ȩ","e"], + ["ȫ","o"], + ["ȭ","o"], + ["ȯ","o"], + ["ȱ","o"], + ["ȳ","y"], + ["ʰ","h"], + ["ʱ","h"], + ["ɦ","h"], + ["ʲ","j"], + ["ʳ","r"], + ["ʴ","ɹ"], + ["ʵ","ɻ"], + ["ʶ","ʁ"], + ["ʷ","w"], + ["ʸ","y"], + ["ˠ","ɣ"], + ["ˡ","l"], + ["ˢ","s"], + ["ˣ","x"], + ["ˤ","ʕ"], + ["ΐ","ι"], + ["ά","α"], + ["έ","ε"], + ["ή","η"], + ["ί","ι"], + ["ΰ","υ"], + ["ϊ","ι"], + ["ϋ","υ"], + ["ό","ο"], + ["ύ","υ"], + ["ώ","ω"], + ["ϐ","β"], + ["ϑ","θ"], + ["ϒ","Υ"], + ["ϓ","Υ"], + ["ϔ","Υ"], + ["ϕ","φ"], + ["ϖ","π"], + ["ϰ","κ"], + ["ϱ","ρ"], + ["ϲ","ς"], + ["ϵ","ε"], + ["й","и"], + ["ѐ","е"], + ["ё","е"], + ["ѓ","г"], + ["ї","і"], + ["ќ","к"], + ["ѝ","и"], + ["ў","у"], + ["ѷ","ѵ"], + ["ӂ","ж"], + ["ӑ","а"], + ["ӓ","а"], + ["ӗ","е"], + ["ӛ","ә"], + ["ӝ","ж"], + ["ӟ","з"], + ["ӣ","и"], + ["ӥ","и"], + ["ӧ","о"], + ["ӫ","ө"], + ["ӭ","э"], + ["ӯ","у"], + ["ӱ","у"], + ["ӳ","у"], + ["ӵ","ч"] + + // Term Separators + + // ["'", ""], // it's -> its + // ["´", ""], + // ["`", ""], + // ["’", ""], + // ["ʼ", ""], + + // Numeric-Separators Chars Removal + + // [",", ""], + // [".", ""] + + // Non-Whitespace Separators + + // already was split by default via p{P} + // ["-", " "], + // [":", " "], + // ["_", " "], + // ["|", " "], + // ["/", " "], + // ["\\", " "] +]); + +/** + * @param options + * @constructor + */ + +export default function Encoder(options = {}){ + + if(!(this instanceof Encoder)){ + return new Encoder(...arguments); + } + + for(let i = 0; i < arguments.length; i++){ + this.assign(arguments[i]); + } +}; + +Encoder.prototype.assign = function(options){ + + // if(options.assign){ + // //options = Object.assign({}, options.assign, options); + // this.assign(options.assign); + // } + + // let tmp = options["normalize"]; + // if(typeof tmp === "function"){ + // const old = this.normalize; + // if(typeof old === "function"){ + // const current = tmp; + // tmp = function(){ + // old(); + // current(); + // } + // } + // } + + /** + * pre-processing string input + * @type {Function|boolean} + */ + this.normalize = /** @type {Function|boolean} */ ( + parse_option(options.normalize, true, this.normalize) + ); + + // { + // letter: true, + // number: true, + // whitespace: true, + // symbol: true, + // punctuation: true, + // control: true, + // char: "" + // } + + let include = options.include; + let tmp = include || options.exclude || options.split; + + if(typeof tmp === "object"){ + let numeric = !include; + let regex = ""; + // split on whitespace by default + options.include || ( + regex += "\\p{Z}" + ); + if(tmp.letter){ + regex += "\\p{L}"; + } + if(tmp.number){ + regex += "\\p{N}"; + numeric = !!include; + } + if(tmp.symbol){ + regex += "\\p{S}"; + } + if(tmp.punctuation){ + regex += "\\p{P}"; + } + if(tmp.control){ + regex += "\\p{C}"; + } + if((tmp = tmp.char)){ + regex += typeof tmp === "object" ? tmp.join("") : tmp; + } + + this.split = new RegExp("[" + (include ? "^" : "") + regex + "]+", "u"); + this.numeric = numeric; + } + else{ + + /** + * split string input into terms + * @type {string|RegExp|boolean|null} + */ + this.split = /** @type {string|RegExp|boolean} */ (parse_option(tmp, whitespace, this.split)); + this.numeric = parse_option(this.numeric, true); + } + + /** + * post-processing terms + * @type {Function|null} + */ + this.prepare = /** @type {Function|null} */ ( + parse_option(options.prepare, null, this.prepare) + ); + /** + * final processing + * @type {Function|null} + */ + this.finalize = /** @type {Function|null} */ ( + parse_option(options.finalize, null, this.finalize) + ); + + // options + + this.rtl = options.rtl || false; + this.dedupe = parse_option(options.dedupe, true, this.dedupe); + this.filter = parse_option((tmp = options.filter) && new Set(tmp), null, this.filter); + this.matcher = parse_option((tmp = options.matcher) && new Map(tmp), null, this.matcher); + this.mapper = parse_option((tmp = options.mapper) && new Map(tmp), null, this.mapper); + this.stemmer = parse_option((tmp = options.stemmer) && new Map(tmp), null, this.stemmer); + this.replacer = parse_option(options.replacer, null, this.replacer); + this.minlength = parse_option(options.minlength, 1, this.minlength); + this.maxlength = parse_option(options.maxlength, 0, this.maxlength); + + // minimum required tokenizer by this encoder + //this.tokenize = options["tokenize"] || ""; + + // auto-balanced cache + if(SUPPORT_CACHE){ + this.cache = tmp = parse_option(options.cache, true, this.cache); + if(tmp){ + this.timer = null; + this.cache_size = typeof tmp === "number" ? tmp : 2e5; + this.cache_enc = new Map(); + this.cache_prt = new Map(); + this.cache_enc_length = 128; + this.cache_prt_length = 128; + } + } + + // regex temporary state + this.matcher_str = ""; + this.matcher_test = null; + this.stemmer_str = ""; + this.stemmer_test = null; + + // prebuilt + // if(this.filter && this.split){ + // for(const key of this.filter){ + // const tmp = key.replace(this.split, ""); + // if(key !== tmp){ + // this.filter.delete(key); + // this.filter.add(tmp); + // } + // } + // } + if(this.matcher){ + for(const key of this.matcher.keys()){ + this.matcher_str += (this.matcher_str ? "|" : "") + key; + } + } + if(this.stemmer){ + for(const key of this.stemmer.keys()){ + this.stemmer_str += (this.stemmer_str ? "|" : "") + key; + } + } + + // if(SUPPORT_COMPRESSION){ + // this.compression = parse_option(options.compress || options.compression, 0, this.compression); + // if(this.compression && !table){ + // table = new Array(radix); + // for(let i = 0; i < radix; i++) table[i] = i + 33; + // table = String.fromCharCode.apply(null, table); + // } + // } + + return this; +}; + +Encoder.prototype.addMatcher = function(match, replace){ + // regex: + if(typeof match === "object"){ + return this.addReplacer(match, replace); + } + // a single char: + if(match.length < 2){ + return this.addMapper(match, replace); + } + this.matcher || (this.matcher = new Map()); + this.matcher.set(match , replace); + this.matcher_str += (this.matcher_str ? "|" : "") + match; + this.matcher_test = null; //new RegExp("(" + this.matcher_str + ")"); + SUPPORT_CACHE && this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addStemmer = function(match, replace){ + this.stemmer || (this.stemmer = new Map()); + this.stemmer.set(match, replace); + this.stemmer_str += (this.stemmer_str ? "|" : "") + match; + this.stemmer_test = null; //new RegExp("(" + this.stemmer_str + ")"); + SUPPORT_CACHE && this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addFilter = function(str){ + this.filter || (this.filter = new Set()); + this.filter.add(str); + SUPPORT_CACHE && this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addMapper = function(char_match, char_replace){ + // regex: + if(typeof char_match === "object"){ + return this.addReplacer(char_match, char_replace); + } + // not a char: + if(char_match.length > 1){ + return this.addMatcher(char_match, char_replace); + } + this.mapper || (this.mapper = new Map()); + this.mapper.set(char_match, char_replace); + SUPPORT_CACHE && this.cache && this.invalidate(); + return this; +}; + +Encoder.prototype.addReplacer = function(match, replace){ + if(typeof match === "string") match = new RegExp(match, "g"); + this.replacer || (this.replacer = []); + this.replacer.push(match, replace || ""); + SUPPORT_CACHE && this.cache && this.invalidate(); + return this; +}; + +if(SUPPORT_CACHE){ + Encoder.prototype.invalidate = function(){ + this.cache_enc.clear(); + this.cache_prt.clear(); + }; +} + +Encoder.prototype.encode = function(str){ + + //if(!str) return str; + // todo remove dupe terms + + if(SUPPORT_CACHE && this.cache && str.length <= this.cache_enc_length){ + if(this.timer){ + if(this.cache_enc.has(str)){ + return this.cache_enc.get(str); + } + } + else{ + this.timer = setTimeout(clear, 0, this); + } + } + + // 1. apply charset normalization + if(this.normalize){ + if(typeof this.normalize === "function"){ + str = this.normalize(str); + } + else if(normalize){ + str = str.normalize("NFKD").replace(normalize, "").toLowerCase(); + } + else{ + str = str.toLowerCase(); + if(SUPPORT_ENCODER){ + this.mapper = this.mapper + // todo replace spread + ? new Map([.../** @type {!Iterable} */(normalize_mapper), ...this.mapper]) + : new Map(/** @type {Map} */ (normalize_mapper)); + } + } + //if(!str) return str; + } + + // 2. apply custom encoder (can replace split) + if(this.prepare){ + str = this.prepare(str); + } + + // 3. split numbers into triplets + if(this.numeric && str.length > 3){ + str = str.replace(numeric_split_prev_char, "$1 $2") + .replace(numeric_split_next_char, "$1 $2") + .replace(numeric_split_length, "$1 "); + } + + // if(this.matcher && (str.length > 1)){ + // this.matcher_test || ( + // this.matcher_test = new RegExp("(" + this.matcher_str + ")", "g") + // ); + // str = str.replace(this.matcher_test, match => this.matcher.get(match)); + // } + // if(this.stemmer){ + // this.stemmer_test || ( + // this.stemmer_test = new RegExp("(?!\\b)(" + this.stemmer_str + ")(\\b|_)", "g") + // ); + // str = str.replace(this.stemmer_test, match => this.stemmer.get(match)); + // } + + const skip = !(this.dedupe || this.mapper || this.filter || this.matcher || this.stemmer || this.replacer); + let final = []; + let words = this.split || this.split === "" + ? str.split(/** @type {string|RegExp} */ (this.split)) + : str; //[str]; + + for(let i = 0, word, base; i < words.length; i++){ + // filter empty entries + if(!(word = base = words[i])){ + continue; + } + if(word.length < this.minlength){ + continue; + } + if(skip) { + final.push(word); + continue; + } + + // 1. pre-filter before cache + if(this.filter && this.filter.has(word)){ + continue; + } + + if(SUPPORT_CACHE && this.cache && word.length <= this.cache_prt_length){ + if(this.timer){ + const tmp = this.cache_prt.get(word); + //if(this.cache_prt.has(word)){ + if(tmp || tmp === ""){ + //word = this.cache_prt.get(word); + tmp && final.push(tmp); + //word ? words[i] = word : words.splice(i--, 1); + continue; + } + } + else{ + this.timer = setTimeout(clear, 0, this); + } + } + + let postfilter; + + // if(this.normalize === true && normalize){ + // word = word.normalize("NFKD").replace(normalize, ""); + // postfilter = 1; + // } + + // if(this.normalize){ + // if(typeof this.normalize === "function"){ + // word = this.normalize(word); + // } + // else if(normalize){ + // word = word.normalize("NFKD").replace(normalize, "").toLowerCase(); + // } + // else{ + // word = word.toLowerCase(); + // this.mapper = this.mapper + // ? new Map([...normalize_mapper, ...this.mapper]) + // : new Map(/** @type {Map} */ normalize_mapper); + // } + // postfilter = 1; + // //if(!str) return str; + // } + + // 2. apply stemmer after matcher + if(this.stemmer && (word.length > 2)){ + // for(const item of this.stemmer){ + // const key = item[0]; + // const value = item[1]; + // + // if(word.length > key.length && word.endsWith(key)){ + // word = word.substring(0, word.length - key.length) + value; + // break; + // } + // + // // const position = word.length - key.length; + // // if(position > 0 && word.substring(position) === key){ + // // word = word.substring(0, position) + value; + // // break; + // // } + // } + this.stemmer_test || ( + this.stemmer_test = new RegExp("(?!^)(" + this.stemmer_str + ")$") + ); + word = word.replace(this.stemmer_test, match => this.stemmer.get(match)); + postfilter = 1; + } + + // 3. apply matcher + if(this.matcher && (word.length > 1)){ + this.matcher_test || ( + this.matcher_test = new RegExp("(" + this.matcher_str + ")", "g") + ); + word = word.replace(this.matcher_test, match => this.matcher.get(match)); + postfilter = 1; + } + + // 4. post-filter after matcher and stemmer was applied + if(word && postfilter && (word.length < this.minlength || (this.filter && this.filter.has(word)))){ + word = ""; + } + + // 5. apply mapper and collapsing + if(word && (this.mapper || (this.dedupe && word.length > 1))){ + //word = this.replace_dedupe(word); + //word = replace_deduped(word, this.mapper, true); + let final = ""; + for(let i = 0, prev = "", char, tmp; i < word.length; i++){ + char = word.charAt(i); + if(char !== prev || !this.dedupe){ + tmp = this.mapper && this.mapper.get(char); + if(!tmp && tmp !== "") + final += (prev = char); + else if((tmp !== prev || !this.dedupe) && (prev = tmp)) + final += tmp; + } + } + word = final; + } + + // apply custom regex + if(word && this.replacer){ + for(let i = 0; word && (i < this.replacer.length); i+=2){ + word = word.replace(this.replacer[i], this.replacer[i+1]); + } + } + + // slower variants for removing same chars in a row: + //word = word.replace(/([^0-9])\1+/g, "$1"); + //word = word.replace(/(.)\1+/g, "$1"); + //word = word.replace(/(?<=(.))\1+/g, ""); + + // if(word){ + // words[i] = word; + // } + + if(SUPPORT_CACHE && this.cache && base.length <= this.cache_prt_length){ + this.cache_prt.set(base, word); + if(this.cache_prt.size > this.cache_size){ + this.cache_prt.clear(); + this.cache_prt_length = this.cache_prt_length / 1.1 | 0; + } + } + + //word || words.splice(i--, 1); + word && final.push(word); + } + + //words = final; + // else if(this.filter){ + // for(let i = 0, word; i < words.length; i++){ + // if((word = words[i]) && !this.filter.has(word)){ + // //filtered.push(word); + // words.splice(i--, 1); + // } + // } + // } + + if(this.finalize){ + final = this.finalize(final) || final; + } + + if(SUPPORT_CACHE && this.cache && str.length <= this.cache_enc_length){ + this.cache_enc.set(str, final); + if(this.cache_enc.size > this.cache_size){ + this.cache_enc.clear(); + this.cache_enc_length = this.cache_enc_length / 1.1 | 0; + } + } + + return final; +}; + +// Encoder.prototype.compress = function(str) { +// +// //return str; +// //if(!str) return str; +// +// if(SUPPORT_CACHE && this.cache && str.length <= this.cache_prt_length){ +// if(this.timer){ +// if(this.cache_cmp.has(str)){ +// return this.cache_cmp.get(str); +// } +// } +// else{ +// this.timer = setTimeout(clear, 0, this); +// } +// } +// +// const result = typeof this.compression === "function" +// ? this.compression(str) +// : hash(str); //window.hash(str); +// +// if(SUPPORT_CACHE && this.cache && str.length <= this.cache_prt_length){ +// this.cache_cmp.set(str, result); +// this.cache_cmp.size > this.cache_size && +// this.cache_cmp.clear(); +// } +// +// return result; +// }; + +// function hash(str){ +// return str; +// } + +function clear(self){ + self.timer = null; + self.cache_enc.clear(); + self.cache_prt.clear(); +} diff --git a/src/engine.js b/src/engine.js deleted file mode 100644 index 0e3b0a8..0000000 --- a/src/engine.js +++ /dev/null @@ -1,49 +0,0 @@ -// COMPILER BLOCK --> -import { DEBUG, SUPPORT_ASYNC, SUPPORT_CACHE } from "./config"; -// <-- COMPILER BLOCK -import { searchCache } from "./cache"; - -/** - * @constructor - * @abstract - */ - -function Engine(index){ - - if(DEBUG){ - - //if(this.constructor === Engine){ - if(this instanceof Engine){ - - throw new Error("Can't instantiate abstract class!"); - } - } - - if(SUPPORT_CACHE){ - - index.prototype.searchCache = searchCache; - } - - if(SUPPORT_ASYNC){ - - index.prototype.addAsync = addAsync; - index.prototype.appendAsync = appendAsync; - index.prototype.searchAsync = searchAsync; - index.prototype.updateAsync = updateAsync; - index.prototype.removeAsync = removeAsync; - } -} - -if(SUPPORT_CACHE){ - - Engine.prototype.searchCache = searchCache; -} - -if(SUPPORT_ASYNC){ - - Engine.prototype.addAsync = addAsync; - Engine.prototype.appendAsync = appendAsync; - Engine.prototype.searchAsync = searchAsync; - Engine.prototype.updateAsync = updateAsync; - Engine.prototype.removeAsync = removeAsync; -} \ No newline at end of file diff --git a/src/global.js b/src/global.js index 9635704..50d6f12 100644 --- a/src/global.js +++ b/src/global.js @@ -1,5 +1,7 @@ -export const global_lang = {}; -export const global_charset = {}; +import { create_object } from "./common.js"; + +export const global_lang = create_object(); +export const global_charset = create_object(); /** * @param {!string} name @@ -7,7 +9,6 @@ export const global_charset = {}; */ export function registerCharset(name, charset){ - global_charset[name] = charset; } @@ -17,6 +18,5 @@ export function registerCharset(name, charset){ */ export function registerLanguage(name, lang){ - global_lang[name] = lang; } diff --git a/src/index.js b/src/index.js index 49fcdad..36c86d5 100644 --- a/src/index.js +++ b/src/index.js @@ -9,103 +9,187 @@ // COMPILER BLOCK --> import { + DEBUG, + PROFILER, SUPPORT_ENCODER, SUPPORT_CACHE, SUPPORT_ASYNC, - SUPPORT_SUGGESTION, - SUPPORT_SERIALIZE + SUPPORT_SERIALIZE, + SUPPORT_PERSISTENT, + SUPPORT_COMPRESSION, + SUPPORT_KEYSTORE, + SUPPORT_RESOLVER } from "./config.js"; // <-- COMPILER BLOCK -import { IndexInterface } from "./type.js"; -import { encode as default_encoder } from "./lang/latin/default.js"; -import { create_object, create_object_array, concat, sort_by_length_down, is_array, is_string, is_object, parse_option } from "./common.js"; -import { pipeline, init_stemmer_or_matcher, init_filter } from "./lang.js"; -import { global_lang, global_charset } from "./global.js"; -import apply_async from "./async.js"; -import { intersect } from "./intersect.js"; +import { IndexOptions } from "./type.js"; +import Encoder from "./encoder.js"; import Cache, { searchCache } from "./cache.js"; -import apply_preset from "./preset.js"; +import { KeystoreMap, KeystoreSet } from "./keystore.js"; +import { is_array, is_string } from "./common.js"; import { exportIndex, importIndex } from "./serialize.js"; +import { global_lang, global_charset } from "./global.js"; +import default_encoder from "./lang/latin/default.js"; +import apply_preset from "./preset.js"; +import apply_async from "./async.js"; +import tick from "./profiler.js"; +import "./index/add.js"; +import "./index/search.js"; +import "./index/remove.js"; /** * @constructor - * @implements IndexInterface - * @param {Object=} options - * @param {Object=} _register - * @return {Index} + * @param {IndexOptions|string=} options + * @param {Map|Set|KeystoreSet|KeystoreMap=} _register */ -function Index(options, _register){ - - if(!(this instanceof Index)) { +export default function Index(options, _register){ + if(!(this instanceof Index)){ return new Index(options); } - let charset, lang, tmp; + PROFILER && tick("Index.create"); if(options){ - if(SUPPORT_ENCODER){ + options = apply_preset(options); + // charset = options.charset; + // // lang = options.lang; + // + // if(is_string(charset)){ + // + // if(!charset.includes(":")){ + // charset += ":default"; + // } + // + // charset = global_charset[charset]; + // } - options = apply_preset(options); - } - - charset = options["charset"]; - lang = options["lang"]; - - if(is_string(charset)){ - - if(charset.indexOf(":") === -1){ - - charset += ":default"; - } - - charset = global_charset[charset]; - } - - if(is_string(lang)){ - - lang = global_lang[lang]; - } + // if(is_string(lang)){ + // + // lang = global_lang[lang]; + // } } else{ - options = {}; } - let resolution, optimize, context = options["context"] || {}; + // let charset, lang, tmp; - this.encode = options["encode"] || (charset && charset.encode) || default_encoder; - this.register = _register || create_object(); - this.resolution = resolution = options["resolution"] || 9; - this.tokenize = tmp = (charset && charset.tokenize) || options["tokenize"] || "strict"; - this.depth = (tmp === "strict") && context["depth"]; - this.bidirectional = parse_option(context["bidirectional"], true); - this.optimize = optimize = parse_option(options["optimize"], true); - this.fastupdate = parse_option(options["fastupdate"], true); - this.minlength = options["minlength"] || 1; - this.boost = options["boost"]; + const context = options.context || {}; + const encoder = options.encode || options.encoder || default_encoder; + this.encoder = encoder.encode + ? encoder + : typeof encoder === "object" + ? new Encoder(encoder) + : { encode: encoder }; - // when not using the memory strategy the score array should not pre-allocated to its full length + if(SUPPORT_COMPRESSION){ + this.compress = options.compress || options.compression || false; + } - this.map = optimize ? create_object_array(resolution) : create_object(); - this.resolution_ctx = resolution = context["resolution"] || 1; - this.ctx = optimize ? create_object_array(resolution) : create_object(); - this.rtl = (charset && charset.rtl) || options["rtl"]; - this.matcher = (tmp = options["matcher"] || (lang && lang.matcher)) && init_stemmer_or_matcher(tmp, false); - this.stemmer = (tmp = options["stemmer"] || (lang && lang.stemmer)) && init_stemmer_or_matcher(tmp, true); - this.filter = (tmp = options["filter"] || (lang && lang.filter)) && init_filter(tmp); + let tmp; + this.resolution = options.resolution || 9; + this.tokenize = tmp = options.tokenize || "strict"; + this.depth = (tmp === "strict" && context.depth) || 0; + this.bidirectional = context.bidirectional !== false; + this.fastupdate = !!options.fastupdate; + this.score = options.score || null; + + tmp = SUPPORT_KEYSTORE && (options.keystore || 0); + tmp && (this.keystore = tmp); + + this.map = tmp ? new KeystoreMap(tmp) : new Map(); + this.ctx = tmp ? new KeystoreMap(tmp) : new Map(); + this.reg = _register || ( + this.fastupdate + ? (tmp ? new KeystoreMap(tmp) : new Map()) + : (tmp ? new KeystoreSet(tmp) : new Set()) + ); + this.resolution_ctx = context.resolution || 1; + this.rtl = (encoder.rtl) || options.rtl || false; if(SUPPORT_CACHE){ + this.cache = (tmp = options.cache || null) && new Cache(tmp); + } - this.cache = (tmp = options["cache"]) && new Cache(tmp); + if(SUPPORT_RESOLVER){ + this.resolve = options.resolve !== false; + } + + if(SUPPORT_PERSISTENT){ + if((tmp = options.db)){ + this.db = tmp.mount(this); + } + this.commit_auto = options.commit !== false; + this.commit_task = []; + this.commit_timer = null; } } -export default Index; +if(SUPPORT_PERSISTENT){ + Index.prototype.mount = function(db){ + if(this.commit_timer){ + clearTimeout(this.commit_timer); + this.commit_timer = null; + } + return db.mount(this); + }; + Index.prototype.commit = function(replace, append){ + if(this.commit_timer){ + clearTimeout(this.commit_timer); + this.commit_timer = null; + } + return this.db.commit(this, replace, append); + }; +} + +// if(SUPPORT_RESOLVER){ +// Index.prototype.resolve = function(params){ +// return new Resolver(params); +// }; +// } + +/** + * @param {!Index} self + * @param {boolean=} replace + * @param {boolean=} append + */ + +export function autoCommit(self, replace, append){ + if(!self.commit_timer){ + self.commit_timer = setTimeout(function(){ + self.commit_timer = null; + self.db.commit(self, replace, append); + }, 0); + } +} + +Index.prototype.clear = function(){ + + //this.map = new Map(); + //this.ctx = new Map(); + //this.reg = this.fastupdate ? new Map() : new Set(); + this.map.clear(); + this.ctx.clear(); + this.reg.clear(); + + if(SUPPORT_CACHE){ + this.cache && + this.cache.clear(); + } + + if(SUPPORT_PERSISTENT && this.db){ + this.commit_timer && clearTimeout(this.commit_timer); + this.commit_timer = null; + this.commit_task = [{ "clear": true }]; + //return this.db.clear(); + } + + return this; +}; //Index.prototype.pipeline = pipeline; @@ -115,697 +199,72 @@ export default Index; */ Index.prototype.append = function(id, content){ - return this.add(id, content, true); }; -// TODO: -// string + number as text -// boolean, null, undefined as ? - -/** - * @param {!number|string} id - * @param {!string} content - * @param {boolean=} _append - * @param {boolean=} _skip_update - */ - -Index.prototype.add = function(id, content, _append, _skip_update){ - - if(content && (id || (id === 0))){ - - if(!_skip_update && !_append && this.register[id]){ - - return this.update(id, content); - } - - content = this.encode("" + content); - const length = content.length; - - if(length){ - - // check context dupes to skip all contextual redundancy along a document - - const dupes_ctx = create_object(); - const dupes = create_object(); - const depth = this.depth; - const resolution = this.resolution; - - for(let i = 0; i < length; i++){ - - let term = content[this.rtl ? length - 1 - i : i]; - let term_length = term.length; - - // skip dupes will break the context chain - - if(term && (term_length >= this.minlength) && (depth || !dupes[term])){ - - let score = get_score(resolution, length, i); - let token = ""; - - switch(this.tokenize){ - - case "full": - - if(term_length > 2){ - - for(let x = 0; x < term_length; x++){ - - for(let y = term_length; y > x; y--){ - - if((y - x) >= this.minlength){ - - const partial_score = get_score(resolution, length, i, term_length, x); - token = term.substring(x, y); - this.push_index(dupes, token, partial_score, id, _append); - } - } - } - - break; - } - - // fallthrough to next case when term length < 3 - - case "reverse": - - // skip last round (this token exist already in "forward") - - if(term_length > 1){ - - for(let x = term_length - 1; x > 0; x--){ - - token = term[x] + token; - - if(token.length >= this.minlength){ - - const partial_score = get_score(resolution, length, i, term_length, x); - this.push_index(dupes, token, partial_score, id, _append); - } - } - - token = ""; - } - - // fallthrough to next case to apply forward also - - case "forward": - - if(term_length > 1){ - - for(let x = 0; x < term_length; x++){ - - token += term[x]; - - if(token.length >= this.minlength){ - - this.push_index(dupes, token, score, id, _append); - } - } - - break; - } - - // fallthrough to next case when token has a length of 1 - - default: - // case "strict": - - if(this.boost){ - - score = Math.min((score / this.boost(content, term, i)) | 0, resolution - 1); - } - - this.push_index(dupes, term, score, id, _append); - - // context is just supported by tokenizer "strict" - - if(depth){ - - if((length > 1) && (i < (length - 1))){ - - // check inner dupes to skip repeating words in the current context - - const dupes_inner = create_object(); - const resolution = this.resolution_ctx; - const keyword = term; - const size = Math.min(depth + 1, length - i); - - dupes_inner[keyword] = 1; - - for(let x = 1; x < size; x++){ - - term = content[this.rtl ? length - 1 - i - x : i + x]; - - if(term && (term.length >= this.minlength) && !dupes_inner[term]){ - - dupes_inner[term] = 1; - - const context_score = get_score(resolution + ((length / 2) > resolution ? 0 : 1), length, i, size - 1, x - 1); - const swap = this.bidirectional && (term > keyword); - this.push_index(dupes_ctx, swap ? keyword : term, context_score, id, _append, swap ? term : keyword); - } - } - } - } - } - } - } - - this.fastupdate || (this.register[id] = 1); - } - } - - return this; -}; - -/** - * @param {number} resolution - * @param {number} length - * @param {number} i - * @param {number=} term_length - * @param {number=} x - * @returns {number} - */ - -function get_score(resolution, length, i, term_length, x){ - - // console.log("resolution", resolution); - // console.log("length", length); - // console.log("term_length", term_length); - // console.log("i", i); - // console.log("x", x); - // console.log((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1); - - // the first resolution slot is reserved for the best match, - // when a query matches the first word(s). - - // also to stretch score to the whole range of resolution, the - // calculation is shift by one and cut the floating point. - // this needs the resolution "1" to be handled additionally. - - // do not stretch the resolution more than the term length will - // improve performance and memory, also it improves scoring in - // most cases between a short document and a long document - - return i && (resolution > 1) ? ( - - (length + (term_length || 0)) <= resolution ? - - i + (x || 0) - : - ((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1) | 0 - ): - 0; -} - -/** - * @private - * @param dupes - * @param value - * @param score - * @param id - * @param {boolean=} append - * @param {string=} keyword - */ - -Index.prototype.push_index = function(dupes, value, score, id, append, keyword){ - - let arr = keyword ? this.ctx : this.map; - - if(!dupes[value] || (keyword && !dupes[value][keyword])){ - - if(this.optimize){ - - arr = arr[score]; - } - - if(keyword){ - - dupes = dupes[value] || (dupes[value] = create_object()); - dupes[keyword] = 1; - - arr = arr[keyword] || (arr[keyword] = create_object()); - } - else{ - - dupes[value] = 1; - } - - arr = arr[value] || (arr[value] = []); - - if(!this.optimize){ - - arr = arr[score] || (arr[score] = []); - } - - if(!append || !arr.includes(id)){ - - arr[arr.length] = id; - - // add a reference to the register for fast updates - - if(this.fastupdate){ - - const tmp = this.register[id] || (this.register[id] = []); - tmp[tmp.length] = arr; - } - } - } -} - -/** - * @param {string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @returns {Array} - */ - -Index.prototype.search = function(query, limit, options){ - - if(!options){ - - if(!limit && is_object(query)){ - - options = /** @type {Object} */ (query); - query = options["query"]; - } - else if(is_object(limit)){ - - options = /** @type {Object} */ (limit); - } - } - - let result = []; - let length; - let context, suggest, offset = 0; - - if(options){ - - query = options["query"] || query; - limit = options["limit"]; - offset = options["offset"] || 0; - context = options["context"]; - suggest = SUPPORT_SUGGESTION && options["suggest"]; - } - - if(query){ - - query = /** @type {Array} */ (this.encode("" + query)); - length = query.length; - - // TODO: solve this in one single loop below - - if(length > 1){ - - const dupes = create_object(); - const query_new = []; - - for(let i = 0, count = 0, term; i < length; i++){ - - term = query[i]; - - if(term && (term.length >= this.minlength) && !dupes[term]){ - - // this fast path can just apply when not in memory-optimized mode - - if(!this.optimize && !suggest && !this.map[term]){ - - // fast path "not found" - - return result; - } - else{ - - query_new[count++] = term; - dupes[term] = 1; - } - } - } - - query = query_new; - length = query.length; - } - } - - if(!length){ - - return result; - } - - limit || (limit = 100); - - let depth = this.depth && (length > 1) && (context !== false); - let index = 0, keyword; - - if(depth){ - - keyword = query[0]; - index = 1; - } - else{ - - if(length > 1){ - - query.sort(sort_by_length_down); - } - } - - for(let arr, term; index < length; index++){ - - term = query[index]; - - // console.log(keyword); - // console.log(term); - // console.log(""); - - if(depth){ - - arr = this.add_result(result, suggest, limit, offset, length === 2, term, keyword); - - // console.log(arr); - // console.log(result); - - // when suggestion enabled just forward keyword if term was found - // as long as the result is empty forward the pointer also - - if(!suggest || (arr !== false) || !result.length){ - - keyword = term; - } - } - else{ - - arr = this.add_result(result, suggest, limit, offset, length === 1, term); - } - - if(arr){ - - return /** @type {Array} */ (arr); - } - - // apply suggestions on last loop or fallback - - if(suggest && (index === length - 1)){ - - let length = result.length; - - if(!length){ - - if(depth){ - - // fallback to non-contextual search when no result was found - - depth = 0; - index = -1; - - continue; - } - - return result; - } - else if(length === 1){ - - // fast path optimization - - return single_result(result[0], limit, offset); - } - } - } - - return intersect(result, limit, offset, suggest); -}; - -/** - * Returns an array when the result is done (to stop the process immediately), - * returns false when suggestions is enabled and no result was found, - * or returns nothing when a set was pushed successfully to the results - * - * @private - * @param {Array} result - * @param {Array} suggest - * @param {number} limit - * @param {number} offset - * @param {boolean} single_term - * @param {string} term - * @param {string=} keyword - * @return {Array>|boolean|undefined} - */ - -Index.prototype.add_result = function(result, suggest, limit, offset, single_term, term, keyword){ - - let word_arr = []; - let arr = keyword ? this.ctx : this.map; - - if(!this.optimize){ - - arr = get_array(arr, term, keyword, this.bidirectional); - } - - if(arr){ - - let count = 0; - const arr_len = Math.min(arr.length, keyword ? this.resolution_ctx : this.resolution); - - // relevance: - for(let x = 0, size = 0, tmp, len; x < arr_len; x++){ - - tmp = arr[x]; - - if(tmp){ - - if(this.optimize){ - - tmp = get_array(tmp, term, keyword, this.bidirectional); - } - - if(offset){ - - if(tmp && single_term){ - - len = tmp.length; - - if(len <= offset){ - - offset -= len; - tmp = null; - } - else{ - - tmp = tmp.slice(offset); - offset = 0; - } - } - } - - if(tmp){ - - // keep score (sparse array): - //word_arr[x] = tmp; - - // simplified score order: - word_arr[count++] = tmp; - - if(single_term){ - - size += tmp.length; - - if(size >= limit){ - - // fast path optimization - - break; - } - } - } - } - } - - if(count){ - - if(single_term){ - - // fast path optimization - // offset was already applied at this point - - return single_result(word_arr, limit, 0); - } - - result[result.length] = word_arr; - return; - } - } - - // return an empty array will stop the loop, - // to prevent stop when using suggestions return a false value - - return !suggest && word_arr; -}; - -function single_result(result, limit, offset){ - - if(result.length === 1){ - - result = result[0]; - } - else{ - - result = concat(result); - } - - return offset || (result.length > limit) ? - - result.slice(offset, offset + limit) - : - result; -} - -function get_array(arr, term, keyword, bidirectional){ - - if(keyword){ - - // the frequency of the starting letter is slightly less - // on the last half of the alphabet (m-z) in almost every latin language, - // so we sort downwards (https://en.wikipedia.org/wiki/Letter_frequency) - - const swap = bidirectional && (term > keyword); - - arr = arr[swap ? term : keyword]; - arr = arr && arr[swap ? keyword : term]; - } - else{ - - arr = arr[term]; - } - - return arr; -} - Index.prototype.contain = function(id){ - return !!this.register[id]; + if(SUPPORT_PERSISTENT && this.db){ + return this.db.has(id); + } + + return this.reg.has(id); }; Index.prototype.update = function(id, content){ + // todo check the async part + if(this.async /*|| (SUPPORT_PERSISTENT && this.db)*/){ + const self = this; + const res = this.remove(id); + return res.then ? res.then( + () => self.add(id, content) + ) : this.add(id, content); + } + return this.remove(id).add(id, content); }; -/** - * @param {boolean=} _skip_deletion - */ - -Index.prototype.remove = function(id, _skip_deletion){ - - const refs = this.register[id]; - - if(refs){ - - if(this.fastupdate){ - - // fast updates performs really fast but did not fully cleanup the key entries - - for(let i = 0, tmp; i < refs.length; i++){ - - tmp = refs[i]; - tmp.splice(tmp.indexOf(id), 1); - } - } - else{ - - remove_index(this.map, id, this.resolution, this.optimize); - - if(this.depth){ - - remove_index(this.ctx, id, this.resolution_ctx, this.optimize); - } - } - - _skip_deletion || delete this.register[id]; - - if(SUPPORT_CACHE && this.cache){ - - this.cache.del(id); - } - } - - return this; -}; - /** * @param map - * @param id - * @param res - * @param optimize - * @param {number=} resolution * @return {number} */ -function remove_index(map, id, res, optimize, resolution){ +function cleanup_index(map){ let count = 0; if(is_array(map)){ - - // the first array is the score array in both strategies - - if(!resolution){ - - resolution = Math.min(map.length, res); - - for(let x = 0, arr; x < resolution; x++){ - - arr = map[x]; - - if(arr){ - - count = remove_index(arr, id, res, optimize, resolution); - - if(!optimize && !count){ - - // when not memory optimized the score index should removed - - delete map[x]; - } - } - } - } - else{ - - const pos = map.indexOf(id); - - if(pos !== -1){ - - // fast path, when length is 1 or lower then the whole field gets deleted - - if(map.length > 1){ - - map.splice(pos, 1); - count++; - } - } - else{ - - count++; - } + for(let i = 0, arr; i < map.length; i++){ + (arr = map[i]) && + (count += arr.length); } } - else{ - - for(let key in map){ - - count = remove_index(map[key], id, res, optimize, resolution); - - if(!count){ - - delete map[key]; - } - } + else for(const item of map){ + const key = item[0]; + const value = item[1]; + const tmp = cleanup_index(value); + tmp ? count += tmp + : map.delete(key); } return count; } +Index.prototype.cleanup = function(){ + + if(!this.fastupdate){ + DEBUG && console.info("Cleanup the index isn't required when not using \"fastupdate\"."); + return this; + } + + cleanup_index(this.map); + this.depth && + cleanup_index(this.ctx); + + return this; +}; + if(SUPPORT_CACHE){ Index.prototype.searchCache = searchCache; diff --git a/src/index/add.js b/src/index/add.js new file mode 100644 index 0000000..07ea41a --- /dev/null +++ b/src/index/add.js @@ -0,0 +1,285 @@ +// COMPILER BLOCK --> +import { + SUPPORT_COMPRESSION, + SUPPORT_KEYSTORE, + SUPPORT_PERSISTENT +} from "../config.js"; +// <-- COMPILER BLOCK +import { create_object } from "../common.js"; +import Index, { autoCommit } from "../index.js"; +import default_compress from "../compress.js"; +import { KeystoreArray } from "../keystore.js"; + +// TODO: +// string + number as text +// boolean, null, undefined as ? + + +/** + * @param {!number|string} id + * @param {!string} content + * @param {boolean=} _append + * @param {boolean=} _skip_update + */ + +Index.prototype.add = function(id, content, _append, _skip_update){ + + if(content && (id || (id === 0))){ + + // todo check skip_update + //_skip_update = false; + + if(!_skip_update && !_append){ + if(this.reg.has(id)){ + return this.update(id, content); + } + } + + // do not force a string as input + // https://github.com/nextapps-de/flexsearch/issues/432 + content = this.encoder.encode(content); + const word_length = content.length; + + if(word_length){ + + // check context dupes to skip all contextual redundancy along a document + + const dupes_ctx = create_object(); + const dupes = create_object(); + const depth = this.depth; + const resolution = this.resolution; + + for(let i = 0; i < word_length; i++){ + + let term = content[this.rtl ? word_length - 1 - i : i]; + let term_length = term.length; + + // skip dupes will break the context chain + + if(term_length /*&& (term_length >= this.minlength)*/ && (depth || !dupes[term])){ + + let score = this.score + ? this.score(content, term, i, null, 0) + : get_score(resolution, word_length, i); + let token = ""; + + switch(this.tokenize){ + + case "full": + if(term_length > 2){ + for(let x = 0; x < term_length; x++){ + for(let y = term_length; y > x; y--){ + + //if((y - x) >= this.minlength){ + token = term.substring(x, y); + const partial_score = this.score + ? this.score(content, term, i, token, x) + : get_score(resolution, word_length, i, term_length, x); + this.push_index(dupes, token, partial_score, id, _append); + //} + } + } + break; + } + // fallthrough to next case when term length < 3 + case "reverse": + // skip last round (this token exist already in "forward") + if(term_length > 1){ + for(let x = term_length - 1; x > 0; x--){ + token = term[x] + token; + //if(token.length >= this.minlength){ + const partial_score = this.score + ? this.score(content, term, i, token, x) + : get_score(resolution, word_length, i, term_length, x); + this.push_index(dupes, token, partial_score, id, _append); + //} + } + token = ""; + } + + // fallthrough to next case to apply forward also + case "forward": + if(term_length > 1){ + for(let x = 0; x < term_length; x++){ + token += term[x]; + //if(token.length >= this.minlength){ + this.push_index(dupes, token, score, id, _append); + //} + } + break; + } + + // fallthrough to next case when token has a length of 1 + default: + // case "strict": + + // todo move boost to search + // if(this.boost){ + // score = Math.min((score / this.boost(content, term, i)) | 0, resolution - 1); + // } + + this.push_index(dupes, term, score, id, _append); + + // context is just supported by tokenizer "strict" + if(depth){ + + if((word_length > 1) && (i < (word_length - 1))){ + + // check inner dupes to skip repeating words in the current context + const dupes_inner = create_object(); + const resolution = this.resolution_ctx; + const keyword = term; + const size = Math.min(depth + 1, word_length - i); + + dupes_inner[keyword] = 1; + + for(let x = 1; x < size; x++){ + + term = content[this.rtl ? word_length - 1 - i - x : i + x]; + + if(term /*&& (term.length >= this.minlength)*/ && !dupes_inner[term]){ + + dupes_inner[term] = 1; + + const context_score = this.score + ? this.score(content, keyword, i, term, x) + : get_score(resolution + ((word_length / 2) > resolution ? 0 : 1), word_length, i, size - 1, x - 1); + const swap = this.bidirectional && (term > keyword); + this.push_index(dupes_ctx, swap ? keyword : term, context_score, id, _append, swap ? term : keyword); + } + } + } + } + } + } + } + + this.fastupdate || this.reg.add(id); + } + else{ + content = ""; + } + } + + if(SUPPORT_PERSISTENT && this.db){ + // when the term has no valid content (e.g. empty), + // then it was not added to the ID registry for removal + content || this.commit_task.push({ "del": id }); + this.commit_auto && autoCommit(this); + } + + return this; +}; + +/** + * @private + * @param dupes + * @param term + * @param score + * @param id + * @param {boolean=} append + * @param {string=} keyword + */ + +Index.prototype.push_index = function(dupes, term, score, id, append, keyword){ + + let arr = keyword ? this.ctx : this.map; + let tmp; + + if(!dupes[term] || !keyword || !(tmp = dupes[term])[keyword]){ + + if(keyword){ + + dupes = tmp || (dupes[term] = create_object()); + dupes[keyword] = 1; + + if(SUPPORT_COMPRESSION && this.compress){ + keyword = default_compress(keyword); + } + + tmp = arr.get(keyword); + tmp ? arr = tmp + : arr.set(keyword, arr = new Map()); + } + else{ + + dupes[term] = 1; + } + + if(SUPPORT_COMPRESSION && this.compress){ + term = default_compress(term); + } + + tmp = arr.get(term); + tmp ? arr = tmp : arr.set(term, arr = tmp = []); + // the ID array will be upgraded dynamically + arr = arr[score] || (arr[score] = []); + + if(!append || !arr.includes(id)){ + + // auto-upgrade to keystore array if max size exceeded + if(SUPPORT_KEYSTORE){ + if(arr.length === 2**31-1 /*|| !(arr instanceof KeystoreArray)*/){ + const keystore = new KeystoreArray(arr); + if(this.fastupdate){ + for(let value of this.reg.values()){ + if(value.includes(arr)){ + value[value.indexOf(arr)] = keystore; + } + } + } + tmp[score] = arr = keystore; + } + } + + arr.push(id); + + // add a reference to the register for fast updates + if(this.fastupdate){ + const tmp = this.reg.get(id); + tmp ? tmp.push(arr) + : this.reg.set(id, [arr]); + } + } + } +} + +/** + * @param {number} resolution + * @param {number} length + * @param {number} i + * @param {number=} term_length + * @param {number=} x + * @returns {number} + */ + +function get_score(resolution, length, i, term_length, x){ + + // console.log("resolution", resolution); + // console.log("length", length); + // console.log("term_length", term_length); + // console.log("i", i); + // console.log("x", x); + // console.log((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1); + + // the first resolution slot is reserved for the best match, + // when a query matches the first word(s). + + // also to stretch score to the whole range of resolution, the + // calculation is shift by one and cut the floating point. + // this needs the resolution "1" to be handled additionally. + + // do not stretch the resolution more than the term length will + // improve performance and memory, also it improves scoring in + // most cases between a short document and a long document + + return i && (resolution > 1) ? ( + + (length + (term_length || 0)) <= resolution ? + + i + (x || 0) + : + ((resolution - 1) / (length + (term_length || 0)) * (i + (x || 0)) + 1) | 0 + ): + 0; +} diff --git a/src/index/remove.js b/src/index/remove.js new file mode 100644 index 0000000..17e71cd --- /dev/null +++ b/src/index/remove.js @@ -0,0 +1,159 @@ +// COMPILER BLOCK --> +import { + SUPPORT_CACHE, + SUPPORT_COMPRESSION, + SUPPORT_KEYSTORE, + SUPPORT_PERSISTENT +} from "../config.js"; +// <-- COMPILER BLOCK +import { create_object, is_array } from "../common.js"; +import Index, { autoCommit } from "../index.js"; +import default_compress from "../compress.js"; + +/** + * @param {boolean=} _skip_deletion + */ + +Index.prototype.remove = function(id, _skip_deletion){ + + const refs = this.reg.size && ( + this.fastupdate + ? this.reg.get(id) + : this.reg.has(id) + ); + + if(refs){ + + if(this.fastupdate){ + + // fast updates did not fully cleanup the key entries + + for(let i = 0, tmp; i < refs.length; i++){ + if((tmp = refs[i])){ + // todo check + //if(tmp.length < 1) throw new Error("invalid length"); + //if(tmp.indexOf(id) < 0) throw new Error("invalid id"); + if(tmp.length < 2){ + tmp.pop(); + } + else{ + const index = tmp.indexOf(id); + index === refs.length - 1 + ? tmp.pop() + : tmp.splice(index, 1); + } + } + else{ + // todo investigate empty entries + } + } + + // todo variation which cleans up, requires to push [ctx, key] instead of arr to the index.reg + // for(let i = 0, arr, term, keyword; i < refs.length; i++){ + // arr = refs[i]; + // if(typeof arr === "string"){ + // arr = this.map.get(term = arr); + // } + // else{ + // arr = this.ctx.get(keyword = arr[0]); + // arr && (arr = arr.get(arr[1])); + // } + // let counter = 0, found; + // if(arr && arr.length){ + // for(let j = 0, tmp; j < arr.length; j++){ + // if((tmp = arr[j])){ + // if(!found && tmp.length){ + // const index = tmp.indexOf(id); + // if(index >= 0){ + // tmp.splice(index, 1); + // // the index [ctx, key]:[res, id] is unique + // found = 1; + // } + // } + // if(tmp.length){ + // counter++; + // if(found){ + // break; + // } + // } + // else{ + // delete arr[j]; + // } + // } + // } + // } + // if(!counter){ + // keyword + // ? this.ctx.delete(keyword) + // : this.map.delete(term); + // } + // } + } + else{ + + remove_index(this.map, id/*, this.resolution*/); + this.depth && + remove_index(this.ctx, id/*, this.resolution_ctx*/); + } + + _skip_deletion || this.reg.delete(id); + } + + if(SUPPORT_PERSISTENT && this.db){ + this.commit_task.push({ "del": id }); + this.commit_auto && autoCommit(this); + //return this.db.remove(id); + } + + // the cache could be used outside the InMemory store + if(SUPPORT_CACHE && this.cache){ + this.cache.remove(id); + } + + return this; +}; + +/** + * @param map + * @param id + * @return {number} + */ + +function remove_index(map, id){ + + // a check counter of filled resolution slots + // to prevent removing the field + let count = 0; + + if(is_array(map)){ + for(let x = 0, arr, index; x < map.length; x++){ + if((arr = map[x]) && arr.length){ + index = arr.indexOf(id); + if(index >= 0){ + if(arr.length > 1){ + arr.splice(index, 1); + count++; + } + else{ + // remove resolution slot + delete map[x]; + } + // the index key:[res, id] is unique + break; + } + else{ + count++; + } + } + } + } + else for(let item of map){ + const key = item[0]; + const value = item[1]; + const tmp = remove_index(value, id); + tmp ? count += tmp + : map.delete(key); + } + + return count; +} diff --git a/src/index/search.js b/src/index/search.js new file mode 100644 index 0000000..c3e8b14 --- /dev/null +++ b/src/index/search.js @@ -0,0 +1,523 @@ +// COMPILER BLOCK --> +import { + SUPPORT_CACHE, + SUPPORT_COMPRESSION, + SUPPORT_DOCUMENT, + SUPPORT_PERSISTENT, + SUPPORT_RESOLVER, + SUPPORT_SUGGESTION, + SUPPORT_TAGS +} from "../config.js"; +// <-- COMPILER BLOCK + +import { SearchOptions } from "../type.js"; +import { create_object, is_object, sort_by_length_down } from "../common.js"; +import Index from "../index.js"; +import default_compress from "../compress.js"; +import Resolver from "../resolver.js"; +import { intersect } from "../intersect.js"; +import resolve_default from "../resolve/default.js"; + +let global_resolve = 1; +export function set_resolve(resolve){ + global_resolve = resolve; +} + +/** + * @param {string|SearchOptions} query + * @param {number|SearchOptions=} limit + * @param {SearchOptions=} options + * @returns {Array|Resolver|Promise} + */ + +Index.prototype.search = function(query, limit, options){ + + if(!options){ + if(!limit && is_object(query)){ + options = /** @type {!SearchOptions} */ (query); + query = ""; + } + else if(is_object(limit)){ + options = /** @type {!SearchOptions} */ (limit); + limit = 0; + } + } + + let result = []; + let length; + let context, suggest, offset = 0, resolve, enrich, tag, cache; + + if(options){ + query = options.query || query; + limit = options.limit || limit; + offset = options.offset || 0; + context = options.context; + suggest = SUPPORT_SUGGESTION && options.suggest; + resolve = !SUPPORT_RESOLVER || (global_resolve && options.resolve !== false); + resolve || (global_resolve = 0); + enrich = resolve && options.enrich; + tag = SUPPORT_PERSISTENT && SUPPORT_DOCUMENT && SUPPORT_TAGS && this.db && options.tag; + } + else{ + resolve = !SUPPORT_RESOLVER || this.resolve || global_resolve; + } + + // todo: term deduplication during encoding when context is disabled + + // do not force a string as input + // https://github.com/nextapps-de/flexsearch/issues/432 + query = /** @type {Array} */ (this.encoder.encode(query)); + length = query.length; + limit || !resolve || (limit = 100); + + // fast path single term + if(length === 1){ + return single_term_query.call( + this, + query[0], // term + "", // ctx + limit, + offset, + resolve, + enrich, + tag + ); + } + + // TODO: dedupe terms within encoder? + // TODO: deduplication will break the context chain + + context = this.depth && context !== false; + + // fast path single context + if(length === 2 && context && !suggest){ + return single_term_query.call( + this, + query[0], // term + query[1], // ctx + limit, + offset, + resolve, + enrich, + tag + ); + } + + let maxlength = 0; + let minlength = 0; + + if(length > 1){ + + // term deduplication will break the context chain + // todo add context to dupe check + const dupes = create_object(); + const query_new = []; + + // if(context){ + // keyword = query[0]; + // dupes[keyword] = 1; + // query_new.push(keyword); + // maxlength = minlength = keyword.length; + // i = 1; + // } + + for(let i = 0, term; i < length; i++){ + + term = query[i]; + + if(term && !dupes[term]){ + + // todo add keyword check + // this fast path can't apply to persistent indexes + if(!suggest && !(SUPPORT_PERSISTENT && this.db) && !this.get_array(term/*, keyword*/)){ + + // fast path "not found" + return !SUPPORT_RESOLVER || resolve + ? result + : new Resolver(result); + } + else{ + + query_new.push(term); + dupes[term] = 1; + } + + const term_length = term.length; + maxlength = Math.max(maxlength, term_length); + minlength = minlength ? Math.min(minlength, term_length) : term_length; + } + // else if(term && (!this.depth || context === false)){ + // query_new.push(term); + // } + } + + query = query_new; + length = query.length; + } + + // the term length could be changed after deduplication + + if(!length){ + return !SUPPORT_RESOLVER || resolve + ? result + : new Resolver(result); + } + + let index = 0, keyword; + + // fast path single term + if(length === 1){ + return single_term_query.call( + this, + query[0], // term + "", // ctx + limit, + offset, + resolve, + enrich, + tag + ); + } + + // fast path single context + if(length === 2 && context && !suggest){ + return single_term_query.call( + this, + query[0], // term + query[1], // ctx + limit, + offset, + resolve, + enrich, + tag + ); + } + + if(length > 1){ + if(context){ + // start with context right away + keyword = query[0]; + index = 1; + } + // todo + else if(maxlength > 9 && (maxlength / minlength) > 3){ + // sorting terms will break the context chain + // bigger terms has less occurrence + // this might also reduce the intersection task + // todo check intersection order + query.sort(sort_by_length_down); + } + } + + if(SUPPORT_PERSISTENT && this.db){ + + if(this.db.search){ + // when the configuration is not supported it returns false + const result = this.db.search(this, query, limit, offset, suggest, resolve, enrich, tag); + if(result !== false) return result; + } + + const self = this; + return (async function(){ + + for(let arr, term; index < length; index++){ + + term = query[index]; + + if(keyword){ + + arr = await self.get_array(term, keyword); + arr = add_result( + arr, + result, + suggest, + self.resolution_ctx, + /** @type {!number} */ (limit), + offset, + length === 2 + /*, term, keyword*/ + ); + + // the context is a moving window where the keyword is going forward like a cursor + // 1. when suggestion enabled just forward keyword if term was found + // 2. as long as the result is empty forward the pointer also + if(!suggest || (arr !== false) || !result.length){ + keyword = term; + } + } + else{ + + arr = await self.get_array(term); + arr = add_result( + arr, + result, + suggest, + self.resolution, + /** @type {!number} */ (limit), + offset, + length === 1 + /*, term*/ + ); + } + + // limit reached + if(arr){ + return arr; + } + + // apply suggestions on last loop + if(suggest && (index === length - 1)){ + let length = result.length; + if(!length){ + // fallback to non-contextual search when no result was found + if(keyword){ + keyword = ""; + index = -1; + continue; + } + return result; + } + else if(length === 1){ + return !SUPPORT_RESOLVER || resolve + ? resolve_default(result[0], /** @type {number} */ (limit), offset) + : new Resolver(result[0]); + } + } + } + + return !SUPPORT_RESOLVER || resolve + ? intersect(result, /** @type {number} */ (limit), offset, suggest) + : new Resolver(result[0]) + }()); + } + + for(let arr, term; index < length; index++){ + + term = query[index]; + + if(keyword){ + + arr = this.get_array(term, keyword); + arr = /*this.*/add_result( + arr, + result, + suggest, + this.resolution_ctx, + /** @type {!number} */ (limit), + offset, + length === 2 + /*, term, keyword*/ + ); + + // 1. when suggestion enabled just forward keyword if term was found + // 2. as long as the result is empty forward the pointer also + if(!suggest || (arr !== false) || !result.length){ + keyword = term; + } + } + else{ + + arr = this.get_array(term); + arr = /*this.*/add_result( + arr, + result, + suggest, + this.resolution, + /** @type {!number} */ (limit), + offset, + length === 1 + /*, term*/ + ); + } + + // limit reached + if(arr){ + return /** @type {Array} */ (arr); + } + + // apply suggestions on last loop + if(suggest && (index === length - 1)){ + const length = result.length; + if(!length){ + // fallback to non-contextual search when no result was found + if(keyword){ + keyword = ""; + index = -1; + continue; + } + return result; + } + else if(length === 1){ + return !SUPPORT_RESOLVER || resolve + ? resolve_default(result[0], limit, offset) + : new Resolver(result[0]); + } + } + } + + return !SUPPORT_RESOLVER || resolve + ? intersect(result, limit, offset, suggest) + : new Resolver(result[0]); +}; + +/** + * @param term + * @param keyword + * @param limit + * @param offset + * @param resolve + * @param enrich + * @param tag + * @this Index + * @return {Array|Resolver} + */ + +function single_term_query(term, keyword, limit, offset, resolve, enrich, tag){ + + const result = this.get_array(term, keyword, limit, offset, resolve, enrich, tag); + + if(SUPPORT_PERSISTENT && this.db){ + return result.then(function(result){ + if(resolve) return result; + return result && result.length + ? (!SUPPORT_RESOLVER || resolve ? resolve_default(result, limit, offset): new Resolver(result)) + : !SUPPORT_RESOLVER || resolve ? [] : new Resolver([]); + }); + } + + return result && result.length + ? (!SUPPORT_RESOLVER || resolve ? resolve_default(result, limit, offset) : new Resolver(result)) + : !SUPPORT_RESOLVER || resolve ? [] : new Resolver([]); +} + +/** + * Returns a 1-dimensional finalized array when the result is done (fast path return), + * returns false when suggestions is enabled and no result was found, + * or returns nothing when a set was pushed successfully to the results + * + * @private + * @param {Array} arr + * @param {Array} result + * @param {Array} suggest + * @param {number} resolution + * @param {number} limit + * @param {number} offset + * @param {boolean} single_term + * @return {Array|boolean|undefined} + */ + +function add_result(arr, result, suggest, resolution, limit, offset, single_term/*, term, keyword*/){ + + let word_arr = []; + //let arr;// = keyword ? this.ctx : this.map; + //arr = this.get_array(term, keyword); + + if(arr){ + + //const resolution = Math.min(arr.length, keyword ? this.resolution_ctx : this.resolution); + // apply reduced resolution for queries + resolution = Math.min(arr.length, resolution); + + for(let x = 0, size = 0, tmp; x < resolution; x++){ + if((tmp = arr[x])){ + + if(offset){ + // apply offset right here on single terms + if(tmp && single_term){ + if(tmp.length <= offset){ + offset -= tmp.length; + tmp = null; + } + else{ + tmp = tmp.slice(offset); + offset = 0; + } + } + } + + if(tmp){ + + // keep score (sparse array): + word_arr[x] = tmp; + // simplified score order: + //word_arr.push(tmp); + + if(single_term){ + size += tmp.length; + if(size >= limit){ + // fast path: + // a single term does not need to pre-collect results + break; + } + } + } + } + } + + if(word_arr.length){ + if(single_term){ + // fast path optimization + // offset was already applied at this point + // return an array will stop the query process immediately + return resolve_default(word_arr, limit, 0); + } + + result.push(word_arr); + // return nothing will continue the query + return; + } + } + + // 1. return an empty array will stop the loop + // 2. return a false value to prevent stop when using suggestions + return !suggest && word_arr; +} + +Index.prototype.get_array = function(term, keyword, limit, offset, resolve, enrich, tag){ + + let arr, swap; + + if(keyword){ + swap = this.bidirectional && (term > keyword); + } + + if(SUPPORT_COMPRESSION && this.compress){ + term = default_compress(term); + keyword && (keyword = default_compress(keyword)); + } + + if(SUPPORT_PERSISTENT && this.db){ + return keyword + ? this.db.get( + swap ? keyword : term, // term + swap ? term : keyword, // ctx + limit, + offset, + resolve, + enrich, + tag + ) + : this.db.get( + term, + "", // ctx + limit, + offset, + resolve, + enrich, + tag + ); + } + + if(keyword){ + // the frequency of the starting letter is slightly less + // on the last half of the alphabet (m-z) in almost every latin language, + // so we sort downwards (https://en.wikipedia.org/wiki/Letter_frequency) + arr = this.ctx.get(swap ? term : keyword); + arr = arr && arr.get(swap ? keyword : term); + } + else{ + arr = this.map.get(term); + } + + return arr; +} diff --git a/src/intersect.js b/src/intersect.js index 16828e6..5e3eee5 100644 --- a/src/intersect.js +++ b/src/intersect.js @@ -1,4 +1,235 @@ -import { create_object, concat } from "./common.js"; +import { create_object, concat, sort_by_length_up, get_max_len } from "./common.js"; + +/* + + from -> result[ + res[score][id], + res[score][id], + ] + + to -> [id] + + */ + +/** + * Implementation based on Object[key] provides better suggestions + * capabilities and has less performance scaling issues on large indexes. + * + * @param arrays + * @param limit + * @param offset + * @param {boolean|Array=} suggest + * @param {boolean=} resolve + * @returns {Array} + */ + +export function intersect(arrays, limit, offset, suggest, resolve) { + + const length = arrays.length; + + // todo remove + // if(length < 2){ + // throw new Error("Not optimized intersect"); + // } + + let result = []; + let size = 0; + let check; + let check_suggest; + let check_new; + let res_arr; + + if(suggest){ + suggest = []; + } + + // 1. a reversed order prioritize the order of words from a query + // because it ends with the first term. + // 2. process terms in reversed order often has advantage for + // the fast path "end reached". + + // alternatively the results could be sorted by length up + //arrays.sort(sort_by_length_up); + + // todo the outer loop should be the res array instead of term array, + // this isn't just simple because the intersection calculation needs to reflect this + //const maxlen = get_max_len(arrays); + + for(let x = length - 1, found; x >= 0; x--){ + //for(let x = 0, found; x < length; x++){ + + res_arr = arrays[x]; + check_new = create_object(); + found = !check; + + // process relevance in forward order (direction is + // important for adding IDs during the last round) + + for(let y = 0, ids; y < res_arr.length; y++){ + + ids = res_arr[y]; + if(!ids || !ids.length) continue; + + for(let z = 0, id; z < ids.length; z++){ + + id = ids[z]; + + // check exists starting from the 2nd slot + if(check){ + if(check[id]){ + + // check if in last round + if(!x){ + //if(x === length - 1){ + + if(offset){ + offset--; + } + else{ + + result[size++] = id; + + if(size === limit){ + // fast path "end reached" + return result /*resolve === false + ? { result, suggest } + :*/ + } + } + } + + if(x || suggest){ + //if((x < length - 1) || suggest){ + check_new[id] = 1; + } + + found = true; + } + + if(suggest){ + + if(!check_suggest[id]){ + check_suggest[id] = 1; + const arr = suggest[y] || (suggest[y] = []); + arr.push(id); + } + + // OLD: + // + // check_idx = (check_suggest[id] || 0) + 1; + // check_suggest[id] = check_idx; + // + // // do not adding IDs which are already included in the result (saves one loop) + // // the first intersection match has the check index 2, so shift by -2 + // + // if(check_idx < length){ + // + // const tmp = suggest[check_idx - 2] || (suggest[check_idx - 2] = []); + // tmp[tmp.length] = id; + // } + } + } + else{ + + // pre-fill in first round + check_new[id] = 1; + } + } + } + + if(suggest){ + + // re-use the first pre-filled check for suggestions + check || (check_suggest = check_new); + } + else if(!found){ + + return []; + } + + check = check_new; + } + + // return intermediate result + // if(resolve === false){ + // return { result, suggest }; + // } + + if(suggest){ + + // needs to iterate in reverse direction + for(let x = suggest.length - 1, ids, len; x >= 0; x--){ + + ids = suggest[x]; + len = ids.length; + + for(let y = 0, id; y < len; y++){ + + id = ids[y]; + + if(!check[id]){ + + if(offset){ + offset--; + } + else{ + + result[size++] = id; + + if(size === limit){ + // fast path "end reached" + return result; + } + } + + check[id] = 1; + } + } + } + } + + return result; +} + +/** + * @param mandatory + * @param arrays + * @returns {Array} + */ + +export function intersect_union(mandatory, arrays) { + + const check = create_object(); + const union = create_object(); + const result = []; + + for(let x = 0; x < mandatory.length; x++){ + + check[mandatory[x]] = 1; + } + + for(let x = 0, arr; x < arrays.length; x++){ + + arr = arrays[x]; + + for(let y = 0, id; y < arr.length; y++){ + + id = arr[y]; + + if(check[id]){ + + if(!union[id]){ + + union[id] = 1; + result.push(id); + } + } + } + } + + return result; +} + /** * Implementation based on Array.includes() provides better performance, @@ -193,205 +424,3 @@ import { create_object, concat } from "./common.js"; // // return result; // } - -/** - * Implementation based on Object[key] provides better suggestions - * capabilities and has less performance scaling issues on large indexes. - * - * @param arrays - * @param limit - * @param offset - * @param {boolean|Array=} suggest - * @returns {Array} - */ - -export function intersect(arrays, limit, offset, suggest) { - - const length = arrays.length; - let result = []; - let check; - let check_suggest; - let size = 0; - - if(suggest){ - - suggest = []; - } - - // process terms in reversed order often has advantage for the fast path "end reached". - // also a reversed order prioritize the order of words from a query. - - for(let x = length - 1; x >= 0; x--){ - - const word_arr = arrays[x]; - const word_arr_len = word_arr.length; - const check_new = create_object(); - - let found = !check; - - // process relevance in forward order (direction is - // important for adding IDs during the last round) - - for(let y = 0; y < word_arr_len; y++){ - - const arr = word_arr[y]; - const arr_len = arr.length; - - if(arr_len){ - - // loop through IDs - - for(let z = 0, check_idx, id; z < arr_len; z++){ - - id = arr[z]; - - if(check){ - - if(check[id]){ - - // check if in last round - - if(!x){ - - if(offset){ - - offset--; - } - else{ - - result[size++] = id; - - if(size === limit){ - - // fast path "end reached" - - return result; - } - } - } - - if(x || suggest){ - - check_new[id] = 1; - } - - found = true; - } - - if(suggest){ - - check_idx = (check_suggest[id] || 0) + 1; - check_suggest[id] = check_idx; - - // do not adding IDs which are already included in the result (saves one loop) - // the first intersection match has the check index 2, so shift by -2 - - if(check_idx < length){ - - const tmp = suggest[check_idx - 2] || (suggest[check_idx - 2] = []); - tmp[tmp.length] = id; - } - } - } - else{ - - // pre-fill in first round - - check_new[id] = 1; - } - } - } - } - - if(suggest){ - - // re-use the first pre-filled check for suggestions - - check || (check_suggest = check_new); - } - else if(!found){ - - return []; - } - - check = check_new; - } - - if(suggest){ - - // needs to iterate in reverse direction - - for(let x = suggest.length - 1, arr, len; x >= 0; x--){ - - arr = suggest[x]; - len = arr.length; - - for(let y = 0, id; y < len; y++){ - - id = arr[y]; - - if(!check[id]){ - - if(offset){ - - offset--; - } - else{ - - result[size++] = id; - - if(size === limit){ - - // fast path "end reached" - - return result; - } - } - - check[id] = 1; - } - } - } - } - - return result; -} - -/** - * @param mandatory - * @param arrays - * @returns {Array} - */ - -export function intersect_union(mandatory, arrays) { - - const check = create_object(); - const union = create_object(); - const result = []; - - for(let x = 0; x < mandatory.length; x++){ - - check[mandatory[x]] = 1; - } - - for(let x = 0, arr; x < arrays.length; x++){ - - arr = arrays[x]; - - for(let y = 0, id; y < arr.length; y++){ - - id = arr[y]; - - if(check[id]){ - - if(!union[id]){ - - union[id] = 1; - result[result.length] = id; - } - } - } - } - - return result; -} \ No newline at end of file diff --git a/src/keystore.js b/src/keystore.js new file mode 100644 index 0000000..83afb90 --- /dev/null +++ b/src/keystore.js @@ -0,0 +1,400 @@ +import { create_object } from "./common.js"; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreObj(bitlength = 8){ + + if(!(this instanceof KeystoreObj)){ + return new KeystoreObj(bitlength); + } + + this.index = create_object(); + this.keys = []; + + if(bitlength > 32){ + this.crc = lcg64; + this.bit = BigInt(bitlength); + } + else { + this.crc = lcg; + this.bit = bitlength; + } + + return /*this.proxy =*/ new Proxy(this, { + get(target, key) { + const address = target.crc(key); + const obj = target.index[address]; + return obj && obj[key]; + }, + set(target, key, value){ + const address = target.crc(key); + let obj = target.index[address]; + if(!obj){ + target.index[address] = obj = create_object(); + target.keys.push(address); + } + obj[key] = value; + return true; + }, + delete(target, key){ + const address = target.crc(key); + const obj = target.index[address]; + obj && delete obj[key]; + return true; + } + }); +} + +KeystoreObj.prototype.clear = function(){ + this.index = create_object(); + this.keys = []; +}; + +// KeystoreObj.prototype.destroy = function(){ +// this.index = null; +// this.keys = null; +// this.proxy = null; +// }; + +function _slice(self, start, end, splice){ + let arr = []; + for(let i = 0, index; i < self.index.length; i++){ + index = self.index[i]; + if(start >= index.length){ + start -= index.length; + } + else{ + const tmp = index[splice ? "splice" : "slice"](start, end); + const length = tmp.length; + if(length){ + arr = arr.length + ? arr.concat(tmp) + : tmp; + end -= length; + if(splice) self.length -= length; + if(!end) break; + } + start = 0; + } + } + return arr; +} + +/** + * @param arr + * @constructor + */ + +export function KeystoreArray(arr){ + + if(!(this instanceof KeystoreArray)){ + return new KeystoreArray(arr); + } + + this.index = arr ? [arr] : []; + this.length = arr ? arr.length : 0; + const self = this; + + return /*this.proxy =*/ new Proxy([], { + get(target, key) { + if(key === "length"){ + return self.length; + } + if(key === "push"){ + return function(value){ + self.index[self.index.length - 1].push(value); + self.length++; + } + } + if(key === "pop"){ + return function(){ + if(self.length){ + self.length--; + return self.index[self.index.length - 1].pop(); + } + } + } + if(key === "indexOf"){ + return function(key){ + let index = 0; + for(let i = 0, arr, tmp; i < self.index.length; i++){ + arr = self.index[i]; + //if(!arr.includes(key)) continue; + tmp = arr.indexOf(key); + if(tmp >= 0) return index + tmp; + index += arr.length; + } + return -1; + } + } + if(key === "includes"){ + return function(key){ + for(let i = 0; i < self.index.length; i++){ + if(self.index[i].includes(key)){ + return true; + } + } + return false; + } + } + if(key === "slice"){ + return function(start, end){ + return _slice( + self, + start || 0, + end || self.length, + false + ); + } + } + if(key === "splice"){ + return function(start, end){ + return _slice( + self, + start || 0, + end || self.length, + // splice: + true + ); + } + } + if(key === "constructor"){ + return Array; + } + if(typeof key === "symbol" /*|| isNaN(key)*/){ + // not supported + return; + } + const index = key / (2**31) | 0; + const arr = self.index[index]; + return arr && arr[key]; + }, + set(target, key, value){ + const index = key / (2**31) | 0; + const arr = self.index[index] || (self.index[index] = []); + arr[key] = value; + self.length++; + return true; + } + }); +} + +KeystoreArray.prototype.clear = function(){ + this.index.length = 0; +}; + +KeystoreArray.prototype.destroy = function(){ + this.index = null; + this.proxy = null; +}; + +KeystoreArray.prototype.push = function(val){}; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreMap(bitlength = 8){ + + if(!(this instanceof KeystoreMap)){ + return new KeystoreMap(bitlength); + } + + this.index = create_object(); + this.refs = []; + this.size = 0; + + if(bitlength > 32){ + this.crc = lcg64; + this.bit = BigInt(bitlength); + } + else { + this.crc = lcg; + this.bit = bitlength; + } +} + +KeystoreMap.prototype.get = function(key) { + const address = this.crc(key); + const map = this.index[address]; + return map && map.get(key); +}; + +KeystoreMap.prototype.set = function(key, value){ + const address = this.crc(key); + let map = this.index[address]; + if(map){ + let size = map.size; + map.set(key, value); + size -= map.size; + size && this.size++; + } + else{ + this.index[address] = map = new Map([[key, value]]); + this.refs.push(map); + } +}; + +/** + * @param bitlength + * @constructor + */ + +export function KeystoreSet(bitlength = 8){ + + if(!(this instanceof KeystoreSet)){ + return new KeystoreSet(bitlength); + } + + // using plain Object with numeric key access + // just for max performance + this.index = create_object(); + this.refs = []; + + if(bitlength > 32){ + this.crc = lcg64; + this.bit = BigInt(bitlength); + } + else { + this.crc = lcg; + this.bit = bitlength; + } +} + +KeystoreSet.prototype.add = function(key){ + const address = this.crc(key); + let set = this.index[address]; + if(set){ + let size = set.size; + set.add(key); + size -= set.size; + size && this.size++; + } + else{ + this.index[address] = set = new Set([key]); + this.refs.push(set); + } +}; + +KeystoreMap.prototype.has = +KeystoreSet.prototype.has = function(key) { + const address = this.crc(key); + const map_or_set = this.index[address]; + return map_or_set && map_or_set.has(key); +}; + +/* +KeystoreMap.prototype.size = +KeystoreSet.prototype.size = function(){ + let size = 0; + const values = Object.values(this.index); + for(let i = 0; i < values.length; i++){ + size += values[i].size; + } + return size; +}; +*/ + +KeystoreMap.prototype.delete = +KeystoreSet.prototype.delete = function(key){ + const address = this.crc(key); + const map_or_set = this.index[address]; + // set && (set.size === 1 + // ? this.index.delete(address) + // : set.delete(key)); + map_or_set && + map_or_set.delete(key) && + this.size--; +}; + +KeystoreMap.prototype.clear = +KeystoreSet.prototype.clear = function(){ + this.index = create_object(); + this.refs = []; + this.size = 0; +}; + +// KeystoreMap.prototype.destroy = +// KeystoreSet.prototype.destroy = function(){ +// this.index = null; +// this.refs = null; +// this.proxy = null; +// }; + +KeystoreMap.prototype.values = +KeystoreSet.prototype.values = function*(){ + // alternatively iterate through this.keys[] + //const refs = Object.values(this.index); + for(let i = 0; i < this.refs.length; i++){ + for(let value of this.refs[i].values()){ + yield value; + } + } +}; + +KeystoreMap.prototype.keys = +KeystoreSet.prototype.keys = function*(){ + //const values = Object.values(this.index); + for(let i = 0; i < this.refs.length; i++){ + for(let key of this.refs[i].keys()){ + yield key; + } + } +}; + +KeystoreMap.prototype.entries = +KeystoreSet.prototype.entries = function*(){ + //const values = Object.values(this.index); + for(let i = 0; i < this.refs.length; i++){ + for(let entry of this.refs[i].entries()){ + yield entry; + } + } +}; + +/** + * Linear Congruential Generator (LCG) + * @param str + * @this {KeystoreMap|KeystoreSet} + */ + +function lcg(str) { + let range = 2 ** this.bit - 1; + if(typeof str == "number"){ + return str & range; + } + let crc = 0, bit = this.bit + 1; + for(let i = 0; i < str.length; i++) { + crc = (crc * bit ^ str.charCodeAt(i)) & range; + } + // shift Int32 to UInt32 because negative numbers + // extremely slows down key lookup + return this.bit === 32 + ? crc + 2 ** 31 + : crc;// & 0xFFFF; +} + +/** + * @param str + * @this {KeystoreMap|KeystoreSet} + */ + +function lcg64(str) { + let range = BigInt(2) ** /** @type {!BigInt} */ (this.bit) - BigInt(1); + let type = typeof str; + if(type === "bigint"){ + return /** @type {!BigInt} */ (str) & range; + } + if(type === "number"){ + return BigInt(str) & range; + } + let crc = BigInt(0), bit = /** @type {!BigInt} */ (this.bit) + BigInt(1); + for(let i = 0; i < str.length; i++){ + crc = (crc * bit ^ BigInt(str.charCodeAt(i))) & range; + } + return crc;// & 0xFFFFFFFFFFFFFFFF; +} diff --git a/src/lang.js b/src/lang.js deleted file mode 100644 index 2882fd9..0000000 --- a/src/lang.js +++ /dev/null @@ -1,321 +0,0 @@ -import { IndexInterface } from "./type.js"; -import { create_object, get_keys } from "./common.js"; - -/** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} _collapse - * @returns {string|Array} - * @this IndexInterface - */ - -export function pipeline(str, normalize, split, _collapse){ - - if(str){ - - if(normalize){ - - str = replace(str, /** @type {Array} */ (normalize)); - } - - if(this.matcher){ - - str = replace(str, this.matcher); - } - - if(this.stemmer && (str.length > 1)){ - - str = replace(str, this.stemmer); - } - - if(_collapse && (str.length > 1)){ - - str = collapse(str); - } - - if(split || (split === "")){ - - const words = str.split(/** @type {string|RegExp} */ (split)); - - return this.filter ? filter(words, this.filter) : words; - } - } - - return str; -} - -// TODO improve normalize + remove non-delimited chars like in "I'm" + split on whitespace+ - -export const regex_whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; -// https://github.com/nextapps-de/flexsearch/pull/414 -//export const regex_whitespace = /[\s\xA0\u2000-\u200B\u2028\u2029\u3000\ufeff!"#$%&'()*+,\-./:;<=>?@[\\\]^_`{|}~]/ -const regex_normalize = /[\u0300-\u036f]/g; - -export function normalize(str){ - - if(str.normalize){ - - str = str.normalize("NFD").replace(regex_normalize, ""); - } - - return str; -} - -/** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} _collapse - * @returns {string|Array} - */ - -// FlexSearch.prototype.pipeline = function(str, normalize, split, _collapse){ -// -// if(str){ -// -// if(normalize && str){ -// -// str = replace(str, /** @type {Array} */ (normalize)); -// } -// -// if(str && this.matcher){ -// -// str = replace(str, this.matcher); -// } -// -// if(this.stemmer && str.length > 1){ -// -// str = replace(str, this.stemmer); -// } -// -// if(_collapse && str.length > 1){ -// -// str = collapse(str); -// } -// -// if(str){ -// -// if(split || (split === "")){ -// -// const words = str.split(/** @type {string|RegExp} */ (split)); -// -// return this.filter ? filter(words, this.filter) : words; -// } -// } -// } -// -// return str; -// }; - -// export function pipeline(str, normalize, matcher, stemmer, split, _filter, _collapse){ -// -// if(str){ -// -// if(normalize && str){ -// -// str = replace(str, normalize); -// } -// -// if(matcher && str){ -// -// str = replace(str, matcher); -// } -// -// if(stemmer && str.length > 1){ -// -// str = replace(str, stemmer); -// } -// -// if(_collapse && str.length > 1){ -// -// str = collapse(str); -// } -// -// if(str){ -// -// if(split !== false){ -// -// str = str.split(split); -// -// if(_filter){ -// -// str = filter(str, _filter); -// } -// } -// } -// } -// -// return str; -// } - - -/** - * @param {Array} words - * @returns {Object} - */ - -export function init_filter(words){ - - const filter = create_object(); - - for(let i = 0, length = words.length; i < length; i++){ - - filter[words[i]] = 1; - } - - return filter; -} - -/** - * @param {!Object} obj - * @param {boolean} is_stemmer - * @returns {Array} - */ - -export function init_stemmer_or_matcher(obj, is_stemmer){ - - const keys = get_keys(obj); - const length = keys.length; - const final = []; - - let removal = "", count = 0; - - for(let i = 0, key, tmp; i < length; i++){ - - key = keys[i]; - tmp = obj[key]; - - if(tmp){ - - final[count++] = regex(is_stemmer ? "(?!\\b)" + key + "(\\b|_)" : key); - final[count++] = tmp; - } - else{ - - removal += (removal ? "|" : "") + key; - } - } - - if(removal){ - - final[count++] = regex(is_stemmer ? "(?!\\b)(" + removal + ")(\\b|_)" : "(" + removal + ")"); - final[count] = ""; - } - - return final; -} - - -/** - * @param {!string} str - * @param {Array} regexp - * @returns {string} - */ - -export function replace(str, regexp){ - - for(let i = 0, len = regexp.length; i < len; i += 2){ - - str = str.replace(regexp[i], regexp[i + 1]); - - if(!str){ - - break; - } - } - - return str; -} - -/** - * @param {!string} str - * @returns {RegExp} - */ - -export function regex(str){ - - return new RegExp(str, "g"); -} - -/** - * Regex: replace(/(?:(\w)(?:\1)*)/g, "$1") - * @param {!string} string - * @returns {string} - */ - -export function collapse(string){ - - let final = "", prev = ""; - - for(let i = 0, len = string.length, char; i < len; i++){ - - if((char = string[i]) !== prev){ - - final += (prev = char); - } - } - - return final; -} - -// TODO using fast-swap -export function filter(words, map){ - - const length = words.length; - const filtered = []; - - for(let i = 0, count = 0; i < length; i++){ - - const word = words[i]; - - if(word && !map[word]){ - - filtered[count++] = word; - } - } - - return filtered; -} - -// const chars = {a:1, e:1, i:1, o:1, u:1, y:1}; -// -// function collapse_repeating_chars(string){ -// -// let collapsed_string = "", -// char_prev = "", -// char_next = ""; -// -// for(let i = 0; i < string.length; i++){ -// -// const char = string[i]; -// -// if(char !== char_prev){ -// -// if(i && (char === "h")){ -// -// if((chars[char_prev] && chars[char_next]) || (char_prev === " ")){ -// -// collapsed_string += char; -// } -// } -// else{ -// -// collapsed_string += char; -// } -// } -// -// char_next = ( -// -// (i === (string.length - 1)) ? -// -// "" -// : -// string[i + 1] -// ); -// -// char_prev = char; -// } -// -// return collapsed_string; -// } diff --git a/src/lang/arabic/default.js b/src/lang/arabic/default.js index 8b63103..4dee96d 100644 --- a/src/lang/arabic/default.js +++ b/src/lang/arabic/default.js @@ -1,4 +1,3 @@ -import { IndexInterface } from "../../type.js"; import { pipeline } from "../../lang.js"; export const rtl = true; @@ -13,7 +12,6 @@ const split = /\s+/; /** * @param {string|number} str - * @this IndexInterface */ export function encode(str){ diff --git a/src/lang/at.js b/src/lang/at.js deleted file mode 100644 index 0fecf05..0000000 --- a/src/lang/at.js +++ /dev/null @@ -1,172 +0,0 @@ -/** - * http://www.ranks.nl/stopwords - * @type {Array} - */ - -export const filter = [ - - "aber", - "als", - "am", - "an", - "auch", - "auf", - "aus", - "bei", - "bin", - "bis", - "bist", - "da", - "dadurch", - "daher", - "darum", - "das", - "daß", - "dass", - "dein", - "deine", - "dem", - "den", - "der", - "des", - "dessen", - "deshalb", - "die", - "dies", - "dieser", - "dieses", - "doch", - "dort", - "du", - "durch", - "ein", - "eine", - "einem", - "einen", - "einer", - "eines", - "er", - "es", - "euer", - "eure", - "für", - "hatte", - "hatten", - "hattest", - "hattet", - "hier", - "hinter", - "ich", - "ihr", - "ihre", - "im", - "in", - "ist", - "ja", - "jede", - "jedem", - "jeden", - "jeder", - "jedes", - "jener", - "jenes", - "jetzt", - "kann", - "kannst", - "können", - "könnt", - "machen", - "mein", - "meine", - "mit", - "muß", - "mußt", - "musst", - "müssen", - "müßt", - "nach", - "nachdem", - "nein", - "nicht", - "nun", - "oder", - "seid", - "sein", - "seine", - "sich", - "sie", - "sind", - "soll", - "sollen", - "sollst", - "sollt", - "sonst", - "soweit", - "sowie", - "und", - "unser", - "unsere", - "unter", - "vom", - "von", - "vor", - "wann", - "warum", - "was", - "weiter", - "weitere", - "wenn", - "wer", - "werde", - "werden", - "werdet", - "weshalb", - "wie", - "wieder", - "wieso", - "wir", - "wird", - "wirst", - "wo", - "woher", - "wohin", - "zu", - "zum", - "zur", - "über" -]; - -/** - * @type {Object} - */ - -export const stemmer = { - - "niss": "", - "isch": "", - "lich": "", - "heit": "", - "keit": "", - "end": "", - "ung": "", - "est": "", - "ern": "", - "em": "", - "er": "", - "en": "", - "es": "", - "st": "", - "ig": "", - "ik": "", - "e": "", - "s": "" -}; - -export const matcher = {}; - -export default { - - filter: filter, - stemmer: stemmer, - matcher: matcher -} diff --git a/src/lang/cjk/default.js b/src/lang/cjk/default.js index a31f0b5..4b1a1ea 100644 --- a/src/lang/cjk/default.js +++ b/src/lang/cjk/default.js @@ -1,4 +1,3 @@ -import { IndexInterface } from "../../type.js"; import { pipeline } from "../../lang.js"; export const rtl = false; @@ -13,7 +12,6 @@ const regex = /[\x00-\x7F]+/g; /** * @param {string|number} str - * @this IndexInterface */ export function encode(str){ diff --git a/src/lang/cyrillic/default.js b/src/lang/cyrillic/default.js index 5bba75d..e331a4e 100644 --- a/src/lang/cyrillic/default.js +++ b/src/lang/cyrillic/default.js @@ -1,4 +1,3 @@ -import { IndexInterface } from "../../type.js"; import { pipeline } from "../../lang.js"; export const rtl = false; @@ -13,7 +12,6 @@ const split = /\s+/; /** * @param {string|number} str - * @this IndexInterface */ export function encode(str){ diff --git a/src/lang/de.js b/src/lang/de.js index 029a6bd..86c7152 100644 --- a/src/lang/de.js +++ b/src/lang/de.js @@ -2,11 +2,10 @@ * Filter are also known as "stopwords", they completely filter out words from being indexed. * Source: http://www.ranks.nl/stopwords * Object Definition: Just provide an array of words. - * @type {Array} + * @type {Set} */ -export const filter = [ - +export const filter = new Set([ "aber", "als", "am", @@ -23,7 +22,7 @@ export const filter = [ "daher", "darum", "das", - "daß", + "dass", "dass", "dein", "deine", @@ -51,7 +50,7 @@ export const filter = [ "es", "euer", "eure", - "für", + "fuer", "hatte", "hatten", "hattest", @@ -73,23 +72,25 @@ export const filter = [ "jener", "jenes", "jetzt", + "ggf", "kann", "kannst", - "können", - "könnt", + "koennen", + "koennt", "machen", "mein", "meine", "mit", - "muß", - "mußt", + "muss", "musst", - "müssen", - "müßt", + "musst", + "muessen", + "muesst", "nach", "nachdem", "nein", "nicht", + "noch", "nun", "oder", "seid", @@ -109,6 +110,8 @@ export const filter = [ "unser", "unsere", "unter", + "usw", + "uvm", "vom", "von", "vor", @@ -135,51 +138,68 @@ export const filter = [ "zu", "zum", "zur", - "über" -]; + "ueber" +]); /** * Stemmer removes word endings and is a kind of "partial normalization". A word ending just matched when the word length is bigger than the matched partial. * Example: The word "correct" and "correctness" could be the same word, so you can define {"ness": ""} to normalize the ending. * Object Definition: the key represents the word ending, the value contains the replacement (or empty string for removal). - * @type {Object} + * http://snowball.tartarus.org/algorithms/german/stemmer.html + * @type {Map} */ -export const stemmer = { - - "niss": "", - "isch": "", - "lich": "", - "heit": "", - "keit": "", - "ell": "", - "bar": "", - "end": "", - "ung": "", - "est": "", - "ern": "", - "em": "", - "er": "", - "en": "", - "es": "", - "st": "", - "ig": "", - "ik": "", - "e": "", - "s": "" -}; +export const stemmer = new Map([ + ["niss", ""], + ["isch", ""], + ["lich", ""], + ["heit", ""], + ["keit", ""], + ["ell", ""], + ["bar", ""], + ["end", ""], + ["ung", ""], + ["est", ""], + ["ern", ""], + ["em", ""], + ["er", ""], + ["en", ""], + ["es", ""], + ["st", ""], + ["ig", ""], + ["ik", ""], + ["e", ""], + ["s", ""] +]); /** * Matcher replaces all occurrences of a given string regardless of its position and is also a kind of "partial normalization". * Object Definition: the key represents the target term, the value contains the search string which should be replaced (could also be an array of multiple terms). - * @type {Object|string>} + * @type {Map} */ - -export const matcher = {}; +const map = new Map([ + ["_", " "], + ["ä", "ae"], + ["ö", "oe"], + ["ü", "ue"], + ["ß", "ss"], + ["&", " und "], + ["€", " EUR "] +]); export default { - + normalize: function(str){ + return str.toLowerCase(); + }, + prepare: function(str){ + // normalization + if(/[_äöüß&€]/.test(str)) + str = str.replace(/[_äöüß&€]/g, match => map.get(match)); + // street names + return str.replace(/str\b/g, "strasse") + .replace(/(?!\b)strasse\b/g, " strasse") + .replace(/\bst\b/g, "sankt"); + }, filter: filter, - stemmer: stemmer, - matcher: matcher -} \ No newline at end of file + stemmer: stemmer +}; \ No newline at end of file diff --git a/src/lang/en.js b/src/lang/en.js index c7e9870..32c0257 100644 --- a/src/lang/en.js +++ b/src/lang/en.js @@ -1,10 +1,11 @@ /** * http://www.ranks.nl/stopwords - * @type {Array} + * @type {Set} */ -export const filter = [ +// todo filter out minlength +export const filter = new Set([ "a", "about", "above", @@ -18,32 +19,32 @@ export const filter = [ "and", "any", "are", - "aren't", + "arent", "as", "at", - //"back", + "back", "be", "because", "been", "before", "being", "below", - //"between", + "between", "both", "but", "by", "can", "cannot", - "can't", + "cant", "come", "could", - "couldn't", + "couldnt", //"day", "did", - "didn't", + "didnt", "do", "does", - "doesn't", + "doesnt", "doing", "dont", "down", @@ -51,27 +52,27 @@ export const filter = [ "each", "even", "few", - "first", + //"first", "for", "from", "further", "get", //"give", "go", - //"good", + "good", "had", - "hadn't", + "hadnt", "has", - "hasn't", + "hasnt", "have", - "haven't", + "havent", "having", "he", "hed", //"hell", "her", "here", - "here's", + "heres", "hers", "herself", "hes", @@ -79,7 +80,7 @@ export const filter = [ "himself", "his", "how", - "how's", + "hows", "i", "id", "if", @@ -88,21 +89,23 @@ export const filter = [ "in", "into", "is", - "isn't", + "isnt", "it", - "it's", + "its", "itself", - "i've", + "ive", "just", "know", - "let's", + "lets", "like", //"look", + "lot", "make", + "made", "me", "more", "most", - "mustn't", + "mustnt", "my", "myself", "new", @@ -114,35 +117,35 @@ export const filter = [ "off", "on", "once", - //"one", + "one", "only", "or", "other", "ought", "our", - "our's", + "ours", "ourselves", "out", "over", "own", - //"people", + "people", "same", "say", "see", - "shan't", + "shant", "she", - "she'd", + "shed", "shell", "shes", "should", - "shouldn't", + "shouldnt", "so", "some", "such", - //"take", + "take", "than", "that", - "that's", + "thats", "the", "their", "theirs", @@ -150,127 +153,232 @@ export const filter = [ "themselves", "then", "there", - "there's", + "theres", "these", "they", - "they'd", - "they'll", - "they're", - "they've", - //"think", + "theyd", + "theyll", + "theyre", + "theyve", + "think", "this", "those", "through", "time", + "times", "to", "too", //"two", - //"under", + "under", "until", "up", "us", - //"use", + "use", "very", "want", "was", - "wasn't", + "wasnt", "way", "we", "wed", "well", "were", - "weren't", - "we've", + "werent", + "weve", "what", - "what's", + "whats", "when", - "when's", + "whens", "where", - "where's", + "wheres", "which", "while", "who", "whom", - "who's", + "whos", "why", - "why's", + "whys", "will", "with", - "won't", - //"work", + "wont", + "work", "would", - "wouldn't", + "wouldnt", //"year", + "ya", "you", - "you'd", - "you'll", + "youd", + "youll", "your", - "you're", - "your's", + "youre", + "yours", "yourself", "yourselves", - "you've" -]; + "youve" +]); /** * @type {Object} */ -export const stemmer = { +export const stemmer = new Map([ + ["ational", ""], + ["iveness", ""], + ["fulness", ""], + ["ousness", ""], + ["ization", ""], + ["tional", ""], + ["biliti", ""], + ["icate", ""], + ["ative", ""], + ["alize", ""], + ["iciti", ""], + ["entli", ""], + ["ousli", ""], + ["alism", ""], + ["ation", ""], + ["aliti", ""], + ["iviti", ""], + ["ement", ""], + ["izer", ""], + ["able", ""], + ["alli", ""], + ["ator", ""], + ["logi", ""], + ["ical", ""], + ["ance", ""], + ["ence", ""], + ["ness", ""], + ["ble", ""], + ["ment", ""], + ["eli", ""], + ["bli", ""], + ["ful", ""], + ["ant", ""], + ["ent", ""], + ["ism", ""], + ["ate", ""], + ["iti", ""], + ["ous", ""], + ["ive", ""], + ["ize", ""], + ["ing", ""], + ["ion", ""], + ["al", ""], + ["ou", ""], + ["er", ""], + ["ic", ""], + ["ly", ""] +]); - "ational": "ate", - "iveness": "ive", - "fulness": "ful", - "ousness": "ous", - "ization": "ize", - "tional": "tion", - "biliti": "ble", - "icate": "ic", - "ative": "", - "alize": "al", - "iciti": "ic", - "entli": "ent", - "ousli": "ous", - "alism": "al", - "ation": "ate", - "aliti": "al", - "iviti": "ive", - "ement": "", - "enci": "ence", - "anci": "ance", - "izer": "ize", - "alli": "al", - "ator": "ate", - "logi": "log", - "ical": "ic", - "ance": "", - "ence": "", - "ness": "", - "able": "", - "ible": "", - "ment": "", - "eli": "e", - "bli": "ble", - "ful": "", - "ant": "", - "ent": "", - "ism": "", - "ate": "", - "iti": "", - "ous": "", - "ive": "", - "ize": "", - "al": "", - "ou": "", - "er": "", - "ic": "" -}; +// export const replacer = new Map([ +// ["&", " and "], +// ]); -export const matcher = {}; +/* + he’s (= he is / he has) + she’s (= she is / she has) + I’ll (= I will) + I’ve (= I have) + I’d (= I would / I had) + don’t (= do not) + doesn’t (= does not) + didn’t (= did not) + isn’t (= is not) + hasn’t (= has not) + can’t (= cannot) + won’t (= will not) +*/ + +// const explode = new Map([ +// ["^i'm$", "i am"], +// ["^can't$", "can not"], +// ["^cannot$", "can not"], +// ["^won't$", "will not"], +// ["'s$", " is has"], +// ["n't$", " not"], +// ["'ll$", " will"], +// ["'re$", " are"], +// ["'ve$", " have"], +// ["'d$", " would had"], +// ]); + +// const pairs = [ +// /´`’ʼ/, /´`’ʼ/g, "'", +// /_/, /_+/g, " ", +// /&/, /&/g, " and ", +// /\bi'm\b/, /\bi'm\b/g, "i am", +// /\b(can't|cannot)\b/, /\b(can't|cannot)\b/g, "can not", +// /\bwon't\b/, /\bwon't\b/g, "will not", +// /[a-z]n't\b/, /[a-z]n't\b/g, "$1 not", +// /[a-z]'s\b/, /([a-z])'s\b/g, "$1 is has", +// /[a-z]'ll\b/, /[a-z]'ll\b/g, "$1 will", +// /[a-z]'re\b/, /[a-z]'re\b/g, "$1 are", +// /[a-z]'ve\b/, /[a-z]'ve\b/g, "$1 have", +// /[a-z]'d\b/, /[a-z]'d\b/g, "$1 is has" +// ]; + +// const map = new Map([ +// ["´", "'"], +// ["`", "'"], +// ["’", "'"], +// ["ʼ", "'"], +// ["_", " "], +// ["&", " and "] +// ]); export default { + prepare: function(str){ + // if(/[´`’ʼ_&]/.test(str)) + // str = str.replace(/[´`’ʼ_&]/g, match => map.get(match)); + // if(/´`’ʼ/.test(str)) + // str = str.replace(/´`’ʼ/g, "'"); + // if(/_/.test(str)) + // str = str.replace(/_+/g, " "); + // if(/&/.test(str)) + // str = str.replace(/&/g, " and "); + + // if(/\bi'm\b/.test(str)) + // str = str.replace(/\bi'm\b/g, "i am"); + // if(/\b(can't|cannot)\b/.test(str)) + // str = str.replace(/\b(can't|cannot)\b/g, "can not"); + // if(/\bwon't\b/.test(str)) + // str = str.replace(/\bwon't\b/g, "will not"); + // if(/[a-z]n't\b/.test(str)) + // str = str.replace(/([a-z])n't\b/g, "$1 not"); + // if(/[a-z]'s\b/.test(str)) + // str = str.replace(/([a-z])'s\b/g, "$1 is has"); + // if(/[a-z]'ll\b/.test(str)) + // str = str.replace(/([a-z])'ll\b/g, "$1 will"); + // if(/[a-z]'re\b/.test(str)) + // str = str.replace(/([a-z])'re\b/g, "$1 are"); + // if(/[a-z]'ve\b/.test(str)) + // str = str.replace(/([a-z])'ve\b/g, "$1 have"); + // if(/[a-z]'d\b/.test(str)) + // str = str.replace(/([a-z])'d\b/g, "$1 would had"); + // return str; + + return str//.replace(/[´`’ʼ_&]/g, match => map.get(match)) + // normalization + .replace(/´`’ʼ/g, "'") + .replace(/_+/g, " ") + .replace(/&/g, " and ") + //.replace(/([0-9 ]|^)\$([0-9 ]|$)/g, "$1 USD $2") + //.replace(/([0-9 ]|^)£([0-9 ]|$)/g, "$1 GBP $2") + .replace(/\$/g, " USD ") + .replace(/£/g, " GBP ") + // explode short forms + .replace(/([a-z])'s\b/g, "$1 is") + .replace(/\bi'm\b/g, "i am") + .replace(/\b(can't|cannot)\b/g, "can not") + .replace(/\bwon't\b/g, "will not") + .replace(/([a-z])n't\b/g, "$1 not") + .replace(/([a-z])'ll\b/g, "$1 will") + .replace(/([a-z])'re\b/g, "$1 are") + .replace(/([a-z])'ve\b/g, "$1 have") + .replace(/([a-z])'d\b/g, "$1 would"); + }, filter: filter, - stemmer: stemmer, - matcher: matcher -} + stemmer: stemmer +}; diff --git a/src/lang/fr.js b/src/lang/fr.js new file mode 100644 index 0000000..eef6e37 --- /dev/null +++ b/src/lang/fr.js @@ -0,0 +1,226 @@ +/** + * http://www.ranks.nl/stopwords + * http://snowball.tartarus.org/algorithms/french/stop.txt + * @type {Set} + */ + +export const filter = new Set([ + "au", + "aux", + "avec", + "ce", + "ces", + "dans", + "de", + "des", + "du", + "elle", + "en", + "et", + "eux", + "il", + "je", + "la", + "le", + "leur", + "lui", + "ma", + "mais", + "me", + "meme", + "mes", + "moi", + "mon", + "ne", + "nos", + "notre", + "nous", + "on", + "ou", + "par", + "pas", + "pour", + "qu", + "que", + "qui", + "sa", + "se", + "ses", + "son", + "sur", + "ta", + "te", + "tes", + "toi", + "ton", + "tu", + "un", + "une", + "vos", + "votre", + "vous", + + "c", + "d", + "j", + "l", + "m", + "n", + "s", + "t", + "a", + "y", + + "ete", + "etee", + "etees", + "etes", + "etant", + "suis", + "es", + "est", + "sommes", + "etes", + "sont", + "serai", + "seras", + "sera", + "serons", + "serez", + "seront", + "serais", + "serait", + "serions", + "seriez", + "seraient", + "etais", + "etait", + "etions", + "etiez", + "etaient", + "fus", + "fut", + "fumes", + "futes", + "furent", + "sois", + "soit", + "soyons", + "soyez", + "soient", + "fusse", + "fusses", + "fut", + "fussions", + "fussiez", + "fussent", + + "ayant", + "eu", + "eue", + "eues", + "eus", + "ai", + "as", + "avons", + "avez", + "ont", + "aurai", + "auras", + "aura", + "aurons", + "aurez", + "auront", + "aurais", + "aurait", + "aurions", + "auriez", + "auraient", + "avais", + "avait", + "avions", + "aviez", + "avaient", + "eut", + "eumes", + "eutes", + "eurent", + "aie", + "aies", + "ait", + "ayons", + "ayez", + "aient", + "eusse", + "eusses", + "eut", + "eussions", + "eussiez", + "eussent", + + "ceci", + "cela", + "cela", + "cet", + "cette", + "ici", + "ils", + "les", + "leurs", + "quel", + "quels", + "quelle", + "quelles", + "sans", + "soi" +]); + +/** + * @type {Object} + */ + +export const stemmer = new Map([ + ["lement", ""], + ["ient", ""], + ["nera", ""], + ["ment", ""], + ["ais", ""], + ["ait", ""], + ["ant", ""], + ["ent", ""], + ["iez", ""], + ["ion", ""], + ["nez", ""], + ["ai", ""], + ["es", ""], + ["er", ""], + ["ez", ""], + ["le", ""], + ["na", ""], + ["ne", ""], + ["a", ""], + ["e", ""] +]); + +export default { + prepare: function(str){ + return str.replace(/´`’ʼ/g, "'") + .replace(/_+/g, " ") + .replace(/&/g, " et ") + .replace(/€/g, " EUR ") + .replace(/\bl'([^\b])/g, "la le $1") + .replace(/\bt'([^\b])/g, "ta te $1") + .replace(/\bc'([^\b])/g, "ca ce $1") + .replace(/\bd'([^\b])/g, "da de $1") + .replace(/\bj'([^\b])/g, "ja je $1") + .replace(/\bn'([^\b])/g, "na ne $1") + .replace(/\bm'([^\b])/g, "ma me $1") + .replace(/\bs'([^\b])/g, "sa se $1") + .replace(/\bau\b/g, "a le") + .replace(/\baux\b/g, "a les") + .replace(/\bdu\b/g, "de le") + .replace(/\bdes\b/g, "de les") + }, + filter: filter, + stemmer: stemmer +}; diff --git a/src/lang/latin/advanced.js b/src/lang/latin/advanced.js index baa67e4..3732d41 100644 --- a/src/lang/latin/advanced.js +++ b/src/lang/latin/advanced.js @@ -1,92 +1,156 @@ -import { IndexInterface } from "../../type.js"; -import { regex, replace, collapse } from "../../lang.js"; -import { encode as encode_balance } from "./balance.js"; +import Encoder from "../../encoder.js"; +import { soundex } from "./balance.js"; -export const rtl = false; -export const tokenize = ""; -export default { - encode: encode, - rtl: rtl, - tokenize: tokenize -} +// const soundex = new Map([ +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// +// //["e", "e"], +// ["i", "e"], +// ["y", "e"], +// +// //["o", "o"], +// ["u", "o"] +// ]); -// Phonetic Normalization +export const matcher = new Map([ + ["ai", "ei"], + ["ae", "a"], + ["oe", "o"], + ["ue", "u"], + ["sh", "s"], + ["ch", "c"], + ["th", "t"], + ["ph", "f"], + ["pf", "f"] +]); -const regex_ae = regex("ae"), - //regex_ai = regex("ai"), - //regex_ay = regex("ay"), - //regex_ey = regex("ey"), - regex_oe = regex("oe"), - //regex_ue = regex("ue"), - //regex_ie = regex("ie"), - //regex_sz = regex("sz"), - //regex_zs = regex("zs"), - //regex_ck = regex("ck"), - //regex_cc = regex("cc"), - regex_sh = regex("sh"), - regex_th = regex("th"), - //regex_dt = regex("dt"), - regex_ph = regex("ph"), - regex_pf = regex("pf"); - //regex_ou = regex("ou"), - //regex_uo = regex("uo"); - -const pairs = [ - regex_ae, "a", - // regex_ai, "ei", - // regex_ay, "ei", - // regex_ey, "ei", - regex_oe, "o", - // regex_ue, "u", - // regex_ie, "i", - // regex_sz, "s", - // regex_zs, "s", - regex_sh, "s", - // regex_ck, "k", - // regex_cc, "k", - regex_th, "t", - // regex_dt, "t", - regex_ph, "f", - regex_pf, "f", - // regex_ou, "o", - // regex_uo, "u" - - // regex("(?![aeiouy])h(?![aeiouy])"), "", - // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "" - regex("(?![aeo])h(?![aeo])"), "", - regex("(?!^[aeo])h(?!^[aeo])"), "" +export const replacer = [ + /([^aeo])h([aeo$])/g, "$1$2", + /([aeo])h([^aeo]|$)/g, "$1$2", ]; -/** - * @param {string|number} str - * @param {boolean=} _skip_postprocessing - * @this IndexInterface - */ +export default { + normalize: true, + dedupe: true, + mapper: soundex, + replacer: replacer, + matcher: matcher +}; -export function encode(str, _skip_postprocessing){ - - if(str){ - - str = encode_balance.call(this, str).join(" "); - - if(str.length > 2){ - - str = replace(str, pairs); - } - - if(!_skip_postprocessing){ - - if(str.length > 1){ - - str = collapse(str); - } - - if(str){ - - str = str.split(" "); - } - } - } - - return str || []; -} +// import { regex, replace, collapse } from "../../lang.js"; +// import { encode as encode_balance } from "./balance.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // Phonetic Normalization +// +// const regex_ae = regex("ae"), +// //regex_ai = regex("ai"), +// //regex_ay = regex("ay"), +// //regex_ey = regex("ey"), +// regex_oe = regex("oe"), +// //regex_ue = regex("ue"), +// //regex_ie = regex("ie"), +// //regex_sz = regex("sz"), +// //regex_zs = regex("zs"), +// //regex_ck = regex("ck"), +// //regex_cc = regex("cc"), +// regex_sh = regex("sh"), +// regex_th = regex("th"), +// //regex_dt = regex("dt"), +// regex_ph = regex("ph"), +// regex_pf = regex("pf"); +// //regex_ou = regex("ou"), +// //regex_uo = regex("uo"); +// +// const pairs = [ +// regex_ae, "a", +// // regex_ai, "ei", +// // regex_ay, "ei", +// // regex_ey, "ei", +// regex_oe, "o", +// // regex_ue, "u", +// // regex_ie, "i", +// // regex_sz, "s", +// // regex_zs, "s", +// regex_sh, "s", +// // regex_ck, "k", +// // regex_cc, "k", +// regex_th, "t", +// // regex_dt, "t", +// regex_ph, "f", +// regex_pf, "f", +// // regex_ou, "o", +// // regex_uo, "u" +// +// // regex("(?![aeiouy])h(?![aeiouy])"), "", +// // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "" +// regex("(?![aeo])h(?![aeo])"), "", +// regex("(?!^[aeo])h(?!^[aeo])"), "" +// ]; +// +// /** +// * @param {string|number} str +// * @param {boolean=} _skip_postprocessing +// */ +// +// export function encode(str, _skip_postprocessing){ +// +// if(str){ +// +// str = encode_balance.call(this, str).join(" "); +// +// if(str.length > 2){ +// +// str = replace(str, pairs); +// } +// +// if(!_skip_postprocessing){ +// +// if(str.length > 1){ +// +// str = collapse(str); +// } +// +// if(str){ +// +// str = str.split(" "); +// } +// } +// } +// +// return str || []; +// } diff --git a/src/lang/latin/balance.js b/src/lang/latin/balance.js index bc20788..c129226 100644 --- a/src/lang/latin/balance.js +++ b/src/lang/latin/balance.js @@ -1,119 +1,288 @@ -import { IndexInterface } from "../../type.js"; -import { encode as encode_simple } from "./simple.js"; +import Encoder from "../../encoder.js"; -// custom soundex implementation +export const soundex = new Map([ + ["b", "p"], + //["p", "p"], + + //["f", "f"], + ["v", "f"], + ["w", "f"], + + //["s", "s"], + ["z", "s"], + ["x", "s"], + + ["d", "t"], + //["t", "t"], + + //["m", "m"], + ["n", "m"], + + //["k", "k"], + ["c", "k"], + ["g", "k"], + ["j", "k"], + ["q", "k"], + + //["r", "r"], + //["h", "h"], + //["l", "l"], + + //["a", "a"], + + //["e", "e"], + ["i", "e"], + ["y", "e"], + + //["o", "o"], + ["u", "o"] +]); -export const rtl = false; -export const tokenize = "strict"; export default { - encode: encode, - rtl: rtl, - tokenize: tokenize -} - -//const regex_whitespace = /[\W_]+/g; -const regex_strip = /[^a-z0-9]+/; - -// const pairs = [ -// regex_whitespace, " ", -// regex_strip, "" -// ]; - -// modified - -const soundex = { - - "b": "p", - //"p": "p", - - //"f": "f", - "v": "f", - "w": "f", - - //"s": "s", - "z": "s", - "x": "s", - "ß": "s", - - "d": "t", - //"t": "t", - - //"l": "l", - - //"m": "m", - "n": "m", - - "c": "k", - "g": "k", - "j": "k", - //"k": "k", - "q": "k", - - //"r": "r", - //"h": "h", - //"a": "a", - - //"e": "e", - "i": "e", - "y": "e", - - //"o": "o", - "u": "o" + normalize: true, + dedupe: true, + mapper: soundex }; -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str){ - - str = encode_simple.call(this, str).join(" "); - - // str = this.pipeline( - // - // /* string: */ normalize("" + str).toLowerCase(), - // /* normalize: */ false, - // /* split: */ false, - // /* collapse: */ false - // ); - - const result = []; - - if(str){ - - const words = str.split(regex_strip); - const length = words.length; - - for(let x = 0, tmp, count = 0; x < length; x++){ - - if((str = words[x]) /*&& (str.length > 2)*/ && (!this.filter || !this.filter[str])){ - - tmp = str[0]; - let code = soundex[tmp] || tmp; //str[0]; - let previous = code; //soundex[code] || code; - - for(let i = 1; i < str.length; i++){ - - tmp = str[i]; - const current = soundex[tmp] || tmp; - - if(current && (current !== previous)){ - - code += current; - previous = current; - - // if(code.length === 7){ - // - // break; - // } - } - } - - result[count++] = code; //(code + "0000").substring(0, 4); - } - } - } - - return result; -} +// //import { encode as encode_simple } from "./simple.js"; +// import { pipeline } from "../../lang.js"; +// +// // custom soundex implementation +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// //const regex_whitespace = /[\W_]+/g; +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// const normalize = "".normalize && /[\u0300-\u036f]/g; +// //const regex_strip = /[^a-z0-9]+/; +// +// // const pairs = [ +// // regex_whitespace, " ", +// // regex_strip, "" +// // ]; +// +// // modified +// const pairs = new Map([ +// +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["ß", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// ["ñ", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["ç", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// ["à", "a"], +// ["á", "a"], +// ["â", "a"], +// ["ã", "a"], +// ["ä", "a"], +// ["å", "a"], +// +// //["e", "e"], +// ["è", "e"], +// ["é", "e"], +// ["ê", "e"], +// ["ë", "e"], +// ["i", "e"], +// ["ì", "e"], +// ["í", "e"], +// ["î", "e"], +// ["ï", "e"], +// ["y", "e"], +// ["ý", "e"], +// ["ŷ", "e"], +// ["ÿ", "e"], +// +// //["o", "o"], +// ["ò", "o"], +// ["ó", "o"], +// ["ô", "o"], +// ["õ", "o"], +// ["ö", "o"], +// ["ő", "o"], +// ["u", "o"], +// ["ù", "o"], +// ["ú", "o"], +// ["û", "o"], +// ["ü", "o"], +// ["ű", "o"], +// ]); +// +// const map_soundex = new Map([ +// ["b", "p"], +// //["p", "p"], +// +// //["f", "f"], +// ["v", "f"], +// ["w", "f"], +// +// //["s", "s"], +// ["z", "s"], +// ["x", "s"], +// +// ["d", "t"], +// //["t", "t"], +// +// //["m", "m"], +// ["n", "m"], +// +// //["k", "k"], +// ["c", "k"], +// ["g", "k"], +// ["j", "k"], +// ["q", "k"], +// +// //["r", "r"], +// //["h", "h"], +// //["l", "l"], +// +// //["a", "a"], +// +// //["e", "e"], +// ["i", "e"], +// ["y", "e"], +// +// //["o", "o"], +// ["u", "o"] +// ]); +// // const soundex = { +// // +// // "b": "p", +// // //"p": "p", +// // +// // //"f": "f", +// // "v": "f", +// // "w": "f", +// // +// // //"s": "s", +// // "z": "s", +// // "x": "s", +// // "ß": "s", +// // +// // "d": "t", +// // //"t": "t", +// // +// // //"l": "l", +// // +// // //"m": "m", +// // "n": "m", +// // +// // "c": "k", +// // "g": "k", +// // "j": "k", +// // //"k": "k", +// // "q": "k", +// // +// // //"r": "r", +// // //"h": "h", +// // //"a": "a", +// // +// // //"e": "e", +// // "i": "e", +// // "y": "e", +// // +// // //"o": "o", +// // "u": "o" +// // }; +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// return pipeline.call( +// this, +// /* string: */ ("" + str).normalize("NFD").replace(normalize, "").toLowerCase(), +// /* normalize: */ map_soundex, +// /* split: */ whitespace, +// ///* collapse: */ false +// ); +// +// // return pipeline.call( +// // this, +// // /* string: */ ("" + str).toLowerCase(), +// // /* normalize: */ /*pairs*/ new Map(), +// // /* split: */ whitespace, +// // ///* collapse: */ false +// // ); +// +// // str = encode_simple.call(this, str).join(" "); +// // +// // // str = this.pipeline( +// // // +// // // /* string: */ normalize("" + str).toLowerCase(), +// // // /* normalize: */ false, +// // // /* split: */ false, +// // // /* collapse: */ false +// // // ); +// // +// // const result = []; +// // +// // if(str){ +// // +// // const words = str.split(regex_strip); +// // const length = words.length; +// // +// // for(let x = 0, tmp, count = 0; x < length; x++){ +// // +// // if((str = words[x]) /*&& (str.length > 2)*/ && (!this.filter || !this.filter.has(str))){ +// // +// // tmp = str[0]; +// // let code = soundex[tmp] || tmp; //str[0]; +// // let previous = code; //soundex[code] || code; +// // +// // for(let i = 1; i < str.length; i++){ +// // +// // tmp = str[i]; +// // const current = soundex[tmp] || tmp; +// // +// // if(current && (current !== previous)){ +// // +// // code += current; +// // previous = current; +// // +// // // if(code.length === 7){ +// // // +// // // break; +// // // } +// // } +// // } +// // +// // result[count++] = code; //(code + "0000").substring(0, 4); +// // } +// // } +// // } +// // +// // return result; +// } diff --git a/src/lang/latin/default.js b/src/lang/latin/default.js index 47870f3..76eef6b 100644 --- a/src/lang/latin/default.js +++ b/src/lang/latin/default.js @@ -1,27 +1,36 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline, normalize, regex_whitespace } from "../../lang.js"; +import Encoder from "../../encoder.js"; -export const rtl = false; -export const tokenize = ""; export default { - encode: encode, - rtl: rtl, - tokenize: tokenize -} + normalize: function(str){ + return str.toLowerCase(); + }, + dedupe: false +}; -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str){ - - return pipeline.call( - - this, - /* string: */ ("" + str).toLowerCase(), - /* normalize: */ false, - /* split: */ regex_whitespace, - /* collapse: */ false - ); -} +// import { pipeline } from "../../lang.js"; +// +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// return pipeline.call( +// +// this, +// /* string: */ ("" + str).toLowerCase(), +// /* normalize: */ false, +// /* split: */ whitespace, +// /* collapse: */ false +// ); +// } diff --git a/src/lang/latin/exact.js b/src/lang/latin/exact.js new file mode 100644 index 0000000..c76ee55 --- /dev/null +++ b/src/lang/latin/exact.js @@ -0,0 +1,4 @@ +export default { + normalize: false, + dedupe: false +}; diff --git a/src/lang/latin/extra.js b/src/lang/latin/extra.js index cc5dafc..a351059 100644 --- a/src/lang/latin/extra.js +++ b/src/lang/latin/extra.js @@ -1,67 +1,82 @@ -import { IndexInterface } from "../../type.js"; -import { regex, replace, collapse } from "../../lang.js"; -import { encode as encode_advanced } from "./advanced.js"; +import Encoder from "../../encoder.js"; +import { soundex } from "./balance.js"; +import { matcher, replacer } from "./advanced.js"; -export const rtl = false; -export const tokenize = ""; -export default { - encode: encode, - rtl: rtl, - tokenize: tokenize -} - -// Soundex Normalization - -const prefix = "(?!\\b)"; -const //soundex_b = regex(prefix + "p"), - // soundex_s = regex(prefix + "z"), - // soundex_k = regex(prefix + "[cgq]"), - // soundex_m = regex(prefix + "n"), - // soundex_t = regex(prefix + "d"), - // soundex_f = regex(prefix + "[vw]"), - //regex_vowel = regex(prefix + "[aeiouy]"); - regex_vowel = regex(prefix + "[aeo]"); - -const pairs = [ - - // soundex_b, "b", - // soundex_s, "s", - // soundex_k, "k", - // soundex_m, "m", - // soundex_t, "t", - // soundex_f, "f", - // regex("(?![aeiouy])h(?![aeiouy])"), "", - // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "", - regex_vowel, "" +export const compact = [ + /(?!^)[aeoy]/g, "" // old: aioy ]; -/** - * @param {string|number} str - * @this IndexInterface - */ +export default { + normalize: true, + dedupe: true, + mapper: soundex, + replacer: replacer.concat(compact), + matcher: matcher +}; -export function encode(str){ - - if(str){ - - str = encode_advanced.call(this, str, /* skip post-processing: */ true); - - if(str.length > 1){ - - //str = replace(str, pairs); - str = str.replace(regex_vowel, ""); - } - - if(str.length > 1){ - - str = collapse(str); - } - - if(str){ - - str = str.split(" "); - } - } - - return str || []; -} +// import { regex, replace, collapse } from "../../lang.js"; +// import { encode as encode_advanced } from "./advanced.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // Soundex Normalization +// +// const prefix = "(?!\\b)"; +// const //soundex_b = regex(prefix + "p"), +// // soundex_s = regex(prefix + "z"), +// // soundex_k = regex(prefix + "[cgq]"), +// // soundex_m = regex(prefix + "n"), +// // soundex_t = regex(prefix + "d"), +// // soundex_f = regex(prefix + "[vw]"), +// //regex_vowel = regex(prefix + "[aeiouy]"); +// regex_vowel = regex(prefix + "[aeo]"); +// +// const pairs = [ +// +// // soundex_b, "b", +// // soundex_s, "s", +// // soundex_k, "k", +// // soundex_m, "m", +// // soundex_t, "t", +// // soundex_f, "f", +// // regex("(?![aeiouy])h(?![aeiouy])"), "", +// // regex("(?!^[aeiouy])h(?!^[aeiouy])"), "", +// regex_vowel, "" +// ]; +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// if(str){ +// +// str = encode_advanced.call(this, str, /* skip post-processing: */ true); +// +// if(str.length > 1){ +// +// //str = replace(str, pairs); +// //str = str.replace(regex_vowel, ""); +// str = str.charAt(0) + str.substring(1).replace(regex_vowel, ""); +// } +// +// if(str.length > 1){ +// +// str = collapse(str); +// } +// +// if(str){ +// +// str = str.split(" "); +// } +// } +// +// return str || []; +// } diff --git a/src/lang/latin/simple.js b/src/lang/latin/simple.js index 8f0cefe..f2da1b3 100644 --- a/src/lang/latin/simple.js +++ b/src/lang/latin/simple.js @@ -1,60 +1,342 @@ -import { IndexInterface } from "../../type.js"; -import { pipeline, normalize, regex_whitespace, regex } from "../../lang.js"; +import Encoder from "../../encoder.js"; -export const rtl = false; -export const tokenize = ""; export default { - encode: encode, - rtl: rtl, - tokenize: tokenize -} + normalize: true, + dedupe: true +}; -// Charset Normalization - -const //regex_whitespace = /\W+/, - //regex_strip = regex("[^a-z0-9 ]"), - regex_a = regex("[àáâãäå]"), - regex_e = regex("[èéêë]"), - regex_i = regex("[ìíîï]"), - regex_o = regex("[òóôõöő]"), - regex_u = regex("[ùúûüű]"), - regex_y = regex("[ýŷÿ]"), - regex_n = regex("ñ"), - regex_c = regex("[çc]"), - regex_s = regex("ß"), - regex_and = regex(" & "); - -const pairs = [ - - regex_a, "a", - regex_e, "e", - regex_i, "i", - regex_o, "o", - regex_u, "u", - regex_y, "y", - regex_n, "n", - regex_c, "k", - regex_s, "s", - regex_and, " and " - //regex_whitespace, " " - //regex_strip, "" -]; - -/** - * @param {string|number} str - * @this IndexInterface - */ - -export function encode(str){ - - str = "" + str; - - return pipeline.call( - - this, - /* string: */ normalize(str).toLowerCase(), - /* normalize: */ !str.normalize && pairs, - /* split: */ regex_whitespace, - /* collapse: */ false - ); -} +// import { pipeline, regex } from "../../lang.js"; +// +// export const rtl = false; +// export const tokenize = ""; +// export default { +// encode: encode, +// rtl: rtl, +// tokenize: tokenize +// } +// +// // p{Z} = whitespaces +// // p{S} = symbols (emotes) +// // p{P} = special chars +// // p{C} = controls (linebreak, tabulator) +// const whitespace = /[\p{Z}\p{S}\p{P}\p{C}]+/u; +// const normalize = "".normalize && /[\u0300-\u036f]/g; +// const pairs = !normalize && new Map([ +// +// // Charset Normalization +// // String.normalize("NFKD").replace(/[\u0300-\u036f]/g, "") +// +// ["ª","a"], +// ["²","2"], +// ["³","3"], +// ["¹","1"], +// ["º","o"], +// ["¼","1⁄4"], +// ["½","1⁄2"], +// ["¾","3⁄4"], +// ["à","a"], +// ["á","a"], +// ["â","a"], +// ["ã","a"], +// ["ä","a"], +// ["å","a"], +// ["ç","c"], +// ["è","e"], +// ["é","e"], +// ["ê","e"], +// ["ë","e"], +// ["ì","i"], +// ["í","i"], +// ["î","i"], +// ["ï","i"], +// ["ñ","n"], +// ["ò","o"], +// ["ó","o"], +// ["ô","o"], +// ["õ","o"], +// ["ö","o"], +// ["ù","u"], +// ["ú","u"], +// ["û","u"], +// ["ü","u"], +// ["ý","y"], +// ["ÿ","y"], +// ["ā","a"], +// ["ă","a"], +// ["ą","a"], +// ["ć","c"], +// ["ĉ","c"], +// ["ċ","c"], +// ["č","c"], +// ["ď","d"], +// ["ē","e"], +// ["ĕ","e"], +// ["ė","e"], +// ["ę","e"], +// ["ě","e"], +// ["ĝ","g"], +// ["ğ","g"], +// ["ġ","g"], +// ["ģ","g"], +// ["ĥ","h"], +// ["ĩ","i"], +// ["ī","i"], +// ["ĭ","i"], +// ["į","i"], +// ["ij","ij"], +// ["ĵ","j"], +// ["ķ","k"], +// ["ĺ","l"], +// ["ļ","l"], +// ["ľ","l"], +// ["ŀ","l"], +// ["ń","n"], +// ["ņ","n"], +// ["ň","n"], +// ["ʼn","n"], +// ["ō","o"], +// ["ŏ","o"], +// ["ő","o"], +// ["ŕ","r"], +// ["ŗ","r"], +// ["ř","r"], +// ["ś","s"], +// ["ŝ","s"], +// ["ş","s"], +// ["š","s"], +// ["ţ","t"], +// ["ť","t"], +// ["ũ","u"], +// ["ū","u"], +// ["ŭ","u"], +// ["ů","u"], +// ["ű","u"], +// ["ų","u"], +// ["ŵ","w"], +// ["ŷ","y"], +// ["ź","z"], +// ["ż","z"], +// ["ž","z"], +// ["ſ","s"], +// ["ơ","o"], +// ["ư","u"], +// ["dž","dz"], +// ["lj","lj"], +// ["nj","nj"], +// ["ǎ","a"], +// ["ǐ","i"], +// ["ǒ","o"], +// ["ǔ","u"], +// ["ǖ","u"], +// ["ǘ","u"], +// ["ǚ","u"], +// ["ǜ","u"], +// ["ǟ","a"], +// ["ǡ","a"], +// ["ǣ","ae"], +// ["æ","ae"], +// ["ǽ","ae"], +// ["ǧ","g"], +// ["ǩ","k"], +// ["ǫ","o"], +// ["ǭ","o"], +// ["ǯ","ʒ"], +// ["ǰ","j"], +// ["dz","dz"], +// ["ǵ","g"], +// ["ǹ","n"], +// ["ǻ","a"], +// ["ǿ","ø"], +// ["ȁ","a"], +// ["ȃ","a"], +// ["ȅ","e"], +// ["ȇ","e"], +// ["ȉ","i"], +// ["ȋ","i"], +// ["ȍ","o"], +// ["ȏ","o"], +// ["ȑ","r"], +// ["ȓ","r"], +// ["ȕ","u"], +// ["ȗ","u"], +// ["ș","s"], +// ["ț","t"], +// ["ȟ","h"], +// ["ȧ","a"], +// ["ȩ","e"], +// ["ȫ","o"], +// ["ȭ","o"], +// ["ȯ","o"], +// ["ȱ","o"], +// ["ȳ","y"], +// ["ʰ","h"], +// ["ʱ","h"], +// ["ɦ","h"], +// ["ʲ","j"], +// ["ʳ","r"], +// ["ʴ","ɹ"], +// ["ʵ","ɻ"], +// ["ʶ","ʁ"], +// ["ʷ","w"], +// ["ʸ","y"], +// ["ˠ","ɣ"], +// ["ˡ","l"], +// ["ˢ","s"], +// ["ˣ","x"], +// ["ˤ","ʕ"], +// ["ΐ","ι"], +// ["ά","α"], +// ["έ","ε"], +// ["ή","η"], +// ["ί","ι"], +// ["ΰ","υ"], +// ["ϊ","ι"], +// ["ϋ","υ"], +// ["ό","ο"], +// ["ύ","υ"], +// ["ώ","ω"], +// ["ϐ","β"], +// ["ϑ","θ"], +// ["ϒ","Υ"], +// ["ϓ","Υ"], +// ["ϔ","Υ"], +// ["ϕ","φ"], +// ["ϖ","π"], +// ["ϰ","κ"], +// ["ϱ","ρ"], +// ["ϲ","ς"], +// ["ϵ","ε"], +// ["й","и"], +// ["ѐ","е"], +// ["ё","е"], +// ["ѓ","г"], +// ["ї","і"], +// ["ќ","к"], +// ["ѝ","и"], +// ["ў","у"], +// ["ѷ","ѵ"], +// ["ӂ","ж"], +// ["ӑ","а"], +// ["ӓ","а"], +// ["ӗ","е"], +// ["ӛ","ә"], +// ["ӝ","ж"], +// ["ӟ","з"], +// ["ӣ","и"], +// ["ӥ","и"], +// ["ӧ","о"], +// ["ӫ","ө"], +// ["ӭ","э"], +// ["ӯ","у"], +// ["ӱ","у"], +// ["ӳ","у"], +// ["ӵ","ч"] +// +// // Charset Normalization +// +// // ["à", "a"], +// // ["á", "a"], +// // ["â", "a"], +// // ["ã", "a"], +// // ["ä", "a"], +// // ["å", "a"], +// // +// // ["è", "e"], +// // ["é", "e"], +// // ["ê", "e"], +// // ["ë", "e"], +// // +// // ["ì", "i"], +// // ["í", "i"], +// // ["î", "i"], +// // ["ï", "i"], +// // +// // ["ò", "o"], +// // ["ó", "o"], +// // ["ô", "o"], +// // ["õ", "o"], +// // ["ö", "o"], +// // ["ő", "o"], +// // +// // ["ù", "u"], +// // ["ú", "u"], +// // ["û", "u"], +// // ["ü", "u"], +// // ["ű", "u"], +// // +// // ["ý", "y"], +// // ["ŷ", "y"], +// // ["ÿ", "y"], +// // +// // ["ñ", "n"], +// // ["ç", "c"], +// // ["ß", "s"] +// +// // Special Chars Removal +// +// // [",", ""], +// // [".", ""], +// // ["'", ""] +// +// // Non-Whitespace Separators +// +// // split by default p{P} +// // ["-", " "], +// // [":", " "], +// // ["_", " "], +// // ["|", " "], +// // ["/", " "], +// // ["\\", " "] +// ]); +// +// // let pairs; +// // +// // if(!normalize){ +// // +// // // Charset Normalization +// // +// // const regex_a = regex("[àáâãäå]"), +// // regex_e = regex("[èéêë]"), +// // regex_i = regex("[ìíîï]"), +// // regex_o = regex("[òóôõöő]"), +// // regex_u = regex("[ùúûüű]"), +// // regex_y = regex("[ýŷÿ]"), +// // regex_n = regex("ñ"), +// // //regex_c = regex("[çc]"), +// // regex_c = regex("ç"), +// // regex_s = regex("ß"), +// // //regex_and = regex(" & "), +// // regex_and = regex("&"); +// // +// // pairs = [ +// // regex_a, "a", +// // regex_e, "e", +// // regex_i, "i", +// // regex_o, "o", +// // regex_u, "u", +// // regex_y, "y", +// // regex_n, "n", +// // //regex_c, "k", +// // regex_c, "c", +// // regex_s, "s", +// // regex_and, " and " +// // ]; +// // } +// +// /** +// * @param {string|number} str +// */ +// +// export function encode(str){ +// +// //str = "" + str; +// +// return pipeline.call( +// +// this, +// /* string: */ (/*normalize ? str.normalize("NFD").replace(normalize, "") :*/ "" + str).toLowerCase(), +// /* normalize: */ pairs, +// /* split: */ whitespace, +// ///* collapse: */ false +// ); +// } diff --git a/src/lang/latin/soundex.js b/src/lang/latin/soundex.js new file mode 100644 index 0000000..b1524b6 --- /dev/null +++ b/src/lang/latin/soundex.js @@ -0,0 +1,51 @@ +export default { + normalize: true, + dedupe: false, + include: { + letter: true + }, + finalize: function(arr){ + for(let i = 0; i < arr.length; i++){ + arr[i] = soundex(arr[i]); + } + } +}; + +const codes = { + "a": "", "e": "", "i": "", "o": "", "u": "", "y": "", + "b": 1, "f": 1, "p": 1, "v": 1, + "c": 2, "g": 2, "j": 2, "k": 2, "q": 2, "s": 2, "x": 2, "z": 2, "ß": 2, + "d": 3, "t": 3, + "l": 4, + "m": 5, "n": 5, + "r": 6 +}; + +function soundex(stringToEncode){ + + let encodedString = stringToEncode.charAt(0); + let last = codes[encodedString]; + for(let i = 1, char; i < stringToEncode.length; i++){ + char = stringToEncode.charAt(i); + // Remove all occurrences of "h" and "w" + if(char !== "h" && char !== "w"){ + // Replace all consonants with digits + char = codes[char]; + // Remove all occurrences of a,e,i,o,u,y except first letter + if(char){ + // Replace all adjacent same digits with one digit + if(char !== last){ + encodedString += char; + last = char; + if(encodedString.length === 4){ + break; + } + } + } + } + } + // while(encodedString.length < 4){ + // encodedString += "0"; + // } + return encodedString; +} \ No newline at end of file diff --git a/src/polyfill.js b/src/polyfill.js deleted file mode 100644 index 394fd8a..0000000 --- a/src/polyfill.js +++ /dev/null @@ -1,79 +0,0 @@ -// COMPILER BLOCK --> -import { POLYFILL, SUPPORT_ASYNC } from "./config.js"; -// <-- COMPILER BLOCK -export let promise = Promise; - -if(POLYFILL){ - - Object.assign || (Object.assign = function(){ - - const args = arguments; - const size = args.length; - const obj = args[0]; - - for(let x = 1, current, keys, length; x < size; x++){ - - current = args[x]; - keys = Object.keys(current); - length = keys.length; - - for(let i = 0, key; i < length; i++){ - - key = keys[i]; - obj[key] = current[key]; - } - } - - return obj; - }); - - // Object.values || (Object.values = function(obj){ - // - // const keys = Object.keys(obj); - // const length = keys.length; - // const values = new Array(length); - // - // for(let x = 0; x < length; x++){ - // - // values[x] = obj[keys[x]]; - // } - // - // return values; - // }); - - if(SUPPORT_ASYNC && !promise){ - - /** - * @param {Function} fn - * @constructor - */ - - function SimplePromise(fn){ - - this.callback = null; - - const self = this; - - fn(function(val){ - - if(self.callback){ - - self.callback(val); - // self.callback = null; - // self = null; - } - }); - } - - /** - * @param {Function} callback - */ - - SimplePromise.prototype.then = function(callback){ - - this.callback = callback; - }; - - promise = SimplePromise; - } -} diff --git a/src/preset.js b/src/preset.js index a10dd00..88ee5fb 100644 --- a/src/preset.js +++ b/src/preset.js @@ -2,98 +2,58 @@ import { DEBUG } from "./config.js"; // <-- COMPILER BLOCK import { is_string } from "./common.js"; +import { IndexOptions } from "./type.js"; /** * @enum {Object} * @const */ -const preset = { +const presets = { "memory": { - charset: "latin:extra", - //tokenize: "strict", - resolution: 3, - //threshold: 0, - minlength: 4, - fastupdate: false + resolution: 1 }, "performance": { - //charset: "latin", - //tokenize: "strict", - resolution: 3, - minlength: 3, - //fastupdate: true, - optimize: false, - //fastupdate: true, + resolution: 6, + fastupdate: true, context: { - depth: 2, - resolution: 1 - //bidirectional: false + depth: 1, + resolution: 3 } }, "match": { - charset: "latin:extra", - tokenize: "reverse", - //resolution: 9, - //threshold: 0 + tokenize: "forward" }, "score": { - charset: "latin:advanced", - //tokenize: "strict", - resolution: 20, - minlength: 3, + resolution: 9, context: { - depth: 3, - resolution: 9, - //bidirectional: true + depth: 2, + resolution: 9 } - }, - - "default": { - // charset: "latin:default", - // tokenize: "strict", - // resolution: 3, - // threshold: 0, - // depth: 3 - }, - - // "fast": { - // //charset: "latin", - // //tokenize: "strict", - // threshold: 8, - // resolution: 9, - // depth: 1 - // } + } }; +/** + * + * @param {!IndexOptions|string} options + * @return {IndexOptions} + */ + export default function apply_preset(options){ - if(is_string(options)){ + const preset = is_string(options) + ? options + : options["preset"]; - if(DEBUG && !preset[options]){ - - console.warn("Preset not found: " + options); - } - - options = preset[options]; - } - else{ - - const preset = options["preset"]; - - if(preset){ - - if(DEBUG && !preset[preset]){ - - console.warn("Preset not found: " + preset); - } - - options = Object.assign({}, preset[preset], /** @type {Object} */ (options)); + if(preset){ + if(DEBUG && !presets[preset]){ + console.warn("Preset not found: " + preset); } + options = Object.assign({}, presets[preset], /** @type {Object} */ (options)); } return options; diff --git a/src/profiler.js b/src/profiler.js new file mode 100644 index 0000000..c4784f8 --- /dev/null +++ b/src/profiler.js @@ -0,0 +1,15 @@ +import { create_object } from "./common.js"; +const data = create_object(); + +/** + * @param {!string} name + */ + +export default function tick(name){ + + /** @type {!Object} */ + const profiler = data["profiler"] || (data["profiler"] = {}); + + profiler[name] || (profiler[name] = 0); + profiler[name]++; +} diff --git a/src/resolve/and.js b/src/resolve/and.js new file mode 100644 index 0000000..5056ea4 --- /dev/null +++ b/src/resolve/and.js @@ -0,0 +1,243 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object, get_max_len } from "../common.js"; +// import xor from "./xor.js"; +// import or from "./or.js"; +// import not from "./not.js"; + +Resolver.prototype.and = function(){ + if(this.result.length){ + + const self = this; + let args = arguments; + let first_argument = args[0]; + + if(first_argument instanceof Promise){ + return first_argument.then(function(){ + return self.and.apply(self, args); + }); + } + + if(first_argument[0]){ + // fix false passed parameter style + if(first_argument[0].index){ + return this.and.apply(this, first_argument); + } + } + + // for(let i = 0; i < args.length; i++){ + // if(args[i].result instanceof Promise){ + // return; + // } + // } + + // if(args.length < 2){ + // if(first_argument.index){ + // first_argument.resolve = false; + // return first_argument.index.search(first_argument); + // } + // } + + let final = []; + let promises = []; + let limit = 0, offset = 0, enrich, resolve; + + for(let i = 0, query; i < args.length; i++){ + if((query = args[i])){ + + let result; + if(query instanceof Resolver){ + result = query.result; + } + else if(query.constructor === Array){ + result = query; + } + else if(query.index){ + query.resolve = false; + result = query.index.search(query).result; + } + else if(query.or){ + result = this.or(query.or); + } + else if(query.xor){ + result = this.xor(query.xor); + } + else if(query.not){ + result = this.not(query.not); + } + else{ + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if(result instanceof Promise){ + promises.push(result); //{ query, result }; + } + } + } + + if(promises.length){ + return Promise.all(promises).then(function(){ + final = [self.result].concat(final); + self.result = intersect(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + final = [this.result].concat(final); + this.result = intersect(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? this.result : this; + } + return this; +} + +/** + * Aggregate the intersection of N raw results + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function intersect(result, limit, offset, enrich, resolve, boost){ + + // if(!result.length){ + // // todo remove + // console.log("Empty Result") + // return result; + // } + + if(result.length < 2){ + // todo remove + //console.log("Single Result") + return []; + // if(resolve){ + // return default_resolver(result[0], limit, offset, enrich); + // } + // else{ + // return result[0]; + // } + } + + let final = []; + let count = 0; + + // fast path single slot + // if(result.length < 2){ + // if(limit || offset){ + // let res = result[0]; + // for(let j = 0, ids; j < res.length; j++){ + // ids = res[j]; + // if(!ids) continue; + // for(let k = 0, id; k < ids.length; k++){ + // id = ids[k]; + // if(offset){ + // offset--; + // continue; + // } + // if(resolve){ + // final.push(id); + // } + // else{ + // final[j + this.boost] || (final[j + this.boost] = []); + // final[j + this.boost].push(id); + // } + // if(limit && ++count === limit){ + // this.boost = 0; + // return final; + // } + // } + // } + // } + // this.boost = 0; + // return result[0]; + // } + + let contain = create_object(); + let maxres = get_max_len(result); + if(!maxres) return final; + + // for(let j = 0, ids, res = result[0]; j < res.length; j++){ + // ids = res[j]; + // for(let k = 0; k < ids.length; k++){ + // contain[ids[k]] = 1; + // } + // } + + for(let i = 0, res; i < result.length; i++){ + res = result[i]; + if(!res || !res.length) return []; + let contain_new = create_object(); + let match = 0; + let last_round = i === result.length - 1; + + for(let j = 0, ids; j < maxres; j++){ + ids = res[j]; + if(!ids) continue; + + for(let k = 0, id, min; k < ids.length; k++){ + id = ids[k]; + // fill in first round + if(!i){ + // shift resolution +1 + // shift resolution by boost (inverse) + contain_new[id] = j + 1 + (i ? boost : 0); + match = 1; + } + // result in last round + else if(last_round){ + if((min = contain[id])){ + match = 1; + //if(!contain_new[id]){ + if(offset){ + offset--; + continue; + } + if(resolve){ + final.push(id); + } + else{ + // reduce resolution -1 + min--; + if(j < min) min = j; + final[min] || (final[min] = []); + final[min].push(id); + } + if(limit && ++count === limit){ + //this.boost = 0; + return final; + } + // shift resolution +1 + //contain_new[id] = min + 1; + //} + } + } + // check for intersection + else if((min = contain[id])){ + // shift resolution +1 + if(j + 1 < min) min = j + 1; + contain_new[id] = min; + match = 1; + } + } + } + + if(!match){ + //this.boost = 0; + return []; + } + + contain = contain_new; + } + + //this.boost = 0; + return final; +} diff --git a/src/resolve/default.js b/src/resolve/default.js new file mode 100644 index 0000000..f41702b --- /dev/null +++ b/src/resolve/default.js @@ -0,0 +1,114 @@ +import { concat } from "../common.js"; + +/* + from -> res[score][id] + to -> [id] +*/ + +/** + * Aggregate the union of a single raw result + * @param {!Array} result + * @param {!number} limit + * @param {number=} offset + * @param {boolean=} enrich + * @return Array + */ + +export default function(result, limit, offset, enrich){ + + // fast path: when there is just one slot in the result + if(result.length === 1){ + result = result[0]; + result = offset || (result.length > limit) + ? (limit + ? result.slice(offset, offset + limit) + : result.slice(offset) + ) + : result; + return enrich + ? enrich_result(result) + : result; + } + + // this is an optimized workaround instead of + // just doing result = concat(result) + + let final = []; + + for(let i = 0, arr, len; i < result.length; i++){ + if(!(arr = result[i]) || !(len = arr.length)) continue; + + if(offset){ + // forward offset pointer + if(offset >= len){ + offset -= len; + continue; + } + // apply offset pointer when length differs + if(offset < len){ + arr = limit + ? arr.slice(offset, offset + limit) + : arr.slice(offset); + len = arr.length; + offset = 0; + } + } + + if(!final.length){ + // fast path: when limit was reached in first slot + if(len >= limit){ + if(len > limit){ + arr = arr.slice(0, limit); + } + return enrich + ? enrich_result(arr) + : arr; + } + final = [arr]; + } + else{ + if(len > limit){ + arr = arr.slice(0, limit); + len = arr.length; + } + final.push(arr); + } + + // reduce limit + limit -= len; + + // todo remove + // if(limit < 0){ + // throw new Error("Impl.Error"); + // } + + // break if limit was reached + if(!limit){ + break; + } + } + + // todo remove + if(!final.length){ + //throw new Error("No results found"); + return final; + } + + final = final.length > 1 + ? concat(final) + : final[0]; + + return enrich + ? enrich_result(final) + : final; +} + +function enrich_result(ids){ + for(let i = 0; i < ids.length; i++){ + ids[i] = { + score: i, + id: ids[i] + }; + } + return ids; +} diff --git a/src/resolve/not.js b/src/resolve/not.js new file mode 100644 index 0000000..aeacf61 --- /dev/null +++ b/src/resolve/not.js @@ -0,0 +1,117 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object } from "../common.js"; +// import or from "./or.js"; +// import and from "./and.js"; +// import xor from "./xor.js"; + +Resolver.prototype.not = function(){ + const self = this; + let args = arguments; + let first_argument = args[0]; + + if(first_argument instanceof Promise){ + return first_argument.then(function(){ + return self.not.apply(self, args); + }); + } + + if(first_argument[0]){ + // fix false passed parameter style + if(first_argument[0].index){ + return this.not.apply(this, first_argument); + } + } + + let final = []; + let promises = []; + let limit = 0, offset = 0, enrich, resolve; + + for(let i = 0, query; i < args.length; i++){ + if((query = args[i])){ + + let result; + if(query instanceof Resolver){ + result = query.result; + } + else if(query.constructor === Array){ + result = query; + } + else if(query.index){ + query.resolve = false; + result = query.index.search(query).result; + } + else if(query.or){ + result = this.or(query.or); + } + else if(query.and){ + result = this.and(query.and); + } + else if(query.xor){ + result = this.xor(query.xor); + } + else{ + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if(result instanceof Promise){ + promises.push(result); //{ query, result }; + } + } + } + + if(promises.length){ + return Promise.all(promises).then(function(){ + self.result = exclusion.call(self, final, limit, offset, resolve); + return resolve ? self.result : self; + }); + } + + this.result = exclusion.call(this, final, limit, offset, resolve); + return resolve ? this.result : this; +} + +/** + * @param result + * @param limit + * @param offset + * @param resolve + * @this Resolver + * @return {Array} + */ + +function exclusion(result, limit, offset, resolve){ + + if(!result.length){ + return this.result; + } + + const final = []; + const exclude = new Set(result.flat().flat()); + + for(let j = 0, ids; j < this.result.length; j++){ + ids = this.result[j]; + if(!ids) continue; + + for(let k = 0, id; k < ids.length; k++){ + id = ids[k]; + if(!exclude.has(id)){ + if(resolve){ + final.push(id); + } + else{ + final[j] || (final[j] = []); + final[j].push(id); + } + } + } + } + + return final; +} diff --git a/src/resolve/or.js b/src/resolve/or.js new file mode 100644 index 0000000..ae701b9 --- /dev/null +++ b/src/resolve/or.js @@ -0,0 +1,184 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object, get_max_len } from "../common.js"; +// import xor from "./xor.js"; +// import and from "./and.js"; +// import not from "./not.js"; + +Resolver.prototype.or = function(){ + + const self = this; + let args = arguments; + let first_argument = args[0]; + + if(first_argument instanceof Promise){ + return first_argument.then(function(){ + return self.or.apply(self, args); + }); + } + + if(first_argument[0]){ + // fix false passed parameter style + if(first_argument[0].index){ + return this.or.apply(this, first_argument); + } + } + + // for(let i = 0; i < args.length; i++){ + // if(args[i].result instanceof Promise){ + // return; + // } + // } + + // if(args.length < 2){ + // if(first_argument.index){ + // first_argument.resolve = false; + // const result = first_argument.index.search(first_argument); + // if(result instanceof Promise){ + // result.then(function(result){ + // final = self.result.concat(result); + // self.result = resolver(final, limit, offset, enrich, !resolve); + // return resolve ? self.result : self; + // }); + // } + // else{ + // final = this.result.concat(result); + // this.result = resolver(final, limit, offset, enrich, !resolve); + // return resolve ? this.result : this; + // } + // } + // } + + let final = []; + let promises = []; + let limit = 0, offset = 0, enrich, resolve; + + for(let i = 0, query; i < args.length; i++){ + if((query = args[i])){ + + let result; + if(query instanceof Resolver){ + result = query.result; + } + else if(query.constructor === Array){ + result = query; + } + else if(query.index){ + query.resolve = false; + result = query.index.search(query).result; + } + else if(query.and){ + result = this.and(query.and); + } + else if(query.xor){ + result = this.xor(query.xor); + } + else if(query.not){ + result = this.not(query.not); + } + else{ + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if(result instanceof Promise){ + promises.push(result); //{ query, result }; + } + } + } + + if(promises.length){ + return Promise.all(promises).then(function(){ + self.result.length && (final = [self.result].concat(final)); + self.result = resolver(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + this.result.length && (final = [this.result].concat(final)); + this.result = resolver(final, limit, offset, enrich, resolve, self.boostval); + return resolve ? this.result : this; +}; + +/** + * Aggregate the union of N raw results + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function resolver(result, limit, offset, enrich, resolve, boost){ + + if(!result.length){ + // todo remove + //console.log("Empty Result") + return result; + } + + if(typeof limit === "object"){ + offset = limit.offset || 0; + enrich = limit.enrich || false; + limit = limit.limit || 0; + } + + if(result.length < 2){ + // todo remove + //console.log("Single Result") + if(resolve){ + return default_resolver(result[0], limit, offset, enrich); + } + else{ + return result[0]; + } + } + + let final = []; + let count = 0; + let dupe = create_object(); + let maxres = get_max_len(result); + + for(let j = 0, ids; j < maxres; j++){ + for(let i = 0, res; i < result.length; i++){ + res = result[i]; + if(!res) continue; + ids = res[j]; + if(!ids) continue; + + for(let k = 0, id; k < ids.length; k++){ + id = ids[k]; + if(!dupe[id]){ + dupe[id] = 1; + if(offset){ + offset--; + continue; + } + if(resolve){ + final.push(id); + } + else{ + // shift resolution by boost (inverse) + const index = j + (i ? boost : 0); + final[index] || (final[index] = []); + final[index].push(id); + } + if(limit && ++count === limit){ + //this.boost = 0; + return final; + } + } + } + } + } + + //this.boost = 0; + return final; +} diff --git a/src/resolve/xor.js b/src/resolve/xor.js new file mode 100644 index 0000000..f85f8ec --- /dev/null +++ b/src/resolve/xor.js @@ -0,0 +1,158 @@ +import Resolver from "../resolver.js"; +import default_resolver from "./default.js"; +import { create_object } from "../common.js"; +// import or from "./or.js"; +// import and from "./and.js"; +// import not from "./not.js"; + +Resolver.prototype.xor = function(){ + const self = this; + let args = arguments; + let first_argument = args[0]; + + if(first_argument instanceof Promise){ + return first_argument.then(function(){ + return self.xor.apply(self, args); + }); + } + + if(first_argument[0]){ + // fix false passed parameter style + if(first_argument[0].index){ + return this.xor.apply(this, first_argument); + } + } + + let final = []; + let promises = []; + let limit = 0, offset = 0, enrich, resolve; + + for(let i = 0, query; i < args.length; i++){ + if((query = args[i])){ + + let result; + if(query instanceof Resolver){ + result = query.result; + } + else if(query.constructor === Array){ + result = query; + } + else if(query.index){ + query.resolve = false; + result = query.index.search(query).result; + } + else if(query.or){ + result = this.or(query.or); + } + else if(query.and){ + result = this.and(query.and); + } + else if(query.not){ + result = this.not(query.not); + } + else{ + limit = query.limit || 0; + offset = query.offset || 0; + enrich = query.enrich; + resolve = query.resolve; + continue; + } + + final[i] = result; + + if(result instanceof Promise){ + promises.push(result); //{ query, result }; + } + } + } + + if(promises.length){ + return Promise.all(promises).then(function(){ + self.result.length && (final = [self.result].concat(final)); + self.result = exclusive(final, limit, offset, enrich, !resolve, self.boostval); + return resolve ? self.result : self; + }); + } + + this.result.length && (final = [this.result].concat(final)); + this.result = exclusive(final, limit, offset, enrich, !resolve, self.boostval); + return resolve ? this.result : this; +} + +/** + * @param result + * @param limit + * @param offset + * @param enrich + * @param resolve + * @param boost + * @return {Array} + */ + +function exclusive(result, limit, offset, enrich, resolve, boost){ + + if(!result.length){ + // todo remove + //console.log("Empty Result") + return result; + } + + if(result.length < 2){ + // todo remove + //console.log("Single Result") + if(resolve){ + return default_resolver(result[0], limit, offset, enrich); + } + else{ + return result[0]; + } + } + + const final = []; + const check = create_object() + + for(let i = 0, res; i < result.length; i++){ + res = result[i]; + if(!res) continue; + + for(let j = 0, ids; j < res.length; j++){ + ids = res[j]; + if(!ids) continue; + + for(let k = 0, id; k < ids.length; k++){ + id = ids[k]; + check[id] + ? check[id]++ + : check[id] = 1; + } + } + } + + for(let i = 0, res; i < result.length; i++){ + res = result[i]; + if(!res) continue; + + for(let j = 0, ids; j < res.length; j++){ + ids = res[j]; + if(!ids) continue; + + for(let k = 0, id; k < ids.length; k++){ + id = ids[k]; + if(check[id] === 1){ + if(resolve){ + final.push(id); + } + else{ + // shift resolution by boost (inverse) + const index = j + (i ? boost : 0); + final[index] || (final[index] = []); + final[index].push(id); + } + } + } + } + } + + //this.boost = 0; + return final; +} \ No newline at end of file diff --git a/src/resolver.js b/src/resolver.js new file mode 100644 index 0000000..f6c62d4 --- /dev/null +++ b/src/resolver.js @@ -0,0 +1,101 @@ +import default_resolver from "./resolve/default.js"; +import { set_resolve } from "./index/search.js"; +// import or from "./resolve/or.js"; +// import and from "./resolve/and.js"; +// import xor from "./resolve/xor.js"; +// import not from "./resolve/not.js"; +import "./resolve/or.js"; +import "./resolve/and.js"; +import "./resolve/xor.js"; +import "./resolve/not.js"; + +/** + * @param result + * @constructor + */ + +export default function Resolver(result){ + if(result && result.index){ + result.resolve = false; + this.index = result.index; + return result.index.search(result); + } + if(!(this instanceof Resolver)){ + return new Resolver(result); + } + if(result instanceof Resolver){ + // todo remove + //console.log("Resolver Loopback") + return result; + } + this.index = null; + this.result = result || []; + this.boostval = 0; +} + +// Resolver.prototype.or = or; +// Resolver.prototype.and = and; +// Resolver.prototype.not = not; +// Resolver.prototype.xor = xor; + +Resolver.prototype.limit = function(limit){ + if(this.result.length){ + const final = []; + let count = 0; + for(let j = 0, ids; j < this.result.length; j++){ + ids = this.result[j]; + if(ids.length + count < limit){ + final[j] = ids; + count += ids.length; + } + else{ + final[j] = ids.slice(0, limit - count); + this.result = final; + break; + } + } + } + return this; +}; + +Resolver.prototype.offset = function(offset){ + if(this.result.length){ + const final = []; + let count = 0; + for(let j = 0, ids; j < this.result.length; j++){ + ids = this.result[j]; + if(ids.length + count < offset){ + count += ids.length; + } + else{ + final[j] = ids.slice(offset - count); + count = offset; + } + } + this.result = final; + } + return this; +}; + +Resolver.prototype.boost = function(boost){ + this.boostval += boost; + return this; +}; + +Resolver.prototype.resolve = function(limit, offset, enrich){ + set_resolve(1); + const result = this.result; + this.index = null; + this.result = null; + + if(result.length){ + if(typeof limit === "object"){ + enrich = limit.enrich; + offset = limit.offset; + limit = limit.limit; + } + return default_resolver(result, limit || 100, offset, enrich); + } + + return result; +}; diff --git a/src/serialize.js b/src/serialize.js index 6362d20..d7dac2b 100644 --- a/src/serialize.js +++ b/src/serialize.js @@ -1,11 +1,12 @@ // TODO return promises instead of inner await -import { IndexInterface, DocumentInterface } from "./type.js"; +import Index from "./index.js"; +import Document from "./document.js"; import { create_object, is_string } from "./common.js"; function async(callback, self, field, key, index_doc, index, data, on_done){ - setTimeout(function(){ + //setTimeout(function(){ const res = callback(field ? field + "." + key : key, JSON.stringify(data)); @@ -22,11 +23,17 @@ function async(callback, self, field, key, index_doc, index, data, on_done){ self.export(callback, self, field, index_doc, index + 1, on_done); } - }); + //}); } /** - * @this IndexInterface + * @param callback + * @param self + * @param field + * @param index_doc + * @param index + * @param on_done + * @this {Index|Document} */ export function exportIndex(callback, self, field, index_doc, index, on_done){ @@ -52,14 +59,14 @@ export function exportIndex(callback, self, field, index_doc, index, on_done){ data = create_object(); - for(let key in this.register){ + for(let key of this.reg.keys()){ data[key] = 1; } } else{ - data = this.register; + data = this.reg; } break; @@ -102,7 +109,7 @@ export function exportIndex(callback, self, field, index_doc, index, on_done){ } /** - * @this IndexInterface + * @this Index */ export function importIndex(key, data){ @@ -129,7 +136,7 @@ export function importIndex(key, data){ // fastupdate isn't supported by import this.fastupdate = false; - this.register = data; + this.reg = data; break; case "map": @@ -145,7 +152,7 @@ export function importIndex(key, data){ } /** - * @this DocumentInterface + * @this Document */ export function exportDocument(callback, self, field, index_doc, index, on_done){ @@ -167,7 +174,7 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) self = this; - setTimeout(function(){ + //setTimeout(function(){ if(!idx.export(callback, self, index ? field/*.replace(":", "-")*/ : "", index_doc, index++, on_done)){ @@ -176,7 +183,7 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) self.export(callback, self, field, index_doc, index, on_done); } - }); + //}); } else{ @@ -217,7 +224,7 @@ export function exportDocument(callback, self, field, index_doc, index, on_done) } /** - * @this DocumentInterface + * @this Document */ export function importDocument(key, data){ @@ -244,12 +251,12 @@ export function importDocument(key, data){ // fastupdate isn't supported by import this.fastupdate = false; - this.register = data; + this.reg = data; for(let i = 0, index; i < this.field.length; i++){ index = this.index[this.field[i]]; - index.register = data; + index.reg = data; index.fastupdate = false; } diff --git a/src/type.js b/src/type.js index f6acd1b..0c377c0 100644 --- a/src/type.js +++ b/src/type.js @@ -1,69 +1,144 @@ -/** - * @interface - */ - -export function IndexInterface(){ - - this.cache = null; - this.matcher = null; - this.stemmer = null; - this.filter = null; -} +// When you are looking for type definitions which fully describes the usage take a look into the index.d.ts file. +// Some of the types here aren't supposed to be used as public, they might be defined just for internal state. +import Encoder from "./encoder.js"; +import StorageInterface from "./db/interface.js"; /** - * @param {!string} str - * @param {boolean|Array=} normalize - * @param {boolean|string|RegExp=} split - * @param {boolean=} collapse - * @returns {string|Array} + * @typedef IndexOptions {{ + * preset: string|undefined, + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * resolve: [boolean=true], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -//IndexInterface.prototype.pipeline; +export let IndexOptions; /** - * @param {!number|string} id - * @param {!string} content + * @typedef DocumentOptions {{ + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * db: StorageInterface|undefined, + * doc: DocumentDescriptor|Array|undefined, + * document: DocumentDescriptor|Array|undefined, + * worker: boolean|string|undefined + * }} */ - -IndexInterface.prototype.add; +export let DocumentOptions; /** - * @param {!number|string} id - * @param {!string} content + * @typedef ContextOptions {{ + * depth: number, + * bidirectional: boolean|undefined, + * resolution: number|undefined + * }} */ - -IndexInterface.prototype.append; +export let ContextOptions; /** - * @param {!string|Object} query - * @param {number|Object=} limit - * @param {Object=} options - * @returns {Array} + * @typedef DocumentDescriptor {{ + * field: FieldOptions|Array|undefined, + * index: FieldOptions|Array|undefined, + * tag: TagOptions|Array|undefined, + * store: StoreOptions|Array|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.search; +export let DocumentDescriptor; /** - * @param {!number|string} id - * @param {!string} content + * @typedef FieldOptions {{ + * field: string, + * filter: Function|undefined, + * custom: Function|undefined, + * context: (IndexOptions|undefined), + * encoder: Encoder|Function|Object|undefined, + * encode: Function|undefined, + * resolution: [number=9], + * tokenize: [string="strict"], + * fastupdate: [boolean:false], + * score: Function]|undefined, + * keystore: [number=0], + * rtl: [boolean=false], + * cache: [number=null], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.update; +export let FieldOptions; /** - * @param {!number|string} id + * @typedef TagOptions {{ + * field: string, + * tag: Object>|Array|string, + * filter: Function|undefined, + * custom: Function|undefined, + * keystore: [number=0], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ - -IndexInterface.prototype.remove; +export let TagOptions; /** - * @interface + * @typedef StoreOptions {{ + * field: string, + * filter: Function|undefined, + * custom: Function|undefined, + * keystore: [number=0], + * db: StorageInterface|undefined, + * config: string|undefined + * }} */ +export let StoreOptions; -export function DocumentInterface(){ +/** + * @typedef SearchOptions {{ + * query: string=, + * limit: [number=100], + * offset: [number=0], + * context: boolean|undefined, + * suggest: [boolean=false], + * resolve: [boolean=true], + * enrich: [boolean=false], + * tag: Array|undefined + * }} + */ +export let SearchOptions; - this.field = null; +/** + * @typedef DocumentSearchOptions {{ + * query: string=, + * limit: [number=100], + * offset: [number=0], + * context: boolean|undefined, + * suggest: [boolean=false], + * enrich: [boolean=false], + * tag: Array|undefined, + * field: FieldOptions|Array|undefined, + * index: FieldOptions|Array|undefined, + * pluck: boolean|undefined, + * merge: [boolean=false] + * }} + */ +export let DocumentSearchOptions; - /** @type IndexInterface */ - this.index = null; -} +export let EncoderOptions; +export let ResolverOptions; \ No newline at end of file diff --git a/src/webpack.js b/src/webpack.js index 1e17b96..7204dee 100644 --- a/src/webpack.js +++ b/src/webpack.js @@ -1,22 +1,40 @@ -import { RELEASE, SUPPORT_ASYNC, SUPPORT_DOCUMENT, SUPPORT_CACHE, SUPPORT_SERIALIZE, SUPPORT_WORKER, SUPPORT_ENCODER } from "./config.js"; +// COMPILER BLOCK --> +import { + RELEASE, + SUPPORT_ASYNC, + SUPPORT_DOCUMENT, + SUPPORT_CACHE, + SUPPORT_SERIALIZE, + SUPPORT_WORKER, + SUPPORT_ENCODER, + SUPPORT_PERSISTENT, + SUPPORT_RESOLVER, + SUPPORT_STORE +} from "./config.js"; +// <-- COMPILER BLOCK +import { + SearchOptions, + ContextOptions, + DocumentDescriptor, + DocumentSearchOptions, + FieldOptions, + IndexOptions, + DocumentOptions +} from "./type.js"; import Document from "./document.js"; import Index from "./index.js"; import WorkerIndex from "./worker/index.js"; -import { registerCharset, registerLanguage } from "./global.js"; +import Resolver from "./resolver.js"; +import Encoder from "./encoder.js"; +import IdxDB from "./db/indexeddb/index.js"; +import { global_charset, global_lang } from "./global.js"; +import charset_exact from "./lang/latin/exact.js" import charset_default from "./lang/latin/default.js" import charset_simple from "./lang/latin/simple.js" import charset_balance from "./lang/latin/balance.js" import charset_advanced from "./lang/latin/advanced.js" import charset_extra from "./lang/latin/extra.js" - -/** @export */ Document.prototype.add; -/** @export */ Document.prototype.append; -/** @export */ Document.prototype.search; -/** @export */ Document.prototype.update; -/** @export */ Document.prototype.remove; -/** @export */ Document.prototype.contain; -/** @export */ Document.prototype.get; -/** @export */ Document.prototype.set; +import charset_soundex from "./lang/latin/soundex.js" /** @export */ Index.prototype.add; /** @export */ Index.prototype.append; @@ -24,77 +42,176 @@ import charset_extra from "./lang/latin/extra.js" /** @export */ Index.prototype.update; /** @export */ Index.prototype.remove; /** @export */ Index.prototype.contain; +/** @export */ Index.prototype.clear; +/** @export */ Index.prototype.cleanup; + +if(SUPPORT_DOCUMENT){ +/** @export */ Document.prototype.add; +/** @export */ Document.prototype.append; +/** @export */ Document.prototype.search; +/** @export */ Document.prototype.update; +/** @export */ Document.prototype.remove; +/** @export */ Document.prototype.contain; +/** @export */ Document.prototype.clear; +/** @export */ Document.prototype.cleanup; +} +if(SUPPORT_DOCUMENT && SUPPORT_STORE){ +/** @export */ Document.prototype.get; +/** @export */ Document.prototype.set; +} if(SUPPORT_CACHE){ - /** @export */ Index.prototype.searchCache; +} +if(SUPPORT_CACHE && SUPPORT_DOCUMENT){ /** @export */ Document.prototype.searchCache; } if(SUPPORT_ASYNC){ - -/** @export */ Document.prototype.addAsync; -/** @export */ Document.prototype.appendAsync; -/** @export */ Document.prototype.searchAsync; -/** @export */ Document.prototype.updateAsync; -/** @export */ Document.prototype.removeAsync; - /** @export */ Index.prototype.addAsync; /** @export */ Index.prototype.appendAsync; /** @export */ Index.prototype.searchAsync; /** @export */ Index.prototype.updateAsync; /** @export */ Index.prototype.removeAsync; } +if(SUPPORT_ASYNC && SUPPORT_DOCUMENT){ +/** @export */ Document.prototype.addAsync; +/** @export */ Document.prototype.appendAsync; +/** @export */ Document.prototype.searchAsync; +/** @export */ Document.prototype.updateAsync; +/** @export */ Document.prototype.removeAsync; +} if(SUPPORT_SERIALIZE){ - /** @export */ Index.prototype.export; /** @export */ Index.prototype.import; +} +if(SUPPORT_SERIALIZE && SUPPORT_DOCUMENT){ /** @export */ Document.prototype.export; /** @export */ Document.prototype.import; } -if(SUPPORT_ENCODER){ +if(SUPPORT_PERSISTENT){ +/** @export */ Index.prototype.mount; +/** @export */ Index.prototype.commit; +/** @export */ Index.db; +} +if(SUPPORT_PERSISTENT && SUPPORT_DOCUMENT){ +/** @export */ Document.prototype.mount; +/** @export */ Document.prototype.commit; +/** @export */ Document.db; +} - registerCharset("latin:default", charset_default); - registerCharset("latin:simple", charset_simple); - registerCharset("latin:balance", charset_balance); - registerCharset("latin:advanced", charset_advanced); - registerCharset("latin:extra", charset_extra); +/** @export */ IndexOptions.preset; +/** @export */ IndexOptions.context; +/** @export */ IndexOptions.encoder; +/** @export */ IndexOptions.encode; +/** @export */ IndexOptions.resolution; +/** @export */ IndexOptions.tokenize; +/** @export */ IndexOptions.fastupdate; +/** @export */ IndexOptions.score; +/** @export */ IndexOptions.keystore; +/** @export */ IndexOptions.rtl; +/** @export */ IndexOptions.cache; +/** @export */ IndexOptions.resolve; +/** @export */ IndexOptions.db; + +/** @export */ DocumentOptions.context; +/** @export */ DocumentOptions.encoder; +/** @export */ DocumentOptions.encode; +/** @export */ DocumentOptions.resolution; +/** @export */ DocumentOptions.tokenize; +/** @export */ DocumentOptions.fastupdate; +/** @export */ DocumentOptions.score; +/** @export */ DocumentOptions.keystore; +/** @export */ DocumentOptions.rtl; +/** @export */ DocumentOptions.cache; +/** @export */ DocumentOptions.db; +/** @export */ DocumentOptions.doc; +/** @export */ DocumentOptions.document; +/** @export */ DocumentOptions.worker; + +/** @export */ DocumentDescriptor.field; +/** @export */ DocumentDescriptor.index; +/** @export */ DocumentDescriptor.tag; +/** @export */ DocumentDescriptor.store; + +/** @export */ ContextOptions.depth; +/** @export */ ContextOptions.bidirectional; +/** @export */ ContextOptions.resolution; + +/** @export */ SearchOptions.query; +/** @export */ SearchOptions.limit; +/** @export */ SearchOptions.offset; +/** @export */ SearchOptions.context; +/** @export */ SearchOptions.suggest; +/** @export */ SearchOptions.resolve; +/** @export */ SearchOptions.enrich; +/** @export */ SearchOptions.tag; + +/** @export */ DocumentSearchOptions.query; +/** @export */ DocumentSearchOptions.limit; +/** @export */ DocumentSearchOptions.offset; +/** @export */ DocumentSearchOptions.context; +/** @export */ DocumentSearchOptions.suggest; +/** @export */ DocumentSearchOptions.enrich; +/** @export */ DocumentSearchOptions.tag; +/** @export */ DocumentSearchOptions.field; +/** @export */ DocumentSearchOptions.index; +/** @export */ DocumentSearchOptions.pluck; +/** @export */ DocumentSearchOptions.merge; + +if(SUPPORT_ENCODER){ + global_charset["latin:exact"] = charset_exact; + global_charset["latin:default"] = charset_default; + global_charset["latin:simple"] = charset_simple; + global_charset["latin:balance"] = charset_balance; + global_charset["latin:advanced"] = charset_advanced; + global_charset["latin:extra"] = charset_extra; + global_charset["latin:soundex"] = charset_soundex; } const FlexSearch = { - "Index": Index, + "Encoder": Encoder, + "Charset": global_charset, + "Language": global_lang, "Document": SUPPORT_DOCUMENT ? Document : null, "Worker": SUPPORT_WORKER ? WorkerIndex : null, - "registerCharset": registerCharset, - "registerLanguage": registerLanguage + "Resolver": SUPPORT_RESOLVER ? Resolver : null, + "IndexedDB": SUPPORT_PERSISTENT ? IdxDB : null, + //"registerCharset": registerCharset, + //"registerLanguage": registerLanguage }; -if(RELEASE !== "bundle.module" && RELEASE !== "light.module" && RELEASE !== "compact.module"){ +// Export as library (Bundle) +// -------------------------------- - let tmp; +if(RELEASE !== "bundle.module" && + RELEASE !== "light.module" && + RELEASE !== "compact.module" && + RELEASE !== "custom.module"){ - if((tmp = self["define"]) && tmp["amd"]){ - - tmp([], function(){ + const root = self; + let prop; + // AMD (RequireJS) + if((prop = root["define"]) && prop["amd"]){ + prop([], function(){ return FlexSearch; }); } - else if(self["exports"]){ - - self["exports"] = FlexSearch; + // CommonJS + else if(typeof root["exports"] === "object"){ + root["exports"] = FlexSearch; } + // Global (window) else{ - /** @export */ - self.FlexSearch = FlexSearch; + root.FlexSearch = FlexSearch; } } else{ - /** @export */ - self.FlexSearch = FlexSearch; -} \ No newline at end of file + window.FlexSearch = FlexSearch; +} diff --git a/src/worker/handler.js b/src/worker/handler.js index b63c847..df9fdbd 100644 --- a/src/worker/handler.js +++ b/src/worker/handler.js @@ -1,6 +1,7 @@ import Index from "../index.js"; +import { IndexOptions } from "../type.js"; -export default function(data) { +export default async function(data) { data = data["data"]; @@ -13,17 +14,24 @@ export default function(data) { case "init": - const options = data["options"] || {}; - const factory = data["factory"]; - const encode = options["encode"]; - - options["cache"] = false; - - if(encode && (encode.indexOf("function") === 0)){ - - options["encode"] = Function("return " + encode)(); + /** @type IndexOptions */ + let options = data["options"] || {}; + let filepath = options.config; + if(filepath){ + options = filepath; + // will be replaced after build with the line below because + // there is an issue with closure compiler dynamic import + //options = await import(filepath); } + // deprecated: + // const encode = options.encode; + // if(encode && (encode.indexOf("function") === 0)){ + // options.encode = Function("return " + encode)(); + // } + + const factory = data["factory"]; + if(factory){ // export the FlexSearch global payload to "self" @@ -40,6 +48,7 @@ export default function(data) { self["_index"] = new Index(options); } + postMessage({ "id": data["id"] }); break; default: diff --git a/src/worker/index.js b/src/worker/index.js index 926e73e..c6be06c 100644 --- a/src/worker/index.js +++ b/src/worker/index.js @@ -1,32 +1,29 @@ //import { promise as Promise } from "../polyfill.js"; +import { IndexOptions } from "../type.js"; import { create_object, is_function, is_object, is_string } from "../common.js"; import handler from "./handler.js"; let pid = 0; /** - * @param {Object=} options + * @param {IndexOptions=} options * @constructor */ function WorkerIndex(options){ if(!(this instanceof WorkerIndex)) { - return new WorkerIndex(options); } - let opt; - if(options){ - - if(is_function(opt = options["encode"])){ - - options["encode"] = opt.toString(); - } + // deprecated: + // was replaced by dynamic config loading + // if(is_function(options.encode)){ + // options.encode = options.encode.toString(); + // } } else{ - options = {}; } @@ -34,16 +31,14 @@ function WorkerIndex(options){ // we use "self" as a trap for node.js let factory = (self||window)["_factory"]; - if(factory){ - factory = factory.toString(); } const is_node_js = typeof window === "undefined" && self["exports"]; const _self = this; - this.worker = create(factory, is_node_js, options["worker"]); + this.worker = create(factory, is_node_js, options.worker); this.resolver = create_object(); if(!this.worker){ @@ -51,26 +46,38 @@ function WorkerIndex(options){ return; } - if(is_node_js){ - - this.worker["on"]("message", function(msg){ - - _self.resolver[msg["id"]](msg["msg"]) ; - delete _self.resolver[msg["id"]]; - }); + function onmessage(msg){ + msg = msg["data"] || msg; + const id = msg["id"]; + const res = id && _self.resolver[id]; + if(res){ + res(msg["msg"]); + delete _self.resolver[id]; + } } - else{ - this.worker.onmessage = function(msg){ + is_node_js + ? this.worker["on"]("message", onmessage) + : this.worker.onmessage = onmessage; - msg = msg["data"]; - _self.resolver[msg["id"]](msg["msg"]); - delete _self.resolver[msg["id"]]; - }; + if(options["config"]){ + + // when extern configuration needs to be loaded + // it needs to return a promise to await for + return new Promise(function(resolve){ + _self.resolver[++pid] = function(){ + resolve(_self); + }; + _self.worker.postMessage({ + "id": pid, + "task": "init", + "factory": factory, + "options": options + }); + }); } this.worker.postMessage({ - "task": "init", "factory": factory, "options": options @@ -96,27 +103,22 @@ function register(key){ let callback; if(is_function(arg)){ - callback = arg; args.splice(args.length - 1, 1); } const promise = new Promise(function(resolve){ - - setTimeout(function(){ - + //setTimeout(function(){ self.resolver[++pid] = resolve; self.worker.postMessage({ - "task": key, "id": pid, "args": args }); - }); + //}); }); if(callback){ - promise.then(callback); return this; } @@ -131,27 +133,21 @@ function create(factory, is_node_js, worker_path){ let worker - try{ - - worker = is_node_js ? - - eval('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') - :( - factory ? - - new Worker(URL.createObjectURL( - - new Blob([ - - "onmessage=" + handler.toString() - - ], { "type": "text/javascript" }) - )) - : - new Worker(is_string(worker_path) ? worker_path : "worker/worker.js", { type: "module" }) - ); - } - catch(e){} + worker = is_node_js ? + // This eval will be removed when compiling, it isn't there in final build + (0,eval)('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') + //eval('new (require("worker_threads")["Worker"])(__dirname + "/node/node.js")') + :( + factory ? + new window.Worker(URL.createObjectURL( + new Blob( + ["onmessage=" + handler.toString()], + { "type": "text/javascript" } + ) + )) + : + new window.Worker(is_string(worker_path) ? worker_path : "worker/worker.js", { type: "module" }) + ); return worker; } \ No newline at end of file diff --git a/src/worker/node.js b/src/worker/node.js index 8a8a468..bea6cd6 100644 --- a/src/worker/node.js +++ b/src/worker/node.js @@ -1,5 +1,8 @@ const { parentPort } = require("worker_threads"); -const { Index } = require("../flexsearch.bundle.min.js"); +const { join } = require("path"); +// TODO EXCHANGE +const { Index } = require("../../dist/flexsearch.bundle.min.js"); +//const { Index } = require("../flexsearch.bundle.min.js"); let index; @@ -14,15 +17,24 @@ parentPort.on("message", function(data){ case "init": - const options = data["options"] || {}; - const encode = options["encode"]; + let options = data["options"] || {}; - options["cache"] = false; - - if(encode && (encode.indexOf("function") === 0)){ - - options["encode"] = new Function("return " + encode)(); + // load extern field configuration + let filepath = options["config"]; + if(filepath[0] !== "/" && filepath[0] !== "\\"){ + // current working directory + const dir = process.cwd(); + filepath = join(dir, filepath); } + if(filepath){ + options = require(filepath); + } + + // deprecated: + // const encode = options["encode"]; + // if(encode && (encode.indexOf("function") === 0)){ + // options["encode"] = new Function("return " + encode)(); + // } index = new Index(options); break; diff --git a/task/babel.bundle.json b/task/babel.bundle.json index 1b52d66..a084f8a 100644 --- a/task/babel.bundle.json +++ b/task/babel.bundle.json @@ -5,16 +5,21 @@ "define": { "RELEASE": "module", "DEBUG": false, - "POLYFILL": true, + "PROFILER": false, + "POLYFILL": false, "SUPPORT_WORKER": true, "SUPPORT_ENCODER": true, "SUPPORT_CACHE": true, "SUPPORT_ASYNC": true, "SUPPORT_STORE": true, - "SUPPORT_TAGS": true, "SUPPORT_SUGGESTION": true, "SUPPORT_SERIALIZE": true, - "SUPPORT_DOCUMENT": true + "SUPPORT_DOCUMENT": true, + "SUPPORT_TAGS": true, + "SUPPORT_PERSISTENT": true, + "SUPPORT_KEYSTORE": true, + "SUPPORT_COMPRESSION": true, + "SUPPORT_RESOLVER": true } }], "babel-plugin-minify-constant-folding", @@ -32,8 +37,8 @@ "babel-plugin-transform-undefined-to-void" ], "ignore": [ - "webpack.js", - "config.js" + "config.js", + "profiler.js" ], "minified": false, "compact": false, diff --git a/task/babel.debug.json b/task/babel.debug.json index 6faad82..b1b7319 100644 --- a/task/babel.debug.json +++ b/task/babel.debug.json @@ -5,16 +5,21 @@ "define": { "RELEASE": "module", "DEBUG": true, - "POLYFILL": true, + "PROFILER": false, + "POLYFILL": false, "SUPPORT_WORKER": true, "SUPPORT_ENCODER": true, "SUPPORT_CACHE": true, "SUPPORT_ASYNC": true, "SUPPORT_STORE": true, - "SUPPORT_TAGS": true, "SUPPORT_SUGGESTION": true, "SUPPORT_SERIALIZE": true, - "SUPPORT_DOCUMENT": true + "SUPPORT_DOCUMENT": true, + "SUPPORT_TAGS": true, + "SUPPORT_PERSISTENT": true, + "SUPPORT_KEYSTORE": true, + "SUPPORT_COMPRESSION": true, + "SUPPORT_RESOLVER": true } }], "babel-plugin-minify-constant-folding", @@ -32,8 +37,8 @@ "babel-plugin-transform-undefined-to-void" ], "ignore": [ - "webpack.js", - "config.js" + "config.js", + "profiler.js" ], "minified": false, "compact": false, diff --git a/task/babel.js b/task/babel.js index 7c1bd5c..f8b2df4 100644 --- a/task/babel.js +++ b/task/babel.js @@ -6,57 +6,67 @@ const minify = process.argv[2] && process.argv[2].toLowerCase().includes("releas console.log("Start build ....."); console.log('Bundle: ' + ('module' /* 'custom' */) + (debug ? ":debug" : (minify ? ":min" : ""))); -//fs.existsSync("log") || fs.mkdirSync("log"); -fs.existsSync("tmp") || fs.mkdirSync("tmp"); +fs.rmSync("tmp/", { recursive: true }); +fs.mkdirSync("tmp"); fs.existsSync("dist") || fs.mkdirSync("dist"); -const files = [ +(async function(){ - "async.js", - "cache.js", - "common.js", - "config.js", - "document.js", - "engine.js", - "global.js", - "index.js", - "intersect.js", - "lang.js", - "polyfill.js", - "preset.js", - "serialize.js", - "type.js", - "webpack.js" -]; + let files = await fs.promises.readdir("./src/"); + files.forEach(function(file){ + if(file.endsWith(".js")){ + let src = fs.readFileSync("src/" + file, "utf8"); + src = src.replace(/\/\/ COMPILER BLOCK -->(.*)<-- COMPILER BLOCK/gs, ""); + fs.writeFileSync("tmp/" + file, src); + } + }); -files.forEach(function(file){ + fs.existsSync("./tmp/db") || fs.mkdirSync("./tmp/db/"); + fs.existsSync("./tmp/lang") || fs.mkdirSync("./tmp/lang/"); - let src = String(fs.readFileSync("src/" + file)); - src = src.replace(/\/\/ COMPILER BLOCK -->(.*)<-- COMPILER BLOCK/gs, ""); - fs.writeFileSync("tmp/" + file, src); -}); + ["db/clickhouse", + "db/indexeddb", + "db/mongo", + "db/postgres", + "db/redis", + "db/sqlite", + "document", + "index", + "resolve", + "worker", + "lang", + "lang/latin", + ].forEach(await async function(path){ + fs.existsSync("./tmp/" + path + "/") || fs.mkdirSync("./tmp/" + path + "/"); + files = await fs.promises.readdir("./src/" + path + "/"); + files.forEach(function(file){ + if(file.endsWith(".old.js")) return; + if(file.endsWith(".wip.js")) return; + if(file.endsWith(".js")){ + let src = fs.readFileSync("src/" + path + "/" + file, "utf8"); + src = src.replace(/\/\/ COMPILER BLOCK -->(.*)<-- COMPILER BLOCK/gs, ""); + fs.writeFileSync("tmp/" + path + "/" + file, src); + } + }); + }); -fs.copyFileSync("task/babel." + (debug ? "debug": (minify ? "min" : "bundle")) + ".json", "tmp/.babelrc"); + fs.copyFileSync("task/babel." + (debug ? "debug": (minify ? "min" : "bundle")) + ".json", "tmp/.babelrc"); + fs.rmSync("dist/module" + (debug ? "-debug" : (minify ? "-min" : "")), { recursive: true }); + fs.mkdirSync("dist/module" + (debug ? "-debug" : (minify ? "-min" : ""))); -exec("npx babel tmp -d dist/module" + (debug ? "-debug" : (minify ? "-min --minified --compact true" : "")) + " --config-file tmp/.babelrc && exit 0", function(){ - - console.log("Build Complete."); -}); + exec("npx babel tmp -d dist/module" + (debug ? "-debug" : (minify ? "-min --minified --compact true" : "")) + " --config-file tmp/.babelrc && exit 0", function(){ + console.log("Build Complete."); + }); +}()); function exec(prompt, callback){ const child = child_process.exec(prompt, function(err, stdout, stderr){ - if(err){ - console.error(err); } else{ - - if(callback){ - - callback(); - } + callback && callback(); } }); diff --git a/task/babel.min.json b/task/babel.min.json index ee5724a..05c355d 100644 --- a/task/babel.min.json +++ b/task/babel.min.json @@ -5,16 +5,21 @@ "define": { "RELEASE": "module", "DEBUG": false, - "POLYFILL": true, + "PROFILER": false, + "POLYFILL": false, "SUPPORT_WORKER": true, "SUPPORT_ENCODER": true, "SUPPORT_CACHE": true, "SUPPORT_ASYNC": true, "SUPPORT_STORE": true, - "SUPPORT_TAGS": true, "SUPPORT_SUGGESTION": true, "SUPPORT_SERIALIZE": true, - "SUPPORT_DOCUMENT": true + "SUPPORT_DOCUMENT": true, + "SUPPORT_TAGS": true, + "SUPPORT_PERSISTENT": true, + "SUPPORT_KEYSTORE": true, + "SUPPORT_COMPRESSION": true, + "SUPPORT_RESOLVER": true } }], "babel-plugin-minify-constant-folding", @@ -34,8 +39,8 @@ "babel-plugin-transform-undefined-to-void" ], "ignore": [ - "webpack.js", - "config.js" + "config.js", + "profiler.js" ], "minified": true, "compact": true, diff --git a/task/build.js b/task/build.js index 2ee21f5..7394301 100644 --- a/task/build.js +++ b/task/build.js @@ -5,20 +5,17 @@ console.log("Start build ....."); fs.rmSync("tmp/", { recursive: true }); fs.mkdirSync("tmp"); -//fs.existsSync("log") || fs.mkdirSync("log"); fs.existsSync("dist") || fs.mkdirSync("dist"); -var supported_lang = [ - +let supported_lang = [ 'en', 'de', - 'at', + 'fr', 'us' ]; -var supported_charset = { - - 'latin': ["default", "advanced", "balance", "extra", "simple"], +let supported_charset = { + 'latin': ["default", "advanced", "balance", "extra", "simple", "extreme"], 'cjk': ["default"], 'cyrillic': ["default"], 'arabic': ["default"], @@ -27,10 +24,10 @@ var supported_charset = { let flag_str = ""; let language_out; let use_polyfill; -var formatting; -var compilation_level; +let formatting; +let compilation_level; -var options = (function(argv){ +let options = (function(argv){ const arr = {}; let count = 0; @@ -47,17 +44,9 @@ var options = (function(argv){ language_out = val; } - else if(index === "FORMATTING"){ - - formatting = val; - } - else if(index === "COMPILATION_LEVEL"){ - - compilation_level = val; - } else if(index === "POLYFILL"){ - use_polyfill = val === "true"; + use_polyfill = val !== "false"; } else{ @@ -78,33 +67,15 @@ const light_version = (release === "light") || (process.argv[2] === "--light"); const es5_version = (release === "es5") || (process.argv[2] === "--es5"); const module_version = (release === "module") || (process.argv[2] === "--module"); -// if(release){ -// -// let filename -// -// if(!fs.existsSync(filename = "src/config/" + release + "/config.js")){ -// -// filename = "src/config/bundle/config.js"; -// } -// -// fs.writeFileSync("tmp/config.js", fs.readFileSync(filename)); -// } - let parameter = (function(opt){ - if(formatting && !opt["formatting"]){ - - opt["formatting"] = formatting; - } - let parameter = ''; for(let index in opt){ if(opt.hasOwnProperty(index)){ - if((release !== "lang") /*|| (index !== "entry_point")*/){ - + if(release !== "lang"){ parameter += ' --' + index + '=' + opt[index]; } } @@ -113,7 +84,7 @@ let parameter = (function(opt){ return parameter; })({ - compilation_level: compilation_level || "ADVANCED_OPTIMIZATIONS", //"SIMPLE" + compilation_level: release === "bundle.profiler" ? "SIMPLE" : "ADVANCED", //"WHITESPACE" use_types_for_optimization: true, generate_exports: true, export_local_property_definitions: true, @@ -133,7 +104,7 @@ let parameter = (function(opt){ //js_module_root: "./", entry_point: "./tmp/webpack.js", //manage_closure_dependencies: true, - dependency_mode: "PRUNE_LEGACY", // PRUNE_LEGACY + dependency_mode: "PRUNE", // PRUNE_LEGACY rewrite_polyfills: use_polyfill || false, //isolation_mode: "IIFE", @@ -141,21 +112,26 @@ let parameter = (function(opt){ //formatting: "PRETTY_PRINT" }); -// if(options["DEBUG"]){ -// parameter += ' --formatting=PRETTY_PRINT'; -// } - -if(release !== "bundle.module" && release !== "light.module"){ - //parameter += ' --isolation_mode=IIFE'; - parameter += ' --emit_use_strict=true'; - parameter += ' --output_wrapper="\"(function(self){%output%}(this));\""'; +if(options["DEBUG"]){ + parameter += ' --formatting=PRETTY_PRINT'; } -const custom = (!release || (release === "custom")); +if(!release.endsWith(".module")){ + //parameter += ' --isolation_mode=IIFE'; + parameter += ' --output_wrapper=\"(function(self){%output%}(this));\"'; + parameter += ' --emit_use_strict=true'; +} + +// if(language_out === "ECMASCRIPT5_STRICT"){ +// parameter += " --js='!tmp/keystore.js'"; +// } + +const custom = (!release || release.startsWith("custom")) + && hashCode(parameter + flag_str).replace(/[^a-zA-Z0-9]/g, "").toLowerCase(); if(custom){ - release = "custom." + hashCode(parameter + flag_str).replace(/[^a-zA-Z0-9]/g, "").toLowerCase(); + release || (options["RELEASE"] = release = "custom"); } // if(release === "lang"){ @@ -214,93 +190,163 @@ if(custom){ // } // else{ - if(release === "lang") throw new Error("disabled"); +if(release === "lang") throw new Error("disabled"); + +(async function(){ + + const files = await fs.promises.readdir("./src/"); + + // const files = [ + // "async.js", + // "cache.js", + // "common.js", + // "compress.js", + // "config.js", + // "document.js", + // "encoder.js", + // "engine.js", + // "global.js", + // "index.js", + // "intersect.js", + // "keystore.js", + // "lang.js", + // "polyfill.js", + // "preset.js", + // "resolver.js", + // "serialize.js", + // "type.js", + // "webpack.js" + // ]; + + files.forEach(function(file){ + + if(file === "config.js"){ + + let src = String(fs.readFileSync("src/" + file)); + + if(custom){ + + let defaults = src.split(/export const /); + defaults.unshift(); + defaults = defaults.filter(str => /^(SUPPORT|RELEASE)/.test(str)).map(str => str.replace(/=[\s\S]+/, "").trim()); + + for(let i = 0, opt; i < defaults.length; i++){ + + opt = defaults[i]; + options[opt] = typeof options[opt] === "undefined" ? false : options[opt]; + } + } + + for(let opt in options){ + + src = src.replace(new RegExp('(export const ' + opt + ' = )(")?[^";]+(")?;'), "$1$2" + options[opt] + "$3;"); + } + + fs.writeFileSync("tmp/" + file, src); + } + else{ + + if(file.endsWith(".js")){ + if(language_out === "ECMASCRIPT5_STRICT" && file === "keystore.js"){ + let content = fs.readFileSync("src/" + file, "utf8"); + + content = content.substring(0, content.indexOf("function lcg64")); + content += "function lcg64(str){ throw new Error('The keystore is limited to 32 for EcmaScript5'); }"; + fs.writeFileSync("tmp/keystore.js", + content + // "/** @constructor */ export function KeystoreMap(arg){};" + + // "/** @constructor */ export function KeystoreSet(arg){};" + + // "/** @constructor */ export function KeystoreArray(arg){}; KeystoreArray.prototype.push = function(arg){};" + ); + } + else{ + fs.copyFileSync("src/" + file, "tmp/" + file); + } + } + } + }); + + fs.existsSync("tmp/db/") || fs.mkdirSync("tmp/db/"); + fs.cpSync("src/lang/", "tmp/lang/", { recursive: true }); + fs.cpSync("src/worker/", "tmp/worker/", { recursive: true }); + fs.cpSync("src/db/indexeddb/", "tmp/db/indexeddb/", { recursive: true }); + fs.copyFileSync("src/db/interface.js", "tmp/db/interface.js"); + fs.cpSync("src/index/", "tmp/index/", { recursive: true }); + fs.cpSync("src/document/", "tmp/document/", { recursive: true }); + fs.cpSync("src/resolve/", "tmp/resolve/", { recursive: true }); + + const filename = "dist/flexsearch." + (release + (custom ? "." + custom : "")) + (options["DEBUG"] ? ".debug" : ".min") + ".js"; + const executable = process.platform === "win32" ? "\"node_modules/google-closure-compiler-windows/compiler.exe\"" : + process.platform === "darwin" ? "\"node_modules/google-closure-compiler-osx/compiler\"" : + "java -jar node_modules/google-closure-compiler-java/compiler.jar"; + + exec(executable + parameter + " --js='tmp/**.js' --js='!tmp/**/node.js'" + flag_str + " --js_output_file='" + filename + "' && exit 0", function(){ + + let build = fs.readFileSync(filename); + let preserve = fs.readFileSync("src/index.js", "utf8"); + + const package_json = require("../package.json"); + + let name = ( + custom ? release.replace("custom.", "Custom/") : + light_version ? "Light" + (release === "light.module" ? "/Module" : "") : + es5_version ? "ES5" : "Bundle" + (release === "bundle.module" ? "/Module" : "") + ); + + if(custom) name += "/" + custom; + if(options["DEBUG"]) name += "/Debug"; + + preserve = preserve.replace("* FlexSearch.js", "* FlexSearch.js v" + package_json.version + " (" + name + ")" ); + build = preserve.substring(0, preserve.indexOf('*/') + 2) + "\n" + build; -const files = [ + if(release === "bundle.module" || + release === "light.module" || + release === "compact.module" || + release === "custom.module"){ - "async.js", - "cache.js", - "common.js", - "config.js", - "document.js", - "engine.js", - "global.js", - "index.js", - "intersect.js", - "lang.js", - "polyfill.js", - "preset.js", - "serialize.js", - "type.js", - "webpack.js" -]; + // export default { + // Index: O, + // Encoder: H, + // Charset: M, + // Language: ma, + // Document: Y, + // Worker: W, + // Resolver: null, + // IndexedDB: null + // }; -files.forEach(function(file){ + const pos_start = build.indexOf("window.FlexSearch"); + const pos_end = build.indexOf("};", pos_start) + 2; - if(file === "config.js"){ + let part = build.substring(build.indexOf("{", pos_start) + 1, pos_end - 2); + part = part.split(","); + part = part.map(entry => "export const " + entry.replace(":", "=")); + part = part.join(";") + ";"; + //console.log(build.substring(pos_start - 50, pos_start) + part + build.substring(pos_end)) - let src = String(fs.readFileSync("src/" + file)); - - for(let opt in options){ - - src = src.replace(new RegExp('(export const ' + opt + ' = )(")?[^";]+(")?;'), "$1$2" + options[opt] + "$3;"); + //build = build.substring(0, pos_start) + part + build.substring(pos_end); + build = build.replace(/window\.FlexSearch(\s+)?=(\s+)?/, "export default ") + part; + //build = build.replace(/self\.FlexSearch(\s+)?=(\s+)?/, "export default "); } - fs.writeFileSync("tmp/" + file, src); - } - else{ + // fix closure compiler dynamic import + build = build.replace(/\(([a-z])=([a-z]).config\)&&\(([a-z])=([a-z])\)/, "($1=$2.config)&&($3=await import($4))"); - fs.copyFileSync("src/" + file, "tmp/" + file); - } -}); + if(release === "bundle"){ + build = build.replace("(function(self){'use strict';", "(function _f(self){'use strict';try{if(module)self=module}catch(e){}self._factory=_f;"); + } -fs.cpSync("src/lang/", "tmp/lang/", { recursive: true }); -fs.cpSync("src/worker/", "tmp/worker/", { recursive: true }); - -const filename = "dist/flexsearch." + (release || "custom") + (options["DEBUG"] ? ".debug" : ".min") + ".js"; - -const executable = process.platform === "win32" ? "\"node_modules/google-closure-compiler-windows/compiler.exe\"" : - process.platform === "darwin" ? "\"node_modules/google-closure-compiler-osx/compiler\"" : - "java -jar node_modules/google-closure-compiler-java/compiler.jar"; - -exec(executable + parameter + " --js='tmp/**.js' --js='!tmp/**/node.js'" + flag_str + " --js_output_file='" + filename + "' && exit 0", function(){ - - let build = fs.readFileSync(filename); - let preserve = fs.readFileSync("src/index.js", "utf8"); - - const package_json = require("../package.json"); - - preserve = preserve.replace("* FlexSearch.js", "* FlexSearch.js v" + package_json.version + (release ? " (" + (release.charAt(0).toUpperCase() + release.substring(1)) + ")" : "")); - build = preserve.substring(0, preserve.indexOf('*/') + 2) + "\n" + build; - - if(release === "bundle"){ - - build = build.replace("(function(self){'use strict';", "(function _f(self){'use strict';try{if(module)self=module}catch(e){}self._factory=_f;"); - } - - build = build.replace(/eval\('(.*)'\)/, "$1"); - - if(release === "bundle.module" || release === "light.module" || release === "compact.module"){ - - build = build.replace(/self\.FlexSearch(\s+)?=(\s+)?/, "export default "); - } - - // if(release === "pre"){ - // - // fs.existsSync("test/dist") || fs.mkdirSync("test/dist"); - // fs.writeFileSync("test/" + filename, build); - // } - // else{ + // replace the eval wrapper + build = build.replace(/\(0,eval\)\('([^']+)'\)/, "$1"); fs.writeFileSync(filename, build); - // } + fs.copyFileSync("src/worker/node.js", "dist/node/node.js"); - fs.copyFileSync("src/worker/node.js", "dist/node/node.js"); - - console.log("Build Complete."); -}); -//} + console.log("Saved to " + filename); + console.log("Build Complete."); + }); +}()); function hashCode(str) {