This PR merges the use of lunr search instead of the default mdbook search to enable support of JP characters.
1.3 KiB
1.3 KiB
elasticlunr.js (included in mdbook output by default)
- implements indexing and searching for English text
- index
- tokenizer: splits the text into tokens (i.e. words)
- pipeline: For each token:
- trimmer: removes prefix/suffix non-word characters
token.replace(/^\W+/, '').replace(/\W+$/, '')(which removes any Japanese characters) - stopWordFilter: remove common and non-meaningful words like "a", "we"
- stemmer: removes inflection e.g. both "applies" and "apply" become "appli"
- trimmer: removes prefix/suffix non-word characters
- builds a prefix tree of the tokens
- search
- applies the same process to the search query, and search the page that contains all/any of the resulting tokens (depending on the
use-boolean-andoption in book.toml)
- applies the same process to the search query, and search the page that contains all/any of the resulting tokens (depending on the
Plugins from https://github.com/weixsong/lunr-languages
lunr.jp.js
- provides stopWordFilter, stemmer for Japanese
- implements a custom tokenizer (which is unusual)
tinyseg.js
- used by lunr.jp.js
- segments Japanese text
lunr.multi.js
- add support for multiple languages
- combines pipeline functions from all specified languages, as well as constructs a new trimmer based on each lang's
wordCharacters(which lunr.jp.js doesn't have, so we added it)
lunr.stemmer.support.js
- provides
trimmerSupportandstemmerSupportfunctions trimmerSupportis used by lunr.multi.js