public-inbox-extindex(1)

PUBLIC-INBOX-EXTINDEX(1)

public-inbox user manual

PUBLIC-INBOX-EXTINDEX(1)

NAME

public-inbox-extindex - create and update external search indices

SYNOPSIS

public-inbox-extindex [OPTIONS] EXTINDEX_DIR INBOX_DIR...

public-inbox-extindex [OPTIONS] [EXTINDEX_DIR] --all

DESCRIPTION

public-inbox-extindex creates and updates an external search and overview database used by the read-only public-inbox PSGI (HTTP), NNTP, and IMAP interfaces. This requires either the Xapian SWIG bindings OR or Search::Xapian XS bindings along with DBD::SQLite and DBI Perl modules.

OPTIONS

-j JOBS

--jobs=JOBS

--no-fsync

--dangerous

--rethread

--max-size SIZE

--batch-size SIZE

These switches behave as they do for public-inbox-index(1)

--all

Index all "publicinbox" entries in "PI_CONFIG".

"publicinbox" entries indexed by "public-inbox-extindex" can have full Xapian searching abilities with the per-"publicinbox" "indexlevel" set to "basic" and their respective Xapian ("xap15" or "xapian15") directories removed. For multiple public-inboxes where cross-posting is common, this allows significant space savings on Xapian indices.

--dedupe=MSGID

--dedupe

Rerun deduplication on messages with the given Message-ID or all messages if no Message-ID is specified. Deduplication rules may change and evolve over time, especially if filters are involved.

"--dedupe=MSGID" may be specified multiple times to deduplicate multiple Message-IDs.

Use this if you see "W: BUG? $MSGID not deduplicated properly" warnings from WWW logs.

--gc

Perform garbage collection instead of indexing. Use this if inboxes are removed from the extindex, a newsgroup name is set or changed, or if messages are purged or removed from some inboxes.

--reindex

Forces a re-index of all messages in the extindex. This can be used for in-place upgrades and bugfixes while read-only server processes are utilizing the index. Keep in mind this roughly doubles the size of the already-large Xapian database.

--fast

Used with "--reindex", it will only look for new and stale entries and not touch already-indexed messages.

FILES

public-inbox-extindex-format(5)

CONFIGURATION

public-inbox-extindex does not write to the public-inbox-config(5) file, it must be entered manually. The extindex name of "all" is a special case which corresponds to indexing "--all" inboxes. An example for "--all" is as follows:

        [extindex "all"]
                topdir = /path/to/extindex_dir
                url = all
                coderepo = foo
                coderepo = bar

Putting an "extindex" entry in the config allows PublicInbox::WWW. You can have any number of "extentry.$NAME" sections where $NAME is something other than "all" to display a union of several inboxes.

It is strongly recommended any public inboxes indexed by this command have a stable "publicinbox.$NAME.newsgroup" entry (regardless of the presence of an NNTP or IMAP server). Otherwise, public-inbox-extindex will use "publicinbox.$NAME.inboxdir" as an internal key which can cause needless reindexing and require --gc if inboxes are relocated.

See public-inbox-config(5) for more details.

ENVIRONMENT

PI_CONFIG

Used to override the default "~/.public-inbox/config" value.

XAPIAN_FLUSH_THRESHOLD

The number of documents to update before committing changes to disk. This environment is handled directly by Xapian, refer to Xapian API documentation for more details.

Setting "XAPIAN_FLUSH_THRESHOLD" or "publicinbox.indexBatchSize" for a large "--reindex" may cause public-inbox-mda(1), public-inbox-learn(1) and public-inbox-watch(1) tasks to wait long and unpredictable periods of time during "--reindex".

Default: none, uses "publicinbox.indexBatchSize"

UPGRADING

Occasionally, public-inbox will update its schema version and require a full index by running this command.

LOCKING

It is safe to use "--dedupe", "--gc" and "--reindex" while other processes are writing to covered inboxes or extindex. The extindex locks will be released roughly every 10s to allow public-inbox-mda(1) and public-inbox-watch(1) processes to write to the extindex.

CONTACT

Feedback welcome via plain-text mail to <mailto:meta@public-inbox.org>

The mail archives are hosted at <https://public-inbox.org/meta/> and <http://4uok3hntl7oi7b4uf4rtfwefqeexfzil2w6kgk2jn5z2f764irre7byd.onion/meta/>

COPYRIGHT

License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>