Data & Databases
Apache Solr: Inverted Indexes, Analysis Chains and Relevance Tuning
Intermediate7 min readUpdated
A search engine answers a different question from a database: not which rows match, but which documents are most relevant. This guide explains the inverted index and the analysis chain that feeds it, then covers Solr's schema, indexing and query parsers, BM25 scoring, faceting and highlighting, SolrCloud collections, and the practical work of tuning relevance.
What you will learn
- Explain why an inverted index answers text queries that a relational index cannot
- Design an analysis chain from tokenisers and filters and know why index and query analysis must agree
- Define Solr field types, index documents and query them with edismax
- Use faceting, highlighting and suggestions to build a real search experience
- Tune relevance with field boosts, phrase boosts and function queries, and run Solr as a cluster
The first time a team replaces a LIKE '%term%' query with a search engine, the surprising part is not the speed. It is that the results come back in a useful order. A database answers a boolean question — does this row match — and any ranking has to be invented by you. A search engine answers a graded one: of the documents that match, which best satisfy the query, given how rare each term is and how much of the document it fills.
Apache Solr is a search server built on Apache Lucene, and it has been running catalogue, site and document search for two decades. This guide covers Solr 9: what the index actually contains, how text becomes tokens, how to define a schema and query it, and how relevance tuning works once the basics run.
Why a search engine is not a database index
A B-tree index maps a whole column value to the rows containing it. A search engine builds an inverted index: a dictionary of terms, each pointing at a posting list of the documents containing that term, with positions and frequencies.
Doc 1: "Estuary birds of northern Portugal"
Doc 3: "Bird migration over the estuary"
term postings (doc: positions, freq)
----------------------------------------------
bird 1:[2]/1 3:[1]/1
estuari 1:[1]/1 3:[5]/1
migrat 3:[2]/1
portug 1:[5]/1
Two things follow. Searching for a word is a dictionary lookup and a list merge, so it stays fast whether the corpus holds ten thousand documents or ten million. And because the index records how often and where each term appears, it can score matches rather than merely find them.
The default scoring model is BM25. In plain terms, a document scores higher when the query term appears more often in it (with strongly diminishing returns), when the term is rare across the whole corpus, and when the field containing it is short — a term in a five-word title counts for more than the same term buried in a thousand-word body. Lucene has used BM25 as its default similarity since Solr 6; the older TF-IDF model is still available for compatibility.
The trade-off is real. Solr has no transactions, no joins worth the name and no update-in-place; changing a single field rewrites the whole document. Treat it as a derived projection of a system of record, never as the record itself — the relational engine keeps that job, with the design and index habits covered in the execution plan guide.
The analysis chain
Raw text becomes index terms through an analyser, which is a character filter chain, one tokeniser, and a chain of token filters. Getting this right is most of the work of a good search experience.
- Character filters run on the raw string: strip HTML, map
œtooe, normalise punctuation. - The tokeniser splits the stream into tokens.
StandardTokenizerfollows Unicode word boundaries;WhitespaceTokenizersplits only on spaces;ClassicTokenizerpreserves things like product codes and hostnames. - Token filters transform the stream: lowercase, remove stop words, apply synonyms, fold accents (
ASCIIFoldingFilter), splitWiFi-2.4GHzinto pieces (WordDelimiterGraphFilter), reduce words to stems (SnowballPorterFilterfor many languages, or the more conservativeKStemFilterfor English).
The rule that causes the most confusion: index-time and query-time analysis must be compatible. If the index stems birds to bird but the query does not, nothing matches. Solr lets you declare the two chains separately for good reason — synonym expansion and EdgeNGram generation are commonly applied only at index time, so that the query stays a single term. The admin UI's Analysis screen shows both token streams side by side and answers "why does this not match" faster than any amount of reasoning.
Schema and field types
A Solr schema declares field types (an analysis chain plus storage options) and the fields that use them. Solr 9 uses a managed schema editable through the Schema API, though the XML form makes the structure clearest:
<fieldType name="text_en" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.SynonymGraphFilterFactory" synonyms="synonyms.txt" expand="true"/>
<filter class="solr.KStemFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.KStemFilterFactory"/>
</analyzer>
</fieldType>
<field name="id" type="string" indexed="true" stored="true" required="true"/>
<field name="title" type="text_en" indexed="true" stored="true"/>
<field name="body" type="text_en" indexed="true" stored="false"/>
<field name="category" type="string" indexed="true" stored="true" docValues="true"/>
<field name="price" type="pdouble" indexed="true" stored="true" docValues="true"/>
<field name="published" type="pdate" indexed="true" stored="true" docValues="true"/>
Four attributes carry the meaning. indexed makes a field searchable. stored makes its original value returnable. docValues builds a column-oriented structure used for sorting, faceting and function queries — enable it on anything you facet or sort on, and you can then leave stored off for large text you never display. And the string type is not analysed: it matches exactly, which is precisely what you want for identifiers, category codes and facet values, and precisely what you do not want for prose.
Fields you filter on structurally should be string; fields humans type words into should be a TextField. Getting that distinction wrong is the most common schema mistake.
Indexing and querying
Documents are posted as JSON over HTTP, individually or in batches. Batch, always — per-document commits are the classic cause of a slow indexer.
curl -X POST 'http://localhost:8983/solr/catalogue/update?commitWithin=10000' \
-H 'Content-Type: application/json' --data-binary '[
{ "id": "BK-1042", "title": "Field Guide to Estuaries",
"body": "A practical guide to birds, tides and salt marshes.",
"category": "reference", "price": 24.90, "published": "2026-02-11T00:00:00Z" }
]'
Use commitWithin (or the autoCommit settings in solrconfig.xml) rather than committing explicitly: a hard commit flushes segments to disk, a soft commit makes documents visible to searchers, and issuing either per document destroys throughput. Near-real-time search means a soft commit every second or two, not on every write.
Queries go to the select handler. The edismax parser is the right default for user-typed input: it never throws a syntax error on stray punctuation, and it spreads a query across several fields with independent weights.
GET /solr/catalogue/select?defType=edismax
&q=estuary birds
&qf=title^4 body^1 category^2
&pf=title^8
&mm=2<-1
&fq=category:reference
&fq=price:[* TO 30]
&sort=score desc, published desc
&rows=20&fl=id,title,price,score
Each parameter earns its place. qf lists the fields to search and their boosts. pf boosts documents where the query terms appear as a phrase, which is how "estuary birds" beats a document mentioning both words in unrelated paragraphs. mm (minimum should match) controls how many terms must be present — 2<-1 means "with two terms require both, beyond that allow one to be missing". fq filter queries are boolean, do not affect the score, and are cached separately, so structural constraints belong there rather than in q. Adding debugQuery=true returns the parsed query and a full score explanation per document; it is the single most useful diagnostic Solr offers.
Faceting, highlighting and suggestions
A search box alone is not a search experience. Three features turn results into navigation:
Faceting counts matching documents per value, powering the filter sidebar. The JSON Facet API handles nesting and statistics that the older parameters cannot:
{
"query": "estuary birds",
"filter": ["category:reference"],
"facet": {
"by_category": { "type": "terms", "field": "category", "limit": 10,
"facet": { "avg_price": "avg(price)" } },
"price_bands": { "type": "range", "field": "price", "start": 0, "end": 100, "gap": 20 }
}
}
Highlighting returns snippets with the matched terms marked, so users can see why a result matched. The unified highlighter is the current default and works best with storeOffsetsWithPositions enabled on the field.
Suggestions and spelling. The suggester component provides type-ahead from a dictionary built off a field; the spellcheck component offers "did you mean" corrections from the index's own terms. Both need to be configured in solrconfig.xml and rebuilt as content changes.
Running Solr as a cluster
SolrCloud is Solr's distributed mode. A collection is a logical index split into shards; each shard has one or more replicas living on different nodes; one replica per shard is elected leader and receives indexing traffic. Cluster state and configuration live in Apache ZooKeeper, which also handles leader election. Configuration is uploaded as a configset, so every node runs the same schema and solrconfig.xml.
# Upload a configset, then create a sharded, replicated collection
solr zk upconfig -n catalogue -d /opt/configsets/catalogue -z zk1:2181
curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=catalogue&numShards=2&replicationFactor=2&collection.configName=catalogue'
Sizing follows a few reliable rules. Shard for index size and indexing throughput, replicate for query throughput and availability; a query fans out to every shard and merges results, so more shards than you need adds latency rather than removing it. Keep the JVM heap moderate and leave plenty of RAM to the operating system page cache, because Lucene relies on memory-mapped files. Running a single-node Solr in a container is fine for development — see Docker fundamentals — but production wants at least three ZooKeeper nodes.
Tuning relevance
Relevance work is empirical. Start from a set of real queries with agreed-good results, change one thing, and re-measure.
- Field boosts (
qf) encode that a match in the title means more than one in the body. Keep the ratios modest; extreme boosts make everything else irrelevant. - Phrase boosts (
pf,pf2,pf3) reward proximity, which is usually the highest-value single change. - Function queries fold non-textual signals into the score:
boost=recip(ms(NOW,published),3.16e-11,1,1)favours fresh documents,bf=sqrt(popularity)favours popular ones. Prefer multiplicativeboostover additivebfso a strong signal cannot swamp the text score. - Synonyms and stop words fix vocabulary mismatch between how you write and how users search. Maintain the synonym file as a product asset — it is where domain knowledge lives.
- Learning to rank reranks the top N results with a trained model. Worth it only once the basics are exhausted and you have click data.
Solr is not the right tool for every search-shaped problem. High-volume machine-generated data with time-based retention belongs in a log platform — see centralised log management with the Elastic stack — and simple keyword lookup inside documents you already store may be satisfied by MongoDB's text indexes, covered in the MongoDB guide.
Key takeaways
- An inverted index maps terms to posting lists with frequencies and positions, which is what makes ranked retrieval possible; BM25 rewards term frequency, term rarity and short fields.
- The analysis chain (character filters, tokeniser, token filters) defines what is searchable, and index-time and query-time chains must produce compatible tokens.
- Use
stringfor exact structural values and aTextFieldfor prose; enabledocValuesfor anything you sort, facet or use in a function. - Batch indexing with
commitWithinand soft commits; per-document hard commits are the standard performance mistake. - Query user input with edismax, put structural constraints in cached
fqfilters, and diagnose scoring withdebugQuery=true. - Tune relevance with field and phrase boosts and multiplicative function queries, measured against a fixed set of judged queries rather than by intuition.
Sources and further reading
Frequently asked questions
- Can I just use my database's full-text search instead of Solr?
- For a search box over a few hundred thousand rows with modest relevance needs, PostgreSQL full-text search or MongoDB text indexes are perfectly adequate and save you a service to operate. Move to Solr when you need configurable analysis per language, faceted navigation, highlighting, suggestions, relevance tuning with field and phrase boosts, or query volumes that would disturb your transactional database.
- Should I use Solr as my primary data store?
- No. Solr has no transactions, no foreign keys and no update-in-place — an update rewrites the whole document — and reindexing after a schema change is a routine operation that assumes the data exists elsewhere. Keep the system of record in a database and treat the Solr index as a derived, rebuildable projection of it.
- Why does my exact phrase not match anything?
- Almost always an analysis mismatch. The index-time and query-time analysis chains must produce comparable tokens, so a stemmer, a lowercase filter or a synonym filter applied on only one side will silently break matching. Open the Analysis screen in the Solr admin UI, paste the indexed value and the query, and compare the token streams step by step.
- How is Solr different from Elasticsearch?
- Both are built on Apache Lucene and share the same index structure, analysis concepts and BM25 scoring, so search behaviour is broadly comparable. They differ in APIs, clustering (ZooKeeper-based SolrCloud versus a built-in coordination layer), licensing and ecosystem. Solr remains Apache-licensed and is strong in classic site and catalogue search; Elasticsearch dominates log and observability workloads.