Swapping OpenAI Embeddings for a Local Qwen3 Model

Introduction

I run a local-first knowledge base over my own notes. Markdown and PDFs in a folder get chunked, embedded, and stored in PostgreSQL with pgvector, and semantic search and a RAG chat sit on top. “Local-first” was true for the files and the database. It was not true for the embeddings: every chunk went to OpenAI’s text-embedding-3-small, through OpenRouter, with a pseudonymization layer replacing names and emails before the text left the machine.

That layer exists because the notes are personal. Meeting notes, a diary, contact details. Pseudonymizing them is a mitigation, not a solution. The real fix is an embedding model that runs on the laptop, and with LM Studio serving Qwen3-Embedding-8B over an OpenAI-compatible endpoint, that is now a realistic option on an Apple Silicon machine.

Three questions decided how much work it would be:

Qwen3-Embedding-8B produces 4096-dimensional vectors. The store is a 1536-wide table. Do I need a second table?
Can both models coexist, so I can switch back if the local one turns out worse?
Which parts of the OpenAI pipeline stop making sense once nothing leaves the machine?

The Dimension Problem

The existing table is content_chunk_embeddings_1536, with an HNSW index on a vector(1536) column. The obvious move is a content_chunk_embeddings_4096 sibling. That runs into pgvector directly: the vector type cannot be indexed above 2000 dimensions. A 4096-wide column means either halfvec with its precision trade-off or no index at all, and either way a second table, a second set of queries, and a second index to maintain.

The OpenAI embeddings API accepts a dimensions field, and LM Studio’s endpoint is OpenAI-compatible, so the easy route would be to ask for 1536 and let the server shorten the vector. LM Studio (version 0.4.16 at the time of writing) ignores that field and returns all 4096 values regardless, so the shortening has to happen on the client. There is an open issue for it, so a later release may honour the field.

What makes shortening legitimate at all is that Qwen3-Embedding is trained with Matryoshka Representation Learning. The model packs the most important information into the leading dimensions, so any prefix of the vector is a valid, if slightly coarser, embedding. Take the first 1536 values, re-normalise to unit length so cosine distance stays meaningful, and the result fits the table that already exists.

private fun List<Double>.truncateAndNormalize(): List<Double> {
    require(size >= targetDimensions) {
        "$modelName returned $size dimensions, expected at least $targetDimensions"
    }
    val head = take(targetDimensions)
    val norm = sqrt(head.sumOf { it * it })
    return if (norm == 0.0) head else head.map { it / norm }
}

The require is there on purpose. If someone loads a different model in LM Studio, the size check fails loudly instead of storing a short vector that pgvector would reject anyway, with a less helpful message.

Two Models, One Table

The table already had a model_id column referencing an embedding_models row, from a time when I thought I might swap models and then never did. That column is what made coexistence cheap. The migration is one insert:

INSERT INTO embedding_models (provider, name, modality, dimensions, max_tokens, distance_metric, normalized)
VALUES ('lmstudio', 'text-embedding-qwen3-embedding-8b', 'text', 1536, 32768, 'cosine', true)
ON CONFLICT (provider, name, modality, dimensions) DO NOTHING;

Note that dimensions records what is stored, not what the model emits. The chunks themselves are shared. A chunk can have an OpenAI vector, a Qwen vector, both, or neither.

The one rule that must hold everywhere: a query embedded by one model is never compared against chunks embedded by the other. The two vector spaces have nothing to do with each other, and a 1536-dimensional OpenAI vector next to a truncated Qwen vector will happily produce a cosine similarity that means nothing. So every search, every note centroid used by the knowledge graph, and every nearest-neighbour query filters on the active model’s id. The repository resolves that id per call from the setting in Postgres, and the vectorizer does the same on the way in:

class SwitchableVectorizer(
    private val activeModel: () -> EmbeddingModel,
    private val byModel: Map<EmbeddingModel, Vectorizer>,
) : Vectorizer {
    fun forModel(model: EmbeddingModel): Vectorizer =
        byModel[model] ?: error("No vectorizer configured for ${model.code}")

    override suspend fun generateQueryEmbedding(query: String): List<Double> =
        forModel(currentModel()).generateQueryEmbedding(query)

    override suspend fun generateEmbeddings(chunks: List<TextChunk>): List<VectorizedContent> =
        forModel(currentModel()).generateEmbeddings(chunks)
}

Switching in Settings is therefore instant and re-indexes nothing. What it does change is coverage. Ingestion only embeds with the active model, so after a week on Qwen the OpenAI side has drifted: new and edited notes have no OpenAI vector. A backfill use case pages through chunks that lack a vector for a given model, keyed on chunk id, and fills the gaps. Settings shows the per-model coverage next to an “Embed missing chunks” button. Switching back is: press the button, wait, switch.

What Changes When Nothing Leaves the Machine

Three things fell away or changed shape in the local vectorizer.

Pseudonymization is gone. The OpenAI vectorizer takes a SensitiveDataPseudonymizer and runs every chunk and every query through it before the HTTP call. The LM Studio vectorizer has no such parameter. The text goes to localhost:1234 and stays there. This is the whole reason for the exercise, and it is worth saying plainly: with Qwen for embeddings and a local model for chat, the app has no outbound network traffic at all.

Queries get an instruction prefix, documents do not. Qwen3-Embedding is trained asymmetrically. A search query is embedded with a task instruction in front of it, and documents are embedded bare. Skip the prefix and retrieval quality drops, quietly.

const val QUERY_INSTRUCTION =
    "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "

override suspend fun generateQueryEmbedding(query: String): List<Double> =
    embed(listOf(QUERY_INSTRUCTION + query)).single()

OpenAI’s model has no equivalent, so this was a genuinely new concept in the pipeline, and the Vectorizer interface with its separate generateQueryEmbedding and generateEmbeddings methods turned out to be exactly the right seam for it.

Batching got smaller. The OpenAI vectorizer sends a whole document’s chunks in one request, within a 280k token budget. A local 8B model on a laptop does not appreciate that. The LM Studio vectorizer sends batches of 16 and sorts the response by index, because the endpoint does not promise order.

A Few Caveats Keep It Honest

I have not benchmarked the two against each other. There is no golden query set for this vault. Search results with Qwen feel comparable in daily use, which is a sentence that should convince nobody. Truncating 4096 to 1536 also costs something; the Matryoshka guarantee is that the prefix is usable, not that it is free.

The 8B model needs a serious machine. It occupies many gigabytes of memory while loaded, and a full re-index of a large vault is slower than the hosted model by a wide margin. This works on a Mac with plenty of unified memory. It is not an option on a modest laptop, and a smaller Qwen3-Embedding variant would be the next thing to try.

LM Studio has to be running. When it is not, ingestion fails and the chunks stay unembedded until the backfill runs. The hosted model failed too, on rate limits and network, but a local server you forgot to start is a new way to be down.

The knowledge graph changes shape when you switch. Semantic edges are derived from document-level centroids in the active model’s space. Switch models and the graph is recomputed from different vectors. It is not wrong, but it is different, and a graph you have grown used to reading will need re-reading.

Conclusion

The local model went in without a new table, without a re-index, and without touching a single search query beyond the model_id filter that was already there. Three decisions carried the weight:

A model_id column on the embeddings table, added long before it was needed. Two integer columns for chunk offsets made retrieval-time expansion possible in an earlier post; one foreign key did the same here.
Matryoshka truncation instead of a wider table. Fit the model to the store rather than the store to the model, as long as the model was trained to allow it.
Query and document embedding as separate operations. Instruction-tuned models need that distinction, and an interface that already had it absorbed the change without a ripple.

The pseudonymizer is still in the codebase. It runs for the chat provider when that one is remote, and for the OpenAI embedding model if I switch back. For embeddings on the local model it does nothing, which is the outcome I wanted: a privacy mitigation that is no longer load-bearing.