Adapting Qwen3-VL Representations for Open-Set Continual Visual Learning

Continual learning is usually described as the ability to acquire new knowledge without erasing old knowledge. I wanted to test a more specific version of this problem: whether the internal visual representations of a general vision-language model could be turned into a memory for concepts it had only just encountered.

Faces were included as the hardest case alongside natural objects. Broad visual categories reward invariance: many different images should resolve to “dog” or “car”. Identity requires the opposite property in the right places. A useful representation must ignore changes in lighting and pose while preserving the small differences between visually similar people.

The investigation developed through four distinct stages. I first tested Qwen's raw hidden states, then applied several training-free normalisation and alignment methods, repeated the apparent best result on a disjoint corpus, and finally trained continual concept learners over preserved Qwen evidence. Compressing these into one final score would miss the most informative part of the experiment: each stage overturned the most plausible interpretation of the stage before it.

Stage One: Is There Already a Metric Space Inside Qwen?

I began by extracting representations from several depths of Qwen3-VL-4B and Qwen3.5-4B. I compared text descriptions with their corresponding video events, using a purpose-trained Qwen3-VL-Embedding-2B model as an offline teacher and control.

The difference was immediate. Across the tested general-model surfaces, mean rank-one retrieval remained between 0.055 and 0.070, close to the random baseline of 0.0625. The embedding teacher reached 0.805.

The geometry explained part of this gap. Off-diagonal cosine similarities between different items were approximately 0.988–0.999 in the general Qwen representations. Almost everything pointed in nearly the same direction. For the embedding teacher, the range was 0.483–0.716, leaving substantially more room to distinguish individual events.

This did not mean that Qwen lacked the visual information. It meant that an arbitrary hidden state was not organised for nearest-neighbour retrieval. The general model was trained to expose information to later attention and language layers; the teacher had received contrastive training and reranker distillation specifically to produce a retrieval space.

That distinction led to the next question. If the information existed but the geometry was distorted, could I repair the space without training another model?

Stage Two: Whitening Appeared to Recover the Space

I screened first-moment and second-moment corrections on the frozen features. Mean-centring did almost nothing. Fifteen-pair Procrustes and ridge alignment were worse: the held-out item was systematically ranked last because the fitted map captured the shared subspace while failing on the unseen residual direction.

Leave-one-out shrinkage ZCA whitening produced a very different result. For separate text and video means, I pooled the centred background observations into a covariance estimate, shrank it towards an isotropic covariance and applied its inverse square root:

$$\begin{aligned} \Sigma &= \frac{1}{2N}C^{\mathsf T}C, \qquad C = \begin{bmatrix} X_t-\mu_t \\ X_v-\mu_v \end{bmatrix}, \\ \widehat{\Sigma}_{\lambda} &= (1-\lambda)\Sigma + \lambda\frac{\operatorname{tr}(\Sigma)}{d}I, \\ z_m(x) &= \operatorname{norm}\!\left((x-\mu_m)\widehat{\Sigma}_{\lambda}^{-1/2}\right), \quad m\in\{t,v\}. \end{aligned}$$

This equation belongs in the account because it describes the intervention being tested. The method suppresses high-variance shared directions while amplifying residual directions, using different modality means but one shared covariance transform.

With (\lambda=0.1), rank-one retrieval rose to 0.438–0.625 across all six general Qwen surfaces. The embedding teacher reached 1.000 under the same leave-one-out treatment. The general-model result was seven to ten times chance, required no gradient updates and reproduced under an independent implementation.

At this point, whitening looked like the important result. It was also misleading.

Stage Three: The Properly Disjoint Test Broke the Result

The leave-one-out procedure estimated its transform from the same small event distribution it evaluated. It omitted the current pair, but every neighbouring event still came from the evaluation set. A deployable system would need to fit its transform on past background data and apply it unchanged to genuinely new events.

I therefore constructed three source-video-disjoint partitions: 256 background events from 128 worlds for fitting, 64 events from 32 fresh worlds for retrieval, and 32 events from 16 further worlds for unknown rejection. I froze the background transform before applying it to either evaluation set.

The apparent recovery disappeared:

Representation	Raw R@1	Fixed-background whitening R@1
Qwen3-VL-4B	0.023	0.035
Qwen3.5-4B	0.016	0.020
Qwen3-VL-Embedding-2B teacher	0.758	0.750

The best general-model result remained more than 21 times below the teacher. Open-set performance was worse: Qwen3-VL produced an AUROC of 0.445 and accepted every known and unknown event at every frozen operating point.

I also tested different shrinkage values, background-bank sizes and compositions, asymmetric whitening, CSLS and Sinkhorn-based normalisation, alongside Procrustes and world-grouped ridge maps. None materially rescued the fixed-background result. Bank composition had a non-monotonic effect, and whitening even reduced the separate SigLIP control from 0.512 to 0.391.

The contrast between the two whitening studies was not noise to average away. It identified the failure. Leave-one-out normalisation could exploit local evaluation geometry that did not transfer across worlds. A representation could also become more isotropic without becoming more useful for retrieval or unknown rejection.

Stage Four: Learning a Concept Programme Instead

The whitening result changed the problem. Rather than continue trying to make one global Qwen vector behave like a universal embedding, I trained learners to compare a small support set with a new query using preserved token and region evidence.

The study used person and natural-object families, one-, two-, four- and eight-shot support sets, three frozen seeds and held-out concepts. I compared four approaches:

Arm	Method	Sample-efficiency AUC
Q0	Training-free global prototype	0.819
Q1	Learned shared support/query comparator	0.794
Q2	Typed fixed-basis concept compiler	0.804
Q3	Gradient-fitted per-concept head	0.736

Q2 was the most distinctive part of the design. It did not generate an unrestricted classifier. From the support examples, it emitted a bounded and inspectable concept programme containing token/region gates, a centroid, variance, calibration terms and coefficients over a shared eight-element basis. In simplified form:

$$\begin{aligned} q_c &= g_{c,t}\,t_q + g_{c,r}\,r_q, \\ D_c &= 1 + 0.25\tanh(\alpha_c^{\mathsf T}B), \\ \widetilde{\mu}_c &= D_c\odot\mu_c, \\ s_c(q) &= \operatorname{softplus}(\ell_c) \left(w^{\mathsf T}\phi(q_c,\widetilde{\mu}_c,\sigma_c^2)+b_c\right). \end{aligned}$$

Here, the learnt gates choose how much token and region evidence to use, while (\alpha_c) composes a concept-specific diagonal transformation from the shared basis (B). The programme remained below 64 KiB and could be reconstructed from the canonical support evidence. The equation shows the actual constrained adaptation mechanism rather than restating a standard similarity measure.

The purpose-trained embedding model established the teacher/control gap that motivated this stage, but I did not train Q1 or Q2 to reproduce its coordinates. Their objective was concept discrimination over Qwen evidence. This matters because a successful imitation of the teacher's vectors would only prove coordinate reconstruction, not continual concept learning.

Despite the additional structure, Q2 improved on Q1 by only 0.010, below the frozen minimum of 0.020. Its 95% confidence interval was ([-0.033,\ 0.052]). It reduced false accepts, but regressed on retention at two of the three registered checkpoints. Most importantly, every learned arm lost to Q0. The per-concept optimisation arm was the worst, consistent with overfitting when only a handful of support examples are available.

Retention Was a Separate Failure

I also tested whether a causal admission rule could decide which experiences deserved storage. Retaining everything used 520,192 bytes and preserved all useful-concept recall. The proposed policy reduced storage by 54%, but useful recall fell to 0.477. A novelty-and-uncertainty policy retained even less and recovered only 0.174.

This changed my interpretation of memory efficiency. Discarding half of the future-useful evidence is not successful compression. In this regime, retaining the evidence and compressing its representation is more defensible than attempting to predict its future value at write time.

What This Means for Venus

Continual learning and test-time training are research directions within Project Venus, not current Venus capabilities. This experiment does not validate continual learning for Venus; it defines several requirements for testing it properly.

The business analogue is a model encountering a newly approved definition, policy or decision pattern. Learning the new rule is only half of the problem. The system must retain the previous approved meaning where it still applies, distinguish changed authority from new evidence, reject unsupported generalisation and make the update reversible.

I would carry four requirements into a Venus experiment: compare against a simple non-parametric baseline; use a representation trained or selected for the required distinction; score retention and rejection after every update; and keep attributable evidence outside the mutable weights so an unsuccessful update can be rolled back.

The main result is therefore not that Qwen cannot support continual learning. It is narrower and more useful. Normalisation can manufacture a convincing local recovery that fails across disjoint data, a strong teacher does not automatically define the correct student objective, and a more elaborate learner does not compensate for an unsuitable evidence surface. Continual learning begins by proving that the representation, update rule and memory policy each transfer beyond the examples that made them look promising.