Skip to content
Discuss your first workflow

Project Venus · Research programme

Research specification · 30 October 2026

Venus Colony

A self-organising tree of agents for continuous database understanding.

George Pullen · Project Venus Research Programme

Abstract

New organisations break on their data before they break on the agent, and today the Venus runtime agent rediscovers each tenant on almost every question. We specify Venus Colony, a self-organising tree of agents that keeps an organisation's data continuously understood.

Each node owns a bounded slice of the data and is run by a main agent with three sub-agents, one each for its prompt cache, its memory and its records. A single rule governs growth: no node may exceed its limits on data, memory or KV cache, and exceeding one creates a sub-node. Information flows only upward, parents settle conflicts with evidence rather than votes, and each territory is private to its node. Records never leave the source tables. Code profiles, embeds, routes and checks every record, while agents read only what is new or surprising, so cost follows change rather than size.

We describe the architecture, its stability mechanisms, a three-stage gated pilot, and the eleven decisions that shape the specification.

One rule
No node may exceed its limits. Exceeding one creates a sub-node.
Flows up
Parents settle conflicts with evidence, never by vote.
Stays at source
Nodes hold record IDs and notes, never copies.
Follows change
Code checks every record. Agents read what is new or surprising.
The colony · an illustrative organisationRecords stay in the canonical tables
The colony. Select a node to see what it holds.

A new Xero invoice lands in raw_events and the canonical tables, then enters the colony at the root.

  • root (L3)
  • internal node (L2)
  • leaf (L1)
  • runtime agent
  • stored data

Explore

Select any node to see what it holds. Hover to trace its path from the root.

Figure 1The colony for an illustrative organisation. Code routes new and changed records down from the root to the leaf that owns them, and agents report findings back up. Depth varies by area: Clients is a leaf one level down, while Tours needed two more levels. Leaves hold pointers into the canonical tables, not copies.

1 · Introduction

Move understanding out of the answer path.

Connectors deliver partial backfills, renamed fields and values that mean something other than they appear to. The runtime agent discovers these mid-answer, in front of users, after spending an estimated 30–120 s per question listing tables, describing them and trying queries. Existing checks stop at logs: the volume check only logs too_low, and coverage drift only posts to Slack.

Venus Colony moves that understanding out of the answer path (Figure 1). Leaves own slices of records, internal nodes hold roll-ups of their children, and the root holds an always-on summary that every node inherits. When a question arrives, the runtime agent hands it to the node that already knows the relevant data. This document specifies the node and its limits (§3); one mechanism for ingestion and growth (§4); the mechanisms that keep the tree stable (§5); how information and authority flow (§6); and the evidence so far, a pilot plan and the open decisions (§7–8).

Today · discovery inside the answer

est. 30–120 s per question, in front of users

  1. Question
  2. List tables
  3. Describe them
  4. Try queries
  5. Answer

Venus Colony · understanding already built

code checks every record ·agents read what changed

  1. Question
  2. Hand to the node that knows
  3. Answer
understanding built beforehand
Figure 1aOut of the answer path. Today each question pays for discovery while the user waits. In the colony, understanding is built continuously beforehand, and the question goes straight to the node that holds it.

2 · Design principles

One tree, one rule, and information that flows up.

  1. P1One tree.Every node has exactly one parent, and depth varies by area.See §4 →
  2. P2One rule.No node may exceed its limits; exceeding one creates a sub-node.See §5 →
  3. P3Information flows up.No peer messages; parents settle conflicts with evidence, not votes.See §6 →
  4. P4Private territories.Only a node and its sub-agents see its state. People and the audit log can read everything.See §3 →
  5. P5Records stay at the source.Nodes hold record IDs and their own notes, never copies.See §3 →
  6. P6Code proposes, agents decide.Code clusters, routes and checks; agents name, split and judge.See §4 →
  7. P7Cost follows change.Agents read what is new or surprising, and code checks the rest.See §4 →
  8. P8Organisation-wide data only.Nothing user-scoped; a task thread only if all of it is org-visible.See §8 →

3 · The node

Every node, from the root to a single-record leaf, has the same structure.

Its main agent sets priorities, decides when to split and chooses what to report, making it the node’s regulator. The KV cache library keeps the node’s prompt as a prefix tree of a base prompt and overlays chosen per query type. The memory palace consolidates memory within the node’s limit, promoting, demoting or discarding items, and writes the roll-up its parent receives. The data representation sub-agent owns the node’s membership (record IDs and a routing rule). It keeps one note per record (embedding, checks passed, whether it has been read, which beliefs cite it) and writes beliefs with SQL checks.

parent
charter and tasks reports and roll-up
canonical tables
← read by ID

Node e.g. Invoices 2025

Limits

data
mem
KV

fills → split

Private state

  • membershipIDs and a rule
  • record notesone per record
  • beliefs+ SQL checks
  • memorybounded
  • promptand overlays
runtime agent
outside caller ⇢

trusted code: permissions · org-wide data only · limits · quotas

Main agent · regulator

Sets priorities, decides when to split and chooses what to report, making it the node's regulator. It watches the three limits.

  • agents
  • code
  • stored data
Figure 2Anatomy of a node. The accent marks agents, grey marks code and the faint wash marks stored data. The three gauges are the node’s limits, and filling any of them triggers a split (§5). Trusted code surrounds the node; no agent can change what it enforces.

Reports and privacy

A report carries the claim, the SQL that tests it and its result, the record versions it rests on (raw_event IDs), the node’s confidence and what depends on it. The parent keeps only what fits within its own limit, and its roll-up lists exactly which children it came from. When a newer version of a cited record arrives, every belief resting on it is re-checked.

Nothing inside a node is visible to other agents, and a parent sees only what it is told. People and the audit log can read everything, and every answer carries its SQL. Nodes propose logical views; physical database views are created only after approval.

Report · Invoices 2025 → Finance

Checked
Claim
Invoice INV-2025-0418 has two payments allocated to it.
SQL that tests it
select invoice_id, count(*) as payments
from invoice_payments
where invoice_id = 'INV-2025-0418'
group by invoice_id
having count(*) > 1;
Result
1 row · payments = 2
Rests on
raw_event re_7q2k · v3raw_event re_7q2m · v1
Confidence
0.86
Depends on it
Finance roll-up · ageing report overlay
  1. 1The leaf writes the belief with the SQL that tests it, and reports it to its parent.
  2. 2A newer version of a cited record arrives: the second payment was reversed.
  3. 3Every belief resting on that record is re-checked by running its SQL again.
  4. 4The claim no longer holds, so the belief is withdrawn and what depends on it is re-checked in turn.
Figure 2aA report, and its re-check. Illustrative. A report carries the claim, the SQL that tests it and its result, the record versions it rests on (raw_event IDs), the node's confidence and what depends on it. When a newer version of a cited record arrives, every belief resting on it is re-checked.

One request extends exactly one path. Everything before its own task tokens is already cached: the root summary for every node, the roll-up for the subtree, the node's own segment and the reconciliation overlay.

Figure 3A node's prompt is one path through a prefix tree, ordered most-stable first. Siblings share one cached copy of everything above their own segment. Overlays chain but never splice, so a frequent combination becomes its own branch or a child. Edits near the root invalidate every descendant's cache, so upper levels change rarely. New memory may lag the cache, but removals, corrections and permission changes invalidate it at once. The cache is rebuildable and never the source of truth.

4 · Ingestion and growth

One mechanism for the first load and every later change.

The colony grows like a B-tree[3] built by clustering. Sources land in raw_events and the canonical tables. Code profiles every record and embeds it with a small frozen encoder, so every record has been processed before any agent reads one. A single root then owns everything. It is over its limit, so code proposes clusters, and the root’s agent names them and writes each child’s charter. Children repeat this in parallel until every leaf fits. Top-level splits follow sources and tables, and deeper splits follow values such as client, period or type.

  1. 1 connectsources → raw_events
  2. 2 profile + embedevery record, in code
  3. 3 root splitsagent names clusters
  4. 4 recurseuntil every leaf fits
  5. 5 learn, then watchagents read the new
  6. 6 route changesdown from the root

1 · code

Sources land in raw_events and the canonical tables. One mechanism handles the first load and every later change, for a database of any size.

Figure 4aOne pipeline for the first load and every later change. Grey stations are code; accented stations involve agents.

A single root owns every record. It is far over its limit.

  • root (L3)
  • internal node (L2)
  • leaf (L1)
  • over the limit
Figure 4bRecursive split. The root splits until every node fits its capacity. F, E and C are illustrative areas; bars show records against capacity, and amber means over the limit.

A leaf learns its slice by reading representative and unusual records until further reading stops changing its beliefs. It then watches: code checks every new record, and the agent reads only failures, new kinds of record and a small random sample. Parents build roll-ups from their children’s reports.

Learning: the leaf reads representative and unusual records until further reading stops changing its beliefs.

Figure 4cAgent reading over time. Agent reading falls as leaves move from learning to watching, and rises only when something new arrives. Illustrative, not measured.

Steady state

High update volume does not mean reclustering.

  • Updates do not move records.

    Splits follow what a record is (source, table, client, period), not its state (status, balance), so a paid invoice re-runs the checks in its own leaf.

  • New records fill leaves.

    Code routes them down from the root, and a full leaf splits locally. The tree is never reclustered as a whole.

  • Misfits wait.

    Records that fit nowhere wait at the level where they stopped fitting, and become a new child once there are enough of them. A newly connected source lands as a new branch.

  • Drifted nodes re-split.

    A node that collects many poor fits is re-split from its own records, which also corrects the order sensitivity of incremental trees.

An invoice in Invoices 2025 is paid. Its status changes; what it is does not.

  • internal node (L2)
  • leaf (L1)
  • code
  • over the limit
Figure 4dSteady state. High update volume does not mean reclustering. Updates do not move records. Splits follow what a record is (source, table, client, period), not its state (status, balance), so a paid invoice re-runs the checks in its own leaf.

The shape is close to BIRCH[1], with capacity-limited nodes, root-down insertion and local splits. It borrows COBWEB’s create, merge and split operators[2], and agents, not fixed distance rules, make each decision. Because code splits in parallel below the first level, building the tree costs about one split per level. The root’s agent reviews cluster summaries and representative records, and never reads the database itself.

Table 1. The September Token Atlas hierarchy[6] and its place in the colony.
Token Atlas (September 2026)Venus Colony
L3: a brief always in the prompt (the “root prefix”)The root's summary, first in every node's prompt
L2: roll-ups with exact child listsInternal nodes' roll-ups of their children's reports
L1: digests of N recordsLeaves: up to C records, with notes, beliefs and checks
Three levels, made variable on 30 SeptemberDepth grows per area from the limit rule
Embedding lookup as a shortcut to detailEmbeddings for clustering, routing and jumping to a leaf; never the memory itself
1 million
1,000
32
1,995

depth = 1 + ⌈log32(1 million / 1,000)⌉ = 3

Depth
3 levels
Leaves
1,000
Nodes
1,033
  1. root1
  2. level 232
  3. leaves1,000
Watched closely ≈ node cap × C1 million · 100%

Within the cap: every node fits, with 962 to spare.

Figure 4eDepth from the limit rule. With C records per leaf and fan-out b, depth is 1 + ⌈logb(N/C)⌉. For C = 1,000 and b = 32 that is three levels at a million records and five at a billion. Node cap × leaf capacity is roughly the most records the colony can watch closely, and dropout decides where that attention goes (§5).

5 · Stability

One rule and one cap keep the tree well formed.

The rule is the limit rule. A parent’s children’s reports count against its own memory, so the rule also bounds fan-out: no parent can gather fifty children, and a parent left with one child merges with it. The cap is a maximum node count, set per tenant from serving capacity. Node cap × leaf capacity is roughly the most records the colony can watch closely, and dropout decides where that attention goes.

A leaf over its limit splits into children.

  • internal node (L2)
  • leaf (L1)
  • over the limit
Figure 5One rule, four moves. Each move is local to one parent and its children. The parent decides, and trusted code checks afterwards that every limit holds.

Dropout. When the cap is reached, the least important leaf is dropped. Candidates are shortlisted by recent use, verified findings, dependants, data churn and cost, never by call frequency alone, and new nodes get a grace period. A fold-in test then replays a sample of the candidate’s own past tasks with its parent answering. The node is dropped only if the parent does about as well, so a rarely used but important node survives. This is pruning by ablation rather than random dropout.

1 · Shortlist

Candidates are shortlisted by recent use, verified findings, dependants, data churn and cost, never by call frequency alone. New nodes get a grace period.

  • recent use
  • verified findings
  • dependants
  • data churn
  • cost
  • call frequency alone
  • new nodes: grace period
Figure 5aDropout. When the cap is reached, the least important leaf is dropped by ablation, not at random, and it can come back as an ordinary split.

6 · Communication and authority

Information flows only upward.

A child reports to its parent and never messages a sibling. Smith’s Contract Net[4] allows such lateral messages; we rule them out. A parent settles conflicting reports by running a query whose result tells the claims apart, never by vote: agents that share a model make correlated errors, so their agreement is not evidence. For an independent check, the parent hands only the claim to another child or a short-lived checker.

The runtime agent is an outside caller. It hands a task to a node and receives the answer, and whatever the node learns travels up as usual. This replaces the two-thirds voting roster of the earlier joint design[5].

A child reports to its parent and never messages a sibling. Each report carries its claim, the SQL that tests it and its result, the record versions it rests on, its confidence and what depends on it.

Figure 6Information flows up, with no channel between siblings. Business definitions go to people. Smith's Contract Net allows lateral messages between siblings; the colony rules them out, and this replaces the two-thirds voting roster of the earlier joint design.
Table 2. Who decides what. Hover or select a row to find that party in Figure 6.
The nodeIts beliefs, memory, notes and overlays, and splitting its own territory
The parentConflicts between its children, and merging, regrouping, moving or dropping them
PeopleBusiness definitions, anything that changes the warehouse or Twin, and data-access grants
Trusted codePermissions, tenant isolation, limits and quotas. No agent can change these

Decision rights

7 · Evidence and pilot

What is known so far, and how the pilot will find out the rest.

What has been shownAs of 30 October 2026
  • The roll-up hierarchyNever builtNever built end to end.
  • Token Atlas runTimed outThe only run timed out before producing results.[6]
  • Learned memory against retrievalLost every testLost to retrieval with a frozen encoder in every test so far, so embeddings remain an index and not the memory.[7, 8]
    Learned memory8/192
    Retrieval, frozen encoder182/192

    An example from the tests: embeddings remain an index, not the memory.

  • Open-weight serving smoke test, 28 OctoberPassedNative tools, prefix caching and yes/no logprobs passed: the mechanics work, which is not yet evidence that the agents are useful.
  • Prefix caching on candidate node modelUnprovenNot yet shown to work.
  • Factory throughputNever measuredHas never been measured.
  • UK dedicated GPU qualificationIn progressQualification runs on dedicated UK GPU capacity.
Figure 7Evidence to date. The mechanics work; whether the agents are useful is what the pilot is for.
Three gated stagesEach with its own spend cap
Figure 7aThree-stage gated pilot. Three gated stages, each with its own spend cap. A stage starts only when the one before it has passed its gate.
Risks, and what answers each.
Correlated errorsEvidence and blind re-checks.
Lossy roll-upsExact child lists and drill-down.
Prompt injection in records such as CRM notesRead-only tools, and outputs limited to reports and triage items.
Parents as bottlenecks for cross-territory questionsThe pilot measures this, and it bears on decision 1.

8 · Decisions

Eleven decisions to freeze the specification.

Table 3. Open decisions and proposed defaults.
#DecisionProposed default
1What the top-level split followsContent plus links (shared IDs, foreign keys), so the same client from Xero and HubSpot lands together. Needs entity matching up front.
2Records stay at the sourceYes. Nodes hold record IDs and notes; contents are read from the canonical tables.
3Runtime agent routingAs an outside caller. Capability-card search is the baseline, and the one-token yes/no broadcast is tested against it.
4ModelsA fast open-weight model for node agents on the factory, with our validated private checkpoint for the pilot while UK capacity qualifies. The runtime agent stays on its configured model.
5When the colony runsFactory batch windows, interactive work first. Always-on is a separate decision.
6Where findings goA triage item with evidence, and dependent views marked impaired. Never notify-only.
7How much gets re-checkedA blind re-check for every flag that reaches people; the rest by sample.
8Task threads as inputNot in the pilot. Added later under the whole-thread rule (P8).
9Dropped nodes' evidenceArchived in cold storage, so the node can be re-created cheaply.
10Pilot data and residencyA read-only role on an isolated research copy, with processing in the UK only.
11Success, kill criteria and budgetMore verified findings per GPU-hour than the single-investigator baseline, with no loss of interactive latency. Budget to be set.

Set now, tuned in the pilot

  • Leaf capacity C
  • Per-node memory and KV limits
  • The node cap
  • Split and drop thresholds, with a grace period
  • The learn-to-watch rule
  • The watch-mode sample rate
  • Dropout weights
  • The embedding model

The frozen Arctic encoder the Twin already uses is the first choice of embedding model.

Approvals before production, not needed for the pilot

  • Explicit governance for agents that keep private state between tasks
  • No notify-only monitoring: every finding creates an inspectable triage item
  • A verified tenant-isolation proof across node state and storage
  • A dedicated model-gateway caller class and quota for the colony

References

  1. [1]T. Zhang, R. Ramakrishnan and M. Livny. BIRCH: an efficient data clustering method for very large databases. Proc. ACM SIGMOD, 1996.
  2. [2]D. H. Fisher. Knowledge acquisition via incremental conceptual clustering. Machine Learning, 2(2):139–172, 1987.
  3. [3]R. Bayer and E. McCreight. Organization and maintenance of large ordered indexes. Acta Informatica, 1(3):173–189, 1972.
  4. [4]R. G. Smith. The Contract Net Protocol: high-level communication and control in a distributed problem solver. IEEE Transactions on Computers, C-29(12):1104–1113, 1980.
  5. [5]Project Venus. Earlier multi-agent roster and staged implementation study. Project Venus research note, October 2026; partly superseded by this specification.
  6. [6]Project Venus. Venus Token Atlas hierarchy and calibration study. Project Venus research study, September 2026.
  7. [7]Project Venus. Fast Weight Memory Campaign. Project Venus research study, October 2026.
  8. [8]Project Venus. Territory memory and frozen-encoder retrieval comparison. Project Venus research study, October 2026.