CAVE Query: Proofread Cells

Version update

We have updated to v1926. Previous versions of this notebook are available from the repository history.

The Connectome Annotation Versioning Engine (CAVE) is a suite of tools developed at the Allen Institute and Seung Lab to manage large connectomics data.

Initial Setup

Before using any programmatic access to the data, you first need to set up your CAVEclient token.

CAVEclient

Most programmatic access to the CAVE services occurs through CAVEclient, a Python client to access various types of data from the online services.

Full documentation for CAVEclient is available here.

To initialize a caveclient, we give it a datastack, which is a name that defines a particular combination of imagery, segmentation, and annotation database. For the MICrONs public data, we use the datastack name minnie65_public.

import numpy as np
import pandas as pd

from caveclient import CAVEclient
datastack_name = 'minnie65_public'
client = CAVEclient(datastack_name)

# Show the description of the datastack
client.info.get_datastack_info()['description']

'This is the publicly released version of the minnie65 volume and segmentation. '

Materialization versions

Data in CAVE is timestamped and periodically versioned - each (materialization) version corresponds to a specific timestamp. Individual versions are made publicly available. The Materialization client allows one to interact with the materialized annotation tables that were posted to the annotation service. These are called queries to the dataset, and available from client.materialize. For more, see the CAVEclient Documentation.

Periodic updates are made to the public datastack, which will include updates to the available tables. Some cells will have different pt_root_id because they have undergone proofreading.

Tip

For analysis consistency, is worth checking the version of the data you are using, and consider specifying the version with client.version = your_version

Read more about setting the version of your analysis

# see the available materialization versions
client.materialize.get_versions()

[1300, 117, 943, 1507, 1621, 1718, 1822, 1926]

And these are their associated timestamps (all timestamps are in UTC):

for version in client.materialize.get_versions():
    print(f"Version {version}: {client.materialize.get_timestamp(version)}")

Version 1300: 2025-01-13 10:10:01.286229+00:00
Version 117: 2021-06-11 08:10:00.215114+00:00
Version 943: 2024-01-22 08:10:01.497934+00:00
Version 1507: 2025-07-31 08:10:01.117494+00:00
Version 1621: 2025-11-25 08:10:01.094430+00:00
Version 1718: 2026-03-07 08:10:01.190228+00:00
Version 1822: 2026-06-27 14:05:22.813147+00:00
Version 1926: 2026-09-28 22:44:42.598563+00:00

# set materialization version, for consistency
client.version = 1926 # current public as of 9/28/2026

Querying Proofread neurons

Proofread neurons

Proofreading is necessary to obtain accurate reconstructions of a cell. In the MICrONS dataset, the general rule is that dendrites onto cells with a single cell body are sufficiently proofread to trust synaptic connections onto a cell. Axons on the other hand require so much proofreading that only ~1800 cells have axons such that their outputs should be used for analysis.

Table name: proofreading_status_and_strategy

The table proofreading_status_and_strategy describes the status of cells that have undergone manual proofreading.

Because of the inherent difference in the difficulty and time required for different kinds of proofreading, we describe the status of axons and dendrites separately.

Each compartment status may be either:

FALSE: indicates no comprehensive proofreading has been performed, or is not applicable.
TRUE: indicates that false merges have been comprehensively removed, and the compartment is at least ‘clean’. Consult the strategy column if completeness of the compartment is relevant to your research.

An axon or dendrite labeled as status=TRUE can be trusted to be correct, but may not be complete. The degree of completion can be read from the strategy column. For more information, please see Proofreading and Data Quality; or also the microns-explorer page on proofreading strategies.

The key columns are:

Proofreading Status Table
Column	Description
`id`	Soma ID for the cell
`pt_position` `pt_supervoxel_id` `pt_root_id`	Bound spatial point columns associated with the centroid of the cell nucleus
`valid_id`	The root id of the neuron when it the proofreading assessment was made. NOTE: if this does not match the `pt_root_id` then the cell has undergone further changes. This is usually and improvement in proofreading, but proceed with caution.
`status_dendrite`	The status of the dendrite proofreading. May be `TRUE` or `FALSE`
`status_axon`	The status of the axon proofreading. May be `TRUE` or `FALSE`
`strategy_dendrite`	The strategy employed to proofread the dendrite. See strategy table below for details
`strategy_axon`	The strategy employed to proofread the axon. See strategy table below for details

The specific strategies are as follows (and will update over time):

Proofreading Strategies

Proofreading Strategy Table
Strategy	Description
`none`	No cleaning, and no extension. Indicates an entry in `proofreading_status` that is `FALSE` for that compartment
`dendrite_clean`	The dendrite had incorrectly-merged axon and dendritic segments comprehensively removed, meaning the input synapses are accurate. The dendrite may be incorrectly truncated by segmentation error. Not all dendrite tips have been checked for extension. No comprehensive attempt was made to re-attach spine heads.
`dendrite_extended`	The dendrite had incorrectly-merged axon and dendritic segments comprehensively removed, meaning the input synapses are accurate. Every tip was identified, manually inspected, and extended if possible. No comprehensive attempt was made to re-attach spine heads.
`axon_column_truncated`	The axon was extended within the V1 cortical column, with a preference for local connections. In some cases the axon was cut at the column boundary and/or the layer boundary, especially the boundary between layers 2/3 and layer 4. Output synapses represent a sampling of potential partners
`axon_interareal`	The axon was extended with a preference for branches that projected to other brain areas. Some axon branches were fully extended, but local connections may be incomplete. Output synapses represent a sampling of potential partners.
`axon_partially_extended`	The axon was extended outward from the soma, following each branch to its termination. Output synapses represent a sampling of potential partners.
`axon_fully_extended`	Axon was extended outward from the soma, following each branch to its termination. After initial extension, every endpoint was identified, manually inspected, and extended again if possible. Output synapses represent a largely complete sampling of partners.

This table, proofreading_status_and_strategy, supercedes proofreading_status_public_release.

# Standard query
client.materialize.query_table('proofreading_status_and_strategy')

# Content-aware query
client.materialize.tables.proofreading_status_and_strategy(status_axon='t').query()

Here we query and return the table as of version 1300.

For the commands used for querying tables, see the previous quickstart notebook on CAVE queries

proof_all_df = client.materialize.query_table("proofreading_status_and_strategy", 
                                              desired_resolution=[1, 1, 1], 
                                              split_positions=True)

proof_all_df["strategy_axon"].value_counts()

strategy_axon
axon_partially_extended    1382
axon_fully_extended         802
axon_interareal             120
none                         41
Name: count, dtype: Int64

Filtering Queries by proofreading status

We can filter our query to only return rows that match a condition by adding a filter to our query:

proof_axon_df = client.materialize.query_table("proofreading_status_and_strategy", 
                                               filter_in_dict={"strategy_axon": ["axon_partially_extended", "axon_fully_extended", "axon_interareal"]}, 
                                               desired_resolution=[1, 1, 1], 
                                               split_positions=True)
proof_axon_df.tail()

	id	created	superceded_id	valid	pt_position_x	pt_position_y	pt_position_z	valid_id	status_dendrite	status_axon	strategy_dendrite	strategy_axon	pt_supervoxel_id	pt_root_id
2299	6052	2026-09-28 19:01:13.712812+00:00	<NA>	True	732736	460032	861880	864691136914803953	True	True	dendrite_clean	axon_partially_extended	89938540196086387	864691136914803953
2300	6053	2026-09-28 19:01:13.732568+00:00	<NA>	True	699264	807360	814560	864691135695429823	True	True	dendrite_extended	axon_partially_extended	88824253679891664	864691135695429823
2301	6054	2026-09-28 19:01:13.755351+00:00	<NA>	True	749248	420032	862520	864691135869973270	True	True	dendrite_clean	axon_partially_extended	90500115759990681	864691135869973270
2302	6055	2026-09-28 19:26:26.104991+00:00	<NA>	True	745088	477696	844000	864691134942943075	True	True	dendrite_clean	axon_partially_extended	90361302349957438	864691135158320484
2303	6056	2026-09-28 19:26:26.144672+00:00	<NA>	True	746112	507776	809920	864691135977473391	True	True	dendrite_extended	axon_fully_extended	90432701752255117	864691135977473391

A more unified filter interface is available through a “table manager” interface.

Rather than passing a table name to the query_table function, client.materialize.tables has a subproperty for each table in the database that can be used to filter that table.

The general pattern for usage is

client.materialize.tables.{table_name}({filter options}).query({format and timestamp options})

where {table_name} is the name of the table you want to filter, {filter options} is a collection of arguments for filtering the query, and {format and timestamp options} are those parameters controlling the format and timestamp of the query.

Caution

Use of this functionality will show a brief warning that the interface is experimental. This is because the interface is still being developed and may change in the near future in response to user feedback.

With this, we can easily query all proofread cells with proofread axons:

proof_axon_df = client.materialize.tables.proofreading_status_and_strategy(strategy_axon=["axon_partially_extended", "axon_fully_extended", "axon_interareal"]).query(
    select_columns=['pt_root_id','status_axon','status_dendrite','strategy_axon','strategy_dendrite'],
)
proof_axon_df.tail()

	pt_supervoxel_id	pt_root_id	status_axon	status_dendrite	strategy_axon	strategy_dendrite
2299	89938540196086387	864691136914803953	True	True	axon_partially_extended	dendrite_clean
2300	88824253679891664	864691135695429823	True	True	axon_partially_extended	dendrite_extended
2301	90500115759990681	864691135869973270	True	True	axon_partially_extended	dendrite_clean
2302	90361302349957438	864691135158320484	True	True	axon_partially_extended	dendrite_clean
2303	90432701752255117	864691135977473391	True	True	axon_fully_extended	dendrite_extended

Combining proofread cells and cell types

For analysis, often you are interested in neurons that are at the intersection of two or more groups. For example: proofread cells that are also layer 2/3 pyramidal cells. The general workflow for this type of analysis is to:

Query from one table, for example the proofreading_status_and_strategy table
Query from another table, for example the aibs_metamodel_celltypes_v661
Merge the two tables on the shared index, in this case pt_root_id

We covered querying cell types in the previous quickstart notebook. Now lets put that together with the proofreading query:

# Query proofread cells with status_axon==True
proof_df = client.materialize.tables.proofreading_status_and_strategy(status_axon="t").query(
    select_columns=['pt_root_id','status_axon','status_dendrite','strategy_axon','strategy_dendrite'],
)

# Query cell types
cell_type_df = client.materialize.tables.aibs_metamodel_celltypes_v661().query(
    select_columns = {'nucleus_detection_v0': ['pt_root_id', 'id'],
                      'aibs_metamodel_celltypes_v661': ['classification_system','cell_type'],
                     },
)

Tip

Note the ‘select_columns’ argument differs between the two tables. Thay is because the second table, aibs_metamodel_celltypes_v661 is itself a reference on nucleus_detection_v0. That means the id column returned here is the same as the nucleus id of the cell. This is handy for referencing the same cell across materialization versions as the nucleus id does not change, whereas the pt_root_id will change with proofreading.

Now we can merge the two tables together on the shared index!

But it is worth checking if there are duplicates in either of the tables. How you handle duplicates will depend on your question, and the table you are using. Here we might see duplicates from multi-soma merges in the cell type table

cell_type_df.value_counts('pt_root_id')

pt_root_id
864691136381686997    482
0                     175
864691136974041116     60
864691135303545767     43
864691135455264362     43
                     ... 
864691137199858753      1
864691137199863361      1
864691137199868225      1
864691137199871553      1
864691137200143169      1
Name: count, Length: 91498, dtype: int64

For analytical simplicity, we will drop any multi-soma objects. We will also rename the id column for clarity

# Drop duplicate pt_root_id and rename the nucleus_id
cell_type_df = (cell_type_df
                .drop_duplicates('pt_root_id', keep=False)
                .rename(columns={'id': 'nucleus_id'})
               )
                        
cell_type_df.head()

	pt_supervoxel_id	pt_root_id	nucleus_id	classification_system	cell_type
0	93606511657924288	864691136274724621	336365	excitatory_neuron	5P-IT
1	79385153184885329	864691135489403194	110648	excitatory_neuron	23P
2	79035988248401958	864691136147292311	112071	excitatory_neuron	23P
3	84529699506051734	864691135655940290	197927	nonneuron	oligo
4	83756261929388963	864691135809440972	198087	nonneuron	astrocyte

Now we can merge the two tables with pandas.merge, on index pt_root_id. We will keep the inner join of the two tables: cells that 1) are proofread, and 2) have a cell type

proof_cell_type_df = pd.merge(proof_df, cell_type_df, on='pt_root_id', how='inner')
proof_cell_type_df.tail()

	pt_supervoxel_id_x	pt_root_id	status_axon	status_dendrite	strategy_axon	strategy_dendrite	pt_supervoxel_id_y	nucleus_id	classification_system	cell_type
2187	89938540196086387	864691136914803953	True	True	axon_partially_extended	dendrite_clean	89938540196086387	292714	excitatory_neuron	23P
2188	88824253679891664	864691135695429823	True	True	axon_partially_extended	dendrite_extended	88824253679891664	267038	excitatory_neuron	5P-ET
2189	90500115759990681	864691135869973270	True	True	axon_partially_extended	dendrite_clean	90500115759990681	291177	excitatory_neuron	23P
2190	90361302349957438	864691135158320484	True	True	axon_partially_extended	dendrite_clean	90361302349957438	292727	excitatory_neuron	23P
2191	90432701752255117	864691135977473391	True	True	axon_fully_extended	dendrite_extended	90432701752255117	292575	excitatory_neuron	23P

And we have the list of all proofread cells, by their cell type!

We can do this same kind of query more simply by: querying the second table by BOTH the root ids of interest and the cell type of interest. If we wanted only the proofread 23P cells, we could do:

# Query the proofread 23P cells, and merge the proofreading status
proof_23p_df = (client.materialize.tables.aibs_metamodel_celltypes_v661(pt_root_id=proof_df.pt_root_id, cell_type='23P').query(
    select_columns = {'nucleus_detection_v0': ['pt_root_id', 'id'],
                      'aibs_metamodel_celltypes_v661': ['classification_system','cell_type'],
                     },  )
                .rename(columns={'id': 'nucleus_id'})
                .merge(proof_df, on='pt_root_id', how='inner')
               )

proof_23p_df.head()

	pt_supervoxel_id_x	pt_root_id	nucleus_id	classification_system	cell_type	pt_supervoxel_id_y	status_axon	status_dendrite	strategy_axon	strategy_dendrite
0	88884727087844075	864691135811910092	258375	excitatory_neuron	23P	88884727087844075	True	True	axon_fully_extended	dendrite_extended
1	88956195276658332	864691135257669039	258377	excitatory_neuron	23P	88956195276658332	True	True	axon_partially_extended	dendrite_clean
2	89378751472802282	864691135469856396	258403	excitatory_neuron	23P	89378751472802282	True	True	axon_fully_extended	dendrite_extended
3	88674101690450393	864691135763433270	258225	excitatory_neuron	23P	88674101690450393	True	True	axon_partially_extended	dendrite_clean
4	90150539848910271	864691134942468707	292833	excitatory_neuron	23P	90150539848910271	True	True	axon_partially_extended	dendrite_clean