Coarse-to-Fine Learning for Multi-Pipette Localisation in Robot-Assisted In Vivo Patch-Clamp

Authors
Lan Wei, Gema Vera Gonzalez, Phatsimo Kgwarae, Alexander Timms, Denis Zahorovsky, Simon Schultz, Dandan Zhang*

Abstract

Fig 1
Fig. 1 Concept overview of our coarse-to-fine learning approach for in vivo multi pipettes localisation under two-photon microscopy.

In vivo image-guided multi-pipette patch-clamp is essential for studying cellular interactions and network dynamics in neuroscience. However, current procedures mainly rely on manual expertise, which limits accessibility and scalability. Robotic automation presents a promising solution, but achieving precise real-time detection of multiple pipettes remains a challenge. Existing methods focus on ex vivo experiments or single pipette use, making them inadequate for in vivo multi-pipette scenarios.

To address these challenges, we propose a heatmap-augmented coarse-to-fine learning technique to facilitate multi-pipette real-time localisation for robot-assissted in vivo patch-clamp. More specifically, we introduce a Generative Adversarial Network (GAN)-based module to remove background noise and enhance pipette visibility. We then introduce a two-stage Transformer model that starts with predicting the coarse heatmap of the pipette tips, followed by the fine-grained coordination regression module for precise tip localisation. To ensure robust training, we use the Hungarian algorithm for optimal matching between the predicted and actual locations of tips. Experimental results demonstrate that our method achieved > 98% accuracy within 10 μm, and > 89% accuracy within 5 μm for the localisation of multi-pipette tips. The average MSE is 2.52 μm.

The main contributions of this paper include: 1. We propose a novel coarse-to-fine localisation framework that employs a Transformer-based encoder to coarsely identify pipette tip regions first, followed by a ResNet-based decoder that precisely predicts their exact coordinates and improves pipette tips detection accuracy. 2. To address the challenges posed by dynamic, low-resolution two-photon images in living organisms, we developed a CycleGAN-based model that effectively eliminates background noise and enhances pipette tip visibility, improving detection accuracy. 3. We introduce an innovative permutation-invariant loss function, leveraging the Hungarian algorithm to construct a bipartite graph between predicted and ground-truth pipette tip locations, ensuring optimal matching for efficient and robust model training. 4. We evaluate our method on in vivo patch-clamp data with a varying number of pipettes (1 to 4), demonstrating superior performance over existing approaches.

Framework

The workflow of the proposed method for multi-pipette tips localisation is illustrated as follows: 1. To reduce the domain gap between the patch-clamp image of ex vivo data and in vivo data, the CycleGAN model is used to translate the pipette images obtained from the live brain (in vivo) with a noisy background to the target domain (ex vivo) with a clear background. 2. A Vision Transformer (ViT) (encoder model) is used to predict the heatmap of the pipette tips' coarse positions. 3. The generated heatmap is fed to a ResNet-based decoder model to obtain the tips' coordination.

Fig 2
Fig 2. Framework Overview. The model takes the original in vivo image as input and first applies cycleGAN to enhance pipette tip features. The enhanced image, augmented with positional embeddings, is then processed by the encoder, which predicts a coarse heatmap of pipette tip locations. Finally, the decoder refines these predictions, generating precise pipette tip coordinates through the finer coordination learner.
Fig 3
Fig 3. Workflow of cycleGAN-based pipette tip feature enhancement. A single pipette is first cropped from the in vivo two-photon microscopy images. CycleGAN is then applied to remove background noise, enhancing pipette tip visibility. Finally, the denoised single pipette images are mapped back to their original positions while preserving the original pipette tip coordinate labels.

Performance

Table 1 highlights the significant improvements achieved by our model across all baselines and accuracy metrics. Even at the smallest error threshold of 3,μm, our method outperforms the best baseline by 17.93%, demonstrating its superior prediction accuracy.

Table 1
Table 1: Comparison with State-of-the-Arts.

Fig. 4 provides a visual comparison of pipette tip localisation across images containing varying numbers of pipettes (1 to 4). The first row represents the simplest scenario with a single pipette, where all models accurately predict the tip position. However, as the number of pipettes increases, the baseline models struggle to correctly identify multiple tip locations, whereas our model maintains high localization accuracy. The last column of Fig. 4 illustrates the heatmap predictions generated by our model, further emphasizing its precision in detecting pipette tips.

Fig 4
Fig 4. Comparison of pipette localization across images with varying numbers of pipettes (1 to 4).

We evaluated the forward inference time of our model across different hardware configurations (CPU, GPU, and TPU) by running 100 iterations on a batch of images, as presented in Table 2. The purpose of this comparison is to demonstrate the applicability of our model to different experimental setups with varying levels of computational power, highlighting the real-time performance that is crucial for robot-assisted in vivo patch-clamp. Our results indicate that the multi-pipette tip localization process takes an average of 0.07 seconds on the CPU and 0.001 seconds on the RTX 4090 GPU. Resutls demonstrate the real-time capability of our method across various computing environments.

Table 3
Table 2: Inference time across different hardware devices. .

To evaluate the contribution of each component in our model, we conduct ablation studies, as presented in Table 3, under an error threshold of 10 μm. The result confirms that each of our design choices positively impacts the overall performance, with the combination of all components achieving the highest accuracy.

table 2
Table 3. Ablation of different blocks under the error range of 10 μm.

Supplementary Video

Code & Data

Codes and data are available at:

https://github.com/LannWei/Patch-Clamp-Localisation and https://huggingface.co/datasets/Lan-2025/Pipette/tree/main

Contact

If you have any questions, feel free to ask us via: l.wei24@imperial.ac.uk or d.zhang17@imperial.ac.uk.