Giving Microscopes a Brain: From Image Analysis to Experimental Control
A microscopy experiment is normally specified before the sample has revealed very much about itself. The researcher selects the fields of view, imaging channels, exposure times, spatial resolution and acquisition frequency, after which the microscope executes the resulting protocol. This approach is straightforward, reproducible and often entirely appropriate. However, it also assumes that the value of a measurement can be determined in advance.
For heterogeneous or dynamic samples, this assumption is frequently incorrect. Most fields of view may contain information already represented elsewhere, while a rare event occurs only briefly in one small region. A fixed acquisition protocol cannot distinguish between these situations. It measures according to the original plan, regardless of what the preceding images have shown.
Artificial intelligence has already become highly capable of analysing microscopy data after acquisition. Models can detect objects, segment cells, track particles, classify phenotypes and extract representations from images that would be difficult to inspect manually. Nevertheless, post-acquisition analysis can only operate on what the instrument has recorded. It cannot recover a transient event that was never imaged, prevent unnecessary exposure that has already damaged a sample, or redirect acquisition towards a feature that became interesting halfway through the experiment.
The next development in scientific imaging is therefore not simply a more accurate analysis model. It is a measurement system in which analysis can influence what is measured next.
Why fixed acquisition leaves information on the table
Microscopy is governed by trade-offs. Increasing spatial resolution generally restricts the area that can be observed efficiently. Increasing temporal resolution produces more data and may increase light exposure. Longer exposure can improve signal-to-noise ratio, but it can also increase photobleaching or phototoxicity in live-cell experiments. Imaging more channels may provide richer biological information while making the acquisition slower and more invasive.
A fixed protocol resolves these trade-offs globally. It applies approximately the same acquisition strategy whether a region is empty, redundant, artefactual or biologically unusual. This is sensible when the sample is sufficiently uniform or when the relevant events are known in advance. It becomes inefficient when useful information is distributed unevenly in space or time.
A human microscopist already responds to this problem by adapting the measurement. They may adjust focus when the sample drifts, inspect an unusual structure at higher magnification, revisit an uncertain region or stop collecting once sufficient evidence has been obtained. The experimentalist is therefore part of the acquisition loop. They interpret the current state of the sample and use that interpretation to decide what the instrument should do next.
Closed-loop microscopy formalises this behaviour. The system first observes the sample and interprets the resulting data. It then selects an action based on the experimental objective, executes that action through the instrument and evaluates the new measurement. The loop may involve moving the stage, changing the acquisition frequency, switching imaging modality, adjusting illumination or requesting human review.
The important distinction is that the model no longer receives the dataset as a finished object. It participates in determining how that dataset is constructed.
Turning model uncertainty into an experimental signal
An adaptive microscope still requires a criterion for deciding which measurement is useful. Simply allowing a model to control the instrument does not provide such a criterion.
One possibility is to respond to the detection of a specific event. In event-driven microscopy, a sample can initially be monitored using a relatively gentle or low-frequency acquisition. When a model detects the onset of a relevant process, the microscope changes to a faster or higher-resolution mode. Mahecic and colleagues demonstrated this principle by using neural-network-based recognition to trigger real-time control of an instant structured illumination microscope. The system captured mitochondrial and bacterial division at acquisition rates matched to the events while extending the overall imaging duration compared with continuously acquiring at high frequency [1].
This approach is useful because the measurement cost is concentrated where the information is. The microscope does not need to image every moment at maximum resolution merely because an interesting event might occur.
A second possibility is to use uncertainty. If a model is already confident that a region resembles many previously observed regions, another nearly identical measurement may add little information. A region close to the model's decision boundary, or one that differs substantially from the existing data, may be considerably more useful. Active learning applies this principle during model development by selecting examples whose labels are expected to improve the model. In microscopy, the same reasoning can be moved upstream and used to select the measurements themselves.
This changes what data efficiency means. It no longer refers only to training a model from fewer images. It also means collecting images more deliberately. Instrument time, sample exposure, storage, annotation and human attention are all limited resources. An acquisition strategy that produces fewer images can therefore be superior if each image contributes more directly to the scientific objective.
However, uncertainty is only useful when it is connected to that objective. A model may be uncertain because it has encountered a genuinely unusual biological structure, but it may also be uncertain because the image is out of focus, contaminated or outside the conditions represented during training. An adaptive system must distinguish informative uncertainty from technical failure. Otherwise, it risks spending most of its acquisition budget investigating artefacts.
From adaptive imaging to experimental control
The simplest closed loop changes how the sample is observed. More advanced systems can also change what happens to the sample.
Recent work on outcome-driven microscopy illustrates this progression. Passmore and colleagues combined real-time microscopy with optogenetic control to guide cell migration and regulate nucleocytoplasmic transport towards predefined outcomes [2]. In this setting, the microscope does more than detect an event and alter the acquisition settings. The observed state is compared with a target state, and the resulting error determines the next experimental intervention.
This can be understood as a feedback-control problem. The microscope measures the current state of the system, a model estimates the relevant biological variables, and a controller selects an action intended to move the system towards a specified objective. The next image then reveals whether the intervention had the intended effect.
The same principle is being explored in materials characterisation. In autonomous scanning probe microscopy, active-learning methods can select which locations or properties should be measured next. The SEEK framework extends this approach by incorporating prior knowledge and human interest into the acquisition policy, allowing the system to direct its exploration towards scientifically relevant structures or spectral features [3].
These examples highlight an important limitation of purely generic autonomy. An instrument cannot determine what is scientifically interesting from uncertainty alone. It requires a representation of the objective, together with relevant physical or biological constraints. Expert knowledge is therefore not an obstacle that automation must eliminate. It is information that should be encoded into the experimental strategy.
Connecting models to instruments
Once an AI system is allowed to influence an experiment, communication between the model and the instrument becomes a central engineering problem. A prediction such as "this region is unusual" has no operational effect until it can be translated into a valid instrument action.
A microscope might expose a bounded set of operations such as moving the stage within an approved region, acquiring a specified channel, changing between predefined objectives or pausing for review. Each operation should have explicit inputs, permitted ranges and predictable failure behaviour. The model can then request an action without receiving unrestricted access to the underlying instrument.
Tool protocols such as the Model Context Protocol provide one possible interface for exposing defined software capabilities to an AI system. MCP servers can make tools available through structured schemas, allowing a model to discover which operations exist and what parameters they accept [4]. In a laboratory architecture, such a protocol could connect a higher-level agent to acquisition software, analysis pipelines, image stores or experiment records.
However, a tool protocol is an interface rather than a control strategy. It does not determine whether an action is scientifically justified, whether a parameter is safe for the instrument or whether a prediction is sufficiently reliable to act upon. Fast control loops may also be better handled by deterministic, instrument-specific software, with an agent operating at a slower orchestration level.
A practical architecture would therefore separate responsibilities. The AI system may select or propose an experimental action. The instrument control layer validates the request against hardware limits and safety constraints. A separate permission layer determines whether the action can proceed automatically or requires confirmation. The complete decision, including the evidence available to the model, should be recorded for subsequent inspection.
This layered design is more important than giving an agent the largest possible collection of tools. A laboratory system becomes useful when its available actions correspond precisely to the authority it should have.
Autonomy should be calibrated to consequence
The principal difficulty in agentic microscopy is not making software perform an action. It is determining when the system has enough evidence to act, how damaging an incorrect action could be and when responsibility should return to the scientist.
Some microscope operations are low-cost and readily reversible. Recentring a tracked object, repeating a low-exposure quality-control image or flagging a field of view for later inspection may be suitable for automatic execution. Other operations can modify the sample, consume scarce material or expose living cells to damaging illumination. These should require stronger evidence and, in many cases, explicit approval.
The appropriate degree of autonomy is therefore action-dependent. A useful system may automatically select fields of view while requiring confirmation before changing an experimental perturbation. It may adjust focus within a validated range but reject illumination settings outside the instrument's safety envelope. It may continue acquisition while the sample remains within the conditions represented during validation, then pause when the observed data move outside that distribution.
Uncertainty estimation is essential in this setting, although it is not sufficient on its own. The system also requires quality-control models, instrument state monitoring, immutable raw data, audit logs and reliable stop conditions. If tools can access external data or software, they must additionally be restricted according to least privilege and bound to their intended resources [5]. Emerging research on MCP implementations also indicates that malicious or misleading tool metadata can influence model behaviour, reinforcing the need to validate tools and isolate their execution rather than treating protocol compatibility as a security boundary [6].
These safeguards do not reduce the usefulness of autonomy. They define the conditions under which it can be trusted.
The scientist remains part of the loop
Scientific judgement is not equivalent to continuous manual instrument control. A researcher contributes by defining the question, deciding which outcomes are meaningful, recognising when an observation contradicts the current model and interpreting the result in the context of existing knowledge. Moving a stage or repeatedly checking image quality is rarely the most valuable use of that expertise.
Closed-loop microscopy can redistribute this work. The scientist specifies the objective and the acceptable experimental boundaries. The system performs repetitive monitoring, identifies informative measurements and handles local acquisition decisions within those boundaries. When the data no longer support autonomous action, control returns to the researcher.
At IFLAI, this is why we see agentic microscopy as a natural extension of data-efficient scientific AI. A model that extracts more information from each measurement can improve analysis after acquisition. A model connected to the acquisition process can additionally help determine which measurements should exist in the first place.
The meaningful transition is therefore not from an unintelligent microscope to an intelligent one. It is from an instrument that follows a fixed sequence to a measurement system that responds to evidence as the experiment develops.
A more capable microscope does not need to act independently at every opportunity. It needs to recognise which next measurement will reduce the relevant uncertainty, execute that measurement within a controlled boundary and know when the decision belongs to the scientist.
References
- [1] Mahecic et al., "Event-driven acquisition for content-enriched microscopy," Nature Methods (2022)
- [2] Passmore et al., "Closed-loop optogenetic control of cell biology enables outcome-driven microscopy," Nature Communications (2025)
- [3] Pratiush et al., "Scientific exploration with expert knowledge (SEEK) in autonomous scanning probe microscopy with active learning," Digital Discovery (2025)
- [4] Model Context Protocol, "Tools specification" (2025)
- [5] Model Context Protocol, "Authorization specification" (2025)
- [6] Huang et al., "Model Context Protocol Threat Modeling and Analyzing Vulnerabilities to Prompt Injection with Tool Poisoning," arXiv preprint (2026)