NTH / M5 EXPERT · OPTIONAL

M5 · Decoding & closed loop

Pretrained decoder demo first, then train your own. End-to-end co-optimization through the differentiable simulator, and the minimal closed-loop demo. This module is optional — it goes deeper than M1–M4 and assumes you are comfortable with model training, so it works best as a self-guided dive after the earlier modules.

Module: M5 Pair with: decoding-and-closed-loop.ipynb · Colab · VS Code
M1
Camera + CV
→
M2
Gaze
→
M3
Stimulation
→
M4
Phosphenes
→
M5 · here
Decoding

01The loop in one picture

The previous four modules ran open loop: image → stim → phosphenes, done. The loop closes when a decoder reads the phosphenes back out and that readout drives the next stim. Without it, a prosthesis can’t cope with drift, adaptation, or a wearer who shifts their gaze mid-sentence.

CAMERA scene in VISION MODEL edges / mask STIM current per ch. DYNAPHOS phosphenes DECODER readout READOUT brightness / label PID · ∇ error → PID · gradient → end-to-end training GAZE crop centre

Hover any node for a one-line tour · click to jump to that section.

The first five blocks already ran end-to-end across M1–M4. This module wires the dashed pink arrow on the bottom: the decoder’s readout flows back into the next stim command (§02 PID) or into the vision model’s weights (§04 end-to-end). §05 puts the whole thing on one page.

What is decoded?

readout target

On the page, brightness of the phosphene field, one number per frame. In the notebook, a small CNN reads the canvas and produces a digit, letter, or brightness scalar.

What is fed back?

actuation

Two flavours. §02 feeds back stim current via a PID controller, one electrode, one number per frame. §04 feeds back vision-model weights via gradients through the (differentiable) dynaphos simulator.

What is the goal?

loss

Either track a target (PID holding a reference brightness despite adaptation) or recover the input (an end-to-end loss that asks the decoder’s output to match the original image). Both close the loop in the same place.

02PID: control the dynamics

Dynaphos has state. Activation builds up and then leaks; an adaptation trace makes the same current dim over seconds. Open-loop stimulation can’t correct for any of that. A controller can — given one number per frame to read. The cheapest such number is the mean of the rendered phosphene’s pixels: a real decoder (§03) does better, but mean-pixel is enough to close the loop. A classical PID reads it, compares to a target, writes the next stim current. Three gains, one electrode.

target achieved |error| stim current
200
150
30
u(t) = 100·e(t)P + 80·∫e dτI + 15·de/dtD
err: — · overshoot: —
Crank Ki to the maximum on the ‘step’ target with Kp and Kd at zero. What does the trace do, and why?

It oscillates and overshoots wildly. Pure integral has no idea where the error is going. It just keeps stuffing accumulated error into the current. By the time the brightness reaches the target, the integral has already commanded a huge overshoot, which has to be unwound with negative error, which then undershoots. Adding a little Kd reads the “we’re moving fast toward the target” signal and pulls back early.

Pure P-only is the simplest, but on the ‘ramp’ target it never quite catches up. Why?

P-only is proportional to the current error. On a ramp, the target keeps moving away, so to make the brightness move at the same rate you need a non-zero error to drive a non-zero control output. That constant gap is the classic steady-state error that integral action exists to eliminate.

03Train your own decoder

The decoder’s job is to take a phosphene render and recover what the user is looking at. Classification is the cleanest version: one digit in, one digit out. The model gets ten training examples (one per digit, MNIST-style, rendered into phosphenes by dynaphos). Hit Train. The weights below update as the network learns to read its own percepts.

Train MLP · 784 → 32 → 10 softmax
untrained
input: 10 digits, one per class
phosphene render → predicted digit
loss · cross-entropy —
first-layer weights 32 hidden units · 28×28 each
What do the first-layer weight maps start to look like after a few hundred training steps?

They develop digit-shaped templates. Some units light up for round shapes (0, 6, 9), some for vertical strokes (1, 7), some for the closed loop of an 8. They’re not pretty after only ten examples, but the structure is unmistakable: the network has carved its hidden layer into class-specific feature detectors. With the full MNIST training set in the notebook, these templates sharpen into proper handwritten-digit prototypes.

04End-to-end through the simulator

So far the decoder has been learning to read a fixed dynaphos render. Because dynaphos is differentiable, gradients flow all the way from a decoder loss back through the simulator into a learnable vision model. Vision model and decoder co-train. The vision model learns what makes the decoder’s job easy, even if that means producing pre-stim patterns that look nothing like Sobel edges.

VISION MODEL learnable θ1 DYNAPHOS frozen, differentiable DECODER learnable θ2 LOSS recover input gradients flow back through the simulator
Same loss, two trainable blocks, frozen physics in the middle. The trick is that dynaphos’s charge → activation → brightness chain is built out of differentiable operations (sigmoids, linear filters, leaky integrators), so backpropagation just works.
Side-by-side same input, two vision models, phosphene fields
Sobel vision model blend α = 100%
L2 against input · lower is better · using documented stand-in until exported weights load

05The whole loop, live

Everything together, on one canvas, with one play button. An input image gets gaze-cropped, run through the vision model, turned into per-channel stimulation, fed to the dynaphos temporal simulator, and decoded. In closed-loop mode, the decoder reading drives the next stim through the PID from §02. Toggle open / closed to see what the feedback actually does.

t = 0.00 s
1 · input after gaze crop
2 · activation mask vision model out
3 · phosphene field dynaphos with state
4 · decoded brightness target vs achieved · error band colored by magnitude ✓ tracking
—

06Self-check

Predict the answer first, then verify with the demos above.

Open loop drifts even with a perfect vision model. Why?

Because dynaphos has state. The adaptation trace builds with stimulation; the same current that produced a bright phosphene at t = 0 evokes a dimmer one at t = 2 s. Open loop never sees that drift, so it never corrects. Closed loop reads the decoder, notices the brightness has dropped, and pushes more current in.

When does PID over-correct, and what does the trace look like?

Whenever Ki dominates and there’s any non-zero delay between command and percept. The integral keeps stuffing accumulated error into the current even after the brightness has reached the target, so you get a big overshoot, then a slow climb-down. The §02 PID trace shows this clearly. Pure I gives high-amplitude oscillations; a small Kd term reads the “we’re heading there fast” signal and pulls back early.

After end-to-end training, what does the vision model learn that hand-tuned Sobel doesn’t?

It learns to encode for the decoder it’s coupled to, not for human perception. Sobel produces edge maps that look right to us. The end-to-end vision model is free to produce whatever pre-stim pattern makes the decoder’s job easiest, even if that pattern looks nothing like a Sobel edge. It’s also why end-to-end systems can outperform hand-tuned ones on the actual recovery loss, even though the intermediate stages look weirder.

Which loop ingredient matters most: a better decoder, a better controller, or a better vision model?

Depends on what’s broken. If the decoder is wrong, the loop chases a phantom target (garbage in, garbage out). If the controller is wrong, the loop oscillates or never converges. If the vision model is wrong, the loop converges to a sub-optimal percept. In practice, real systems iterate on all three. End-to-end training (§04) is the one trick that lets them co-improve.

Could you replace PID with a learned controller? What would that look like?

Yes. Train a small recurrent network on the error signal with the same loss as the PID (track the target). It would essentially learn to be a non-linear PID (with state-dependent gains), and could in principle handle the dynaphos non-linearities better than a fixed-gain PID. The cost is opacity. PID gives you three numbers you can reason about; a learned controller is a black box that mostly works. Real prosthesis pipelines tend to use the simplest controller that works.

07Where to next

The notebook is the optional, deeper Python version of everything you just touched. The three workshop tracks pick up from here.

Notebook companion

real CNN, real dataset

The page demo trains a 1-layer MLP on synthetic canvases. The notebook trains a small CNN end-to-end through dynaphos on a small real dataset and plots a proper learning curve.

decoding-and-closed-loop.ipynb

Experimental track

perceptual eval

Pick a target image. Compare phosphene renders from hand-tuned vs end-to-end pipelines; collect subjective ratings; quantify the difference.

Developer track

real-time / control

Port §02’s PID to per-channel control on the Utah array from M3. Investigate where it breaks (channel coupling, charge budget). Optionally try a learned gain-scheduler.

08Tools & references

Further reading — vision-restoration field