M2 · Gaze & DeepGaze

Order and time. The heatmap is a sum over all possible looks. The scanpath is one specific sequence, and it captures the fact that the eye does not jump to its second-favourite region right after its favourite - it explores. The next fixation depends on the recent ones. The heatmap cannot express that.

It darkens. Every click multiplies in another suppression bump at that location, so the region effectively becomes a probability hole. Real DeepGaze III only sees the last four fixations, so it cannot keep darkening forever - but the qualitative effect is the same.

It tells you the user will not stare at the same point forever. If your pipeline only re-renders when the gaze moves, you need a realistic model of when it moves and where it goes. Without inhibition of return, simulated gaze would just camp on the most salient point and never explore the scene - which is the opposite of what real users do.

On the face - especially the eyes. We usually find faces to be very important in a scene and we tend to fixate our gaze initially onto them .DeepGaze was trained on humans looking at natural images and so it captures this behavioural pattern.

It is a fudge factor that says "if you have no idea what is in the image, guess that the user will look near the middle, because that is what people usually do." It improves predictions on average and lets the rest of the model focus on learning content-driven attention.

No - the sampling step is random. The same probability map can produce many different scanpaths. This is realistic: two humans looking at the same image do not produce identical scanpaths either. The model gives you a distribution over plausible behaviours.

Yes. The first fixation enters the history; the model is then asked "given that the user just looked at (x, y), where would they look next?" - and the map shifts. That shift is the model expressing inhibition of return and natural saccade lengths.

Because most of the time the next region of interest is near the current one. The implant does not have to re-render the whole field; it can update mostly the local area. This is what makes foveated rendering with limited bandwidth feasible at all.

At least as fast as the dwell time - say, every 100-200 ms - or the user will saccade away before the new region is ever rendered. In practice you want headroom: ~60 Hz refresh is comfortable, well below the saccade rate.

The scanpath. A prosthesis only stimulates a small patch of cortex at a time, so at every fixation it has to pick one next location — that is a sequence of decisions, not a set. A heatmap is a sum across many viewers and many looks; it tells you where attention pools on average but not the order in which a single user would visit those points.

It pushes the model to explore: without it, the most salient point wins every sample and the gaze camps there forever (this is the failure mode of §03). Cranked too high, the suppression around recent fixations becomes so strong that the gaze is pushed into low-saliency regions just to avoid where it has been — exploration without any content guiding it.

You're back to a per-frame saliency model: the same image always produces the same map. The first sampled fixation is unchanged, but every later fixation ignores where the eyes have already been. The scanpath becomes a sequence of independent draws from the same map — no inhibition of return, no exploration, no temporal structure.

Two contributions. (a) Real saliency maps are peaky: a few bright regions on a wide dim background. Sampling proportional to the map gives mostly short jumps to the dominant peak near the current fixation, occasional medium jumps to a secondary peak, and rare long jumps when the background scores a hit — that mixture has a long right tail. (b) Biology layers in short corrective saccades within an object and longer between-object saccades. The combination is approximately lognormal. Real DeepGaze III on natural images does recover this shape; the toy model on this page does not, because its “scene” has only five Gaussian blobs, so the peaks-vs-tails geometry is too uniform — which is exactly why §06 substitutes literature-derived draws.