Almost every image sensor in commercial production is a compromise negotiated with the human retina. The negotiation happened decades ago, it was entirely reasonable at the time, and the party it was negotiated on behalf of is no longer the primary consumer of the output. Most images captured today are never looked at by anyone. They are ingested by a model, reduced to a decision, and deleted.
The interesting consequence is not philosophical. It is that a long list of specific engineering decisions inside a camera exist for reasons that stopped applying, and each one costs photons, watts, or information.
Three Colors, Sixty Frames, One Gamma Curve
Start with the color filter array. A Bayer mosaic assigns each photosite to red, green or blue in a repeating two-by-two pattern, with green given twice the representation of the other two. This is not an optical requirement. It is a direct transcription of trichromatic human vision, with the green weighting chosen because human luminance sensitivity peaks in the middle of the visible band. The cost is immediate: a filtered pixel absorbs and discards the majority of the light that reaches it, and two-thirds of the color values in the final image are not measured but interpolated during demosaicing. A camera throws away most of its incoming photons and then invents most of its color, because that is what produces a picture a person finds convincing.
There is nothing sacred about three channels. Materials have spectral signatures that spread across bands no human cone responds to. A system trying to distinguish plastic from skin, or a healthy cell from a stressed one, or silicon nitride from silicon oxide, would rather have eight narrow bands in the right places than three broad ones tuned to a primate.
Frame rate carries the same fingerprint. Twenty-four, thirty, sixty, one hundred twenty — these are descendants of flicker fusion, the frequency above which human perception stops noticing that a light is being switched on and off. Machines have no flicker fusion threshold. Nothing in the physics of a moving object requires that it be sampled at uniform intervals, and uniform sampling is precisely the wrong strategy for a scene where almost nothing changes between frames and the thing that matters changes very fast.
Then the image signal processor applies gamma encoding to match perceptual lightness, white balance to satisfy human color constancy, noise reduction, sharpening, and lossy compression. Every one of those stages is tuned against a human aesthetic judgment. Denoising in particular removes high-frequency structure on the assumption that it is not signal — an assumption made on behalf of an observer who would find it ugly, not an observer who might find it diagnostic. Networks trained on raw sensor output rather than processed output frequently do better on downstream tasks, which is a polite way of saying the pipeline is destroying evidence.
Aberration Is Not Noise
Lens design carries the deepest version of the inherited assumption. A lens is optimized to minimize aberration, which is to say, to produce an image a human would judge sharp, flat and undistorted. That objective treats the image as the product.
If the image is instead an intermediate code that a model will decode, aberration stops being a defect and becomes an encoding scheme. Chromatic aberration encodes depth in the structure of the blur. A coded aperture encodes range in the shape of the point spread function. Wavefront coding deliberately degrades the image in a known, invertible way so that more information about the scene survives the optical path than a clean lens would have allowed through. In every case the optics are doing computation, for free, at the speed of light, before a single electron is read out.
This is the substantive version of the argument that vision should be redesigned for machines. Not better sensors — a different division of labor between the optics and the processor, in which the optical stack performs the first layer of inference and the sensor records a representation that would be worthless to a human looking at a screen.
The Energy Argument
The economic case sits in the data path. A conventional pipeline captures everything at a fixed resolution and rate, compresses it for a viewer who does not exist, moves it across a bus, decompresses it, and runs inference to identify the tiny fraction that mattered. At the edge, the energy cost of moving pixels off the sensor and into memory routinely exceeds the cost of the arithmetic performed on them. Every joule spent transporting a pixel that was always going to be discarded is a joule spent serving a design decision made for the benefit of the human visual system.
Compress at the point of capture — optically, or in silicon stacked directly beneath the pixel array — and the bus, the memory, and the thermal envelope all shrink together. That is a real and quantifiable argument, and it is the one most likely to drive adoption, because it shows up on a bill.
What Has Actually Shipped, and What Has Not
The history here is instructive and not particularly encouraging.
Event cameras are the purest expression of the idea. Each pixel fires asynchronously when its own brightness changes, producing microsecond temporal resolution and dynamic range far beyond a conventional sensor, with a data rate proportional to activity rather than to time. The concept is roughly fifteen years old, technically superior on paper for most of that period, backed by serious silicon partnerships, and still confined to niches. Dynamic range and latency turned out not to be the binding constraint in most applications.
Near-sensor computing has done better. Stacking a logic die directly beneath the pixel array and running inference on the package is now a shipping product rather than a research proposal, and it has succeeded precisely because it does not ask anyone to abandon the RGB frame. It keeps the familiar output and moves the computation closer, which is the incremental version of the same insight.
Metasurface optics have reached consumer devices, mostly for polarization and depth sensing rather than as replacements for conventional imaging lenses. Polarimetric sensors with on-chip wire-grid filters are standard equipment in industrial inspection. Short-wave infrared imaging, which sees through haze because scattering falls off sharply with wavelength and sees through silicon because silicon is transparent past roughly eleven hundred nanometers, is migrating from expensive indium gallium arsenide toward quantum dot films deposited on ordinary silicon wafers. Every one of these is a modality chosen because a machine wants it and a human cannot use it.
The pattern is consistent: the departures from human-centric design that succeed are the ones adopted inside industrial and scientific instruments, where the sensor was never a consumer part in the first place.
The Real Obstacle Is the Training Corpus
The standard objection to machine-native imaging is cost. Human-format sensors are cheap because smartphones consume them by the billion, and a specialized sensor competes against a part whose marginal cost has been ground down by a decade of consumer volume. That objection is correct and usually decisive.
But it is not the deepest one. The deeper problem is that the entire pretrained foundation of computer vision is human-format data. Every large vision model was trained on gamma-encoded, demosaiced, JPEG-compressed RGB images produced by human-serving pipelines and, in most cases, framed and selected by human photographers. A sensor that outputs eight spectral bands, or an asynchronous event stream, or a coded pattern that only inverts into a scene through a learned decoder, has no pretrained backbone to attach to. It has to fund its own data flywheel from zero, in a field where the primary competitive advantage of the last decade has been access to somebody else’s.
This is the hardware version of the bitter lesson, and it has been winning. A mediocre sensor feeding a very large model trained on very large amounts of human-format data has repeatedly outperformed an elegant sensor feeding a model trained on the small corpus its own novelty forced it to collect. Tesla stripping radar and ultrasonics out of its stack in favor of cameras and a bigger network is the most public instance, and whatever one thinks of the outcome, the reasoning was not stupid.
Where the Argument Holds
The case for redesigning vision around machines is strongest exactly where the human-format corpus provides no advantage and the physics is punishing. Semiconductor metrology, where the defect is buried under bonded layers and the inspection has to be non-destructive. Live-cell microscopy, where staining the sample to make it visible is the thing you are trying to avoid. Industrial sorting by material composition. Long-range electro-optical detection of small fast objects in conditions that defeat conventional optics.
In those domains there is no billion-unit consumer part to compete with, no pretrained backbone to forgo, and a customer who will pay for an instrument. That is where an AI-native sensor is a product rather than a thesis.
Everywhere else, the honest position is that the human eye’s design constraints are inherited, arbitrary, demonstrably wasteful, and still winning.
Leave a Reply