Skip to content

The vision pipeline, running

Three models in cascade, on real images. Every stage shows its decision and the number it rests on, from the initial sort through to the caption produced word by word. See the project page

  1. 01Sort: photograph, or not?
  2. 02Measure the noise, then denoise
  3. 03Describe, word by word
  4. 04See where the model looks

The model is trained on COCO, an English-language dataset: English is all it produces. Translating its output would distort the demonstration; you would be judging a translation, not the model. Everything else on this page follows the chosen language.

The full chain, image by image

Ten images run through all three models in sequence. Every stage shows what the model actually produced: its decision, the number that decision rests on, and how long it took.

How to read these numbers
Stage 1: photograph, or not?
The sorter's confidence, from 0 to 1. Above 0.5 it answers "photograph"; below that, it rejects the image. The closer to 0 or 1, the more certain the model is.
measured agitation
How much neighbouring pixels disagree, measured without any learning. A clean photograph shifts gently from one pixel to the next; noise creates abrupt jumps everywhere and drives the value up. Above the calibrated threshold, the image goes to the denoiser.
Generated caption
These scores compare the generated caption with five captions written by humans for the same image. BLEU-4 counts shared runs of four words: above 0.22 is good, and this architecture family tops out near 0.24. METEOR accounts for synonyms, ROUGE-L for the longest common sequence.
Stage 3: describe
Wall-clock time on the processor used to produce this demo. It is what rules out real-time use, and why this page is served statically.

surfboard

Example image: surfboard

Path taken

  1. Stage 1: photograph, or not?photograph64.7 %
  2. Stage 2: does it need denoising?below the threshold: image judged clean, nothing to domeasured agitation 0.0677 · trigger threshold 0.1052
  3. Stage 3: describe788 ms

Generated captionmodel output, in English

a man riding a wave on a surfboard

Click a word: the region the model was looking at while producing it lights up on the image.

BLEU-1
1
BLEU-2
1
BLEU-3
1
BLEU-4
1
METEOR
0.999
ROUGE-L
1

These scores all run from 0 to 1: 0, no word in common with the human captions; 1, the exact same sentence. Across the whole test set the model sits around 0.23 on BLEU-4, which is good for this family of models.

A score of 1 means the generated caption reproduces one of the five human references word for word. METEOR never reaches exactly 1: it weights matches by lemma and by synonym.

The five human reference captions
  • A person that is surfing in the water.
  • A man on a surfboard surfing a wave in the ocean.
  • A man riding a wave on a surfboard.
  • a surfer in a white shirt surfing on a sunny day
  • A man on a surfboard riding an ocean wave.

pizza

Example image: pizza

Path taken

  1. Stage 1: photograph, or not?photograph68 %
  2. Stage 2: does it need denoising?below the threshold: image judged clean, nothing to domeasured agitation 0.0578 · trigger threshold 0.1052
  3. Stage 3: describe1362 ms

Generated captionmodel output, in English

a pizza sitting on top of a white plate

Click a word: the region the model was looking at while producing it lights up on the image.

BLEU-1
1
BLEU-2
0.866
BLEU-3
0.7539
BLEU-4
0.6804
METEOR
0.7052
ROUGE-L
0.7778

These scores all run from 0 to 1: 0, no word in common with the human captions; 1, the exact same sentence. Across the whole test set the model sits around 0.23 on BLEU-4, which is good for this family of models.

A score of 1 means the generated caption reproduces one of the five human references word for word. METEOR never reaches exactly 1: it weights matches by lemma and by synonym.

The five human reference captions
  • A cheese pizza sitting on top of white paper and a large plate.
  • Cheese pizza is sliced into small pieces on a plate.
  • The pizza has been cut into tiny square pieces.
  • a pizza on a pan cut into squares
  • A delicious pizza sets on top of a plate.

sharp photograph

Example image: sharp photograph

Path taken

  1. Stage 1: photograph, or not?photograph89 %
  2. Stage 2: does it need denoising?below the threshold: image judged clean, nothing to domeasured agitation 0.0522 · trigger threshold 0.1052
  3. Stage 3: describe812 ms

Generated captionmodel output, in English

a group of people riding skis on a snow covered slope

Click a word: the region the model was looking at while producing it lights up on the image.

sharp photograph

Example image: sharp photograph

Path taken

  1. Stage 1: photograph, or not?photograph75.3 %
  2. Stage 2: does it need denoising?below the threshold: image judged clean, nothing to domeasured agitation 0.0482 · trigger threshold 0.1052
  3. Stage 3: describe930 ms

Generated captionmodel output, in English

a pizza sitting on top of a wooden table

Click a word: the region the model was looking at while producing it lights up on the image.

sharp photograph

Example image: sharp photograph

Path taken

  1. Stage 1: photograph, or not?photograph63 %
  2. Stage 2: does it need denoising?below the threshold: image judged clean, nothing to domeasured agitation 0.0333 · trigger threshold 0.1052
  3. Stage 3: describe690 ms

Generated captionmodel output, in English

a man in a red shirt is holding a skateboard

Click a word: the region the model was looking at while producing it lights up on the image.

noisy photograph

Example image: noisy photograph
before
The same image after the denoiser: noisy photograph
after denoising

Path taken

  1. Stage 1: photograph, or not?photograph80.6 %
  2. Stage 2: does it need denoising?above the threshold: denoising appliedmeasured agitation 0.1754 · trigger threshold 0.1052
  3. Stage 3: describe549 ms

Generated captionmodel output, in English

a group of people skiing down a hill

Click a word: the region the model was looking at while producing it lights up on the image.

noisy photograph

Example image: noisy photograph
before
The same image after the denoiser: noisy photograph
after denoising

Path taken

  1. Stage 1: photograph, or not?photograph68 %
  2. Stage 2: does it need denoising?above the threshold: denoising appliedmeasured agitation 0.2065 · trigger threshold 0.1052
  3. Stage 3: describe1507 ms

Generated captionmodel output, in English

a plate with a plate of food on it

Click a word: the region the model was looking at while producing it lights up on the image.

diagram

Example image: diagram

Path taken

  1. Stage 1: photograph, or not?rejected: not a photograph3.7 %
  2. the chain stops here

scanned text

Example image: scanned text

Path taken

  1. Stage 1: photograph, or not?rejected: not a photograph0 %
  2. the chain stops here

drawing

Example image: drawing

Path taken

  1. Stage 1: photograph, or not?rejected: not a photograph0.4 %
  2. the chain stops here

Scores on the test set

4,000 images from COCO val2017, never seen during training.

DecodingBLEU-1BLEU-2BLEU-3BLEU-4CIDERMETEOR
beam search0.64090.45690.32050.22910.60030.3936
greedy decoding0.62770.43470.28420.18670.5460.3697

How the model was trained

The captioner saw all of COCO 2017. After each pass over the data, how much it hesitates is measured on images it has NOT seen: that is the curve below. It falls while the model is learning, and rises once it starts reciting from memory. So the model kept is the one at the lowest point, not the last one.

Each point measures how many words the model hesitates between before writing the next one. It starts at 21 and falls to 14, out of a 10,296-word vocabulary: the lower the number, the more the model knows what to write.

how much the model hesitates on images it has never seenThe number of words the model hesitates between at each step, on images it has never seen, across 8 passes over the data. It falls from 21 to 14 at pass 5, then rises: the model kept is the one from pass 5.211814model keptbeyond this, the model recitespass over the data 18lower = the model hesitates lesscandidate words

The model

architecture
Frozen InceptionV3, Bahdanau attention, LSTM decoder
training
All of COCO 2017, 118,287 images, about 590,000 captions
vocabulary
10,296
passes over the data
Early stopping at the eighth pass, best validation at the fifth
inference hardware
NVIDIA GeForce RTX 4080

The twelve images of the full run

Within the training distribution

Categories present in COCO 2017, the dataset the model was trained on.

  • 3 correct
  • 4 partial
  • 0 wrong
  1. Photograph: golden retriever sitting on grass

    Generated captionmodel output, in Englisha black and white dog laying on top of a wooden bench

    partial · dog correct; colour, posture and surface wrong

  2. Photograph: bicycles parked on a street

    Generated captionmodel output, in Englisha person riding a bike down a street

    partial · object and place correct; action invented, the bicycles are parked

  3. Photograph: train at a platform

    Generated captionmodel output, in Englisha train that is parked on the tracks

    partial · the place is right, these are tracks; but no train is visible

  4. Photograph: pizza on a table

    Generated captionmodel output, in Englisha pizza sitting on top of a wooden table

    correct

  5. Photograph: cat sitting

    Generated captionmodel output, in Englisha cat is laying on a bed

    correct

  6. Photograph: double-decker bus on a street

    Generated captionmodel output, in Englisha double decker bus parked in front of a building

    correct

  7. Photograph: romanesco broccoli close-up

    Generated captionmodel output, in Englisha close up of a plate of food on a table

    partial · "close up" and "food" correct; no plate, no table

Outside the distribution

Subjects absent from the COCO categories.

  • 0 correct
  • 0 partial
  • 5 wrong
  1. Photograph: aurora borealis over a forest, at night

    Generated captionmodel output, in Englisha group of people standing next to each other

    wrong · no people, no group: an aurora over a forest

  2. Photograph: pollen grains under an electron microscope, colourised

    Generated captionmodel output, in Englisha bunch of bananas sitting on top of a table

    wrong · no bananas, no table: pollen grains seen under a microscope

  3. Photograph: oil painting, mountain landscape

    Generated captionmodel output, in Englisha group of people standing on top of a body of water

    wrong · hallucinates people

  4. Photograph: refracting telescope on its mount

    Generated captionmodel output, in Englisha fire hydrant sitting on top of a table

    wrong · no fire hydrant, no table: a refracting telescope

  5. Photograph: sand dunes, empty desert

    Generated captionmodel output, in Englisha group of people standing on a beach

    wrong · hallucinates people; mistakes sand for a beach

The model falls back on the nearest COCO scene for every unfamiliar image. On three of the six it describes people; none of the images contain any.

Image credits

All from Wikimedia Commons, under licences permitting redistribution.