Masks, alpha and edges: why edits fail on real uploads

The first bug report said the eraser tool was "inverted". A user had brushed over a coffee cup on a desk, asked for it to be removed, and got back a photograph in which the cup was untouched and the entire rest of the desk had been reimagined — different laptop, different notebook, a window that had not been there. The mask generation code looked correct on inspection, and it was the shortest possible expression of the intent:

# The version that produces an inverted edit
mask = Image.new("RGB", src.size, "black")
ImageDraw.Draw(mask).ellipse(brush_bbox, fill="white")   # white = "edit here"
mask.save(buf, format="PNG")

The operation behind that tool is inpainting, which regenerates a chosen region of an existing picture from the prompt while leaving every pixel outside the region as it was, and the region is named by the mask this code is building. The white-means-edit convention is real and widely used, which is what makes this so easy to write. It is the convention of several open-weight inpainting pipelines and of most tutorials. Several hosted APIs, including the ones most likely to be reached for first, use the opposite convention: the mask is an RGBA image — red, green and blue plus an alpha channel, a fourth number per pixel recording opacity, 0 for fully transparent and 255 for fully opaque — and the transparent pixels, alpha zero, are the region to replace, while opaque pixels are preserved. Feeding an opaque white ellipse on an opaque black field into such an endpoint gives it a mask with no transparency anywhere, and the behaviour then depends on the implementation — often the whole image is treated as editable except nothing, or the alpha channel is absent entirely and the endpoint falls back to editing broadly.

The correct version for that convention paints transparency into the region to be changed:

mask = src.convert("RGBA")
draw = ImageDraw.Draw(mask)
draw.ellipse(brush_bbox, fill=(0, 0, 0, 0))    # alpha 0 = replace this
mask.save(buf, format="PNG")                   # PNG, so alpha survives

Two details in three lines carry the whole thing. The image must be RGBA, since a mask converted to RGB has no alpha channel and therefore marks nothing. And it must be saved as PNG, because JPEG has no alpha channel at all and a mask silently re-encoded as JPEG anywhere in your upload path arrives fully opaque.

Verify polarity rather than reading about it#

Conventions differ between providers, between endpoints on the same provider, and occasionally between versions, so the durable answer is a probe that asserts the behaviour in your test suite. Send an edit whose prompt produces something unmistakable, then measure which pixels moved.

def probe_mask_polarity(src: Image.Image, mask: Image.Image) -> str:
    out = client.images.edit(image=to_png(src), mask=to_png(mask),
                             prompt="a flat block of solid magenta", n=1)
    a = np.asarray(src.convert("RGB"), dtype=np.int16)
    b = np.asarray(decode(out), dtype=np.int16)
    changed = np.abs(a - b).sum(axis=-1) > 24

    alpha = np.asarray(mask.convert("RGBA"))[..., 3]
    transparent_changed = changed[alpha == 0].mean()
    opaque_changed      = changed[alpha == 255].mean()
    return ("transparent-is-edited" if transparent_changed > opaque_changed
            else "opaque-is-edited")

Run it once against a 256×256 fixture with a centred circle, assert the result, and the day a provider flips the convention you find out from a red test rather than from a user whose photograph came back with everything except the coffee cup rearranged. The same probe, pointed at a different endpoint, is how you verify the second editing operation you ship: background replacement uses a mask covering everything except the subject, and confirming that the subject survives is exactly the same measurement with the regions swapped.

The mask that is the right shape in the wrong orientation#

Once polarity is right, the next class of failure is geometric. Endpoints require the mask to match the source in width and height, and a mismatch either returns a 400 — the bad_request class from chapter three, which no retry will fix — or, worse, is accepted after a silent resize that shifts every boundary by a few pixels.

The version of this that survives code review and then fails in production comes from phone cameras. A photograph taken in portrait orientation on most phones is stored as a landscape pixel array with an EXIF orientation tag instructing the viewer to rotate it — EXIF being the block of metadata a camera writes into the file alongside the pixels, carrying exposure, timestamp, location and, here, which way up the picture goes. Browsers honour that tag when displaying the image, so the user draws their mask on a portrait image, and the canvas exports a portrait-shaped mask. Server-side image libraries read the raw pixel array, so the source arrives as 4032×3024 landscape while the mask is 3024×4032 portrait. The dimensions are transposed, the request is rejected, and it is rejected only for users on phones — which is to say, not for the designer, and not in the test fixtures.

from PIL import Image, ImageOps

def load_normalized(path: str) -> Image.Image:
    img = Image.open(path)
    img = ImageOps.exif_transpose(img)     # bake rotation into the pixels
    return img.convert("RGBA")

Normalising orientation on ingest, before anything else touches the file, removes the whole family. The related check worth adding at the same point is an explicit assertion that mask and source dimensions are equal, raised as your own error with both sizes in the message, since the provider's 400 will not tell you which of the two was wrong.

Alpha arithmetic, and where halos come from#

The third class is the one that produces the symptom in the interview question: edges that look clean in the designer's tests and show a pale or dark fringe on real uploads. Part of that is compositing arithmetic and part is the matte itself, a matte being the per-pixel map of how much of each pixel belongs to the subject — the alpha channel again, now produced by a model rather than drawn by a user, and the thing a background replacement lives or dies on.

Compositing a subject onto a new background uses the straight-alpha formula, where each output channel is C_out = C_fg · a + C_bg · (1 − a). Straight alpha means the foreground colour is stored independently of its transparency. Premultiplied alpha means the colour channels have already been multiplied by alpha, so a half-transparent red pixel is stored as (0.5, 0, 0, 0.5) rather than (1, 0, 0, 0.5). Browsers composite premultiplied internally, and an HTML canvas round-trip through getImageData can hand you premultiplied values depending on how the data got there.

Composite premultiplied data with the straight-alpha formula and every partially transparent pixel is multiplied by alpha twice, so it is darkened toward black in proportion to its transparency. Hair strands at alpha 0.3 come out at 30% of their intended brightness, which reads as a dark outline around the subject. The inverse mistake — un-premultiplying data that was already straight — brightens the same pixels into a pale halo.

def unpremultiply(rgba: np.ndarray) -> np.ndarray:
    rgb, a = rgba[..., :3].astype(np.float32), rgba[..., 3:4].astype(np.float32) / 255
    out = np.zeros_like(rgb)
    np.divide(rgb, a, out=out, where=a > 0)          # a == 0 has no colour
    return np.clip(np.concatenate([out, a * 255], -1), 0, 255).astype(np.uint8)

The diagnostic that separates this from a matting problem takes about a minute: composite the same cut-out onto pure white and onto pure black. If the edge fringe is dark on white and invisible on black, the data is being multiplied twice. If the fringe carries the colour of the original background in both cases — a green rim from a lawn, a blue rim from a sky — the arithmetic is fine and the matte is the problem.

Why studio shots pass and phone photos do not#

A binary segmentation mask assigns every pixel to subject or background. Real edges are not binary. A single pixel along a hair strand contains perhaps 30% hair and 70% of whatever was behind it, and a matte that calls it "subject" carries that background colour into the new composite, which is the coloured rim described above. Glass, smoke, motion blur and out-of-focus edges have the same property across much larger regions.

JPEG compounds it. Phone photographs are 4:2:0 chroma-subsampled, which stores colour at half resolution in each dimension, so colour information bleeds across edges before your pipeline ever sees the file. A green background against dark hair leaves a green tint one or two pixels into the subject that no matte can separate, because the separation was destroyed at capture.

This is why the acceptance numbers split so cleanly by source. On 40 studio images — subject on a seamless backdrop, even lighting, sharp focus, minimal PNG-quality compression — a straightforward segment-and-composite pipeline was accepted on 38. On 120 real user uploads the same pipeline was accepted on 49, and the failures were concentrated: 6 of 33 images with loose hair, 2 of 11 with glasses, 1 of 7 with visible motion blur.

The pipeline that handles the real distribution has more stages than the demo did, and each one exists because of a row in that table:

Colour decontamination is the stage most often missing. Within a two-to-three-pixel band along the matte, estimate the old background colour from just outside the subject and subtract its contribution from each partially transparent pixel in proportion to 1 − a. It is a handful of lines and it removes most of what people describe as a halo.

What to say to the designer#

The shallow version of this conversation blames the model and proposes a stronger one plus "clean edges, no halo" in the prompt. Neither can work, and it is worth being able to explain why in one sentence: the compositing happens in your code after the model has returned, so no wording changes arithmetic that the model never participates in. Where a model does participate — generating the replacement background — it is not being asked about edges at all.

What the designer can supply is the thing that actually blocked progress, which is a representative fixture set. Forty studio images established a bar the product never had to clear. Ask for a hundred images that look like the traffic: handheld phone photos, tungsten and mixed lighting, loose hair, glasses, patterned backgrounds, motion, and JPEGs at whatever quality the upload path produces. With consent recorded, those become fixtures in the golden set from the previous chapter, and the acceptance bar is defined on them rather than on the studio set. That reframes the conversation from a quality complaint into a measurable target, and it usually surfaces a product answer as well — that images with loose hair against a busy background get a warning and a manual touch-up path, rather than a silently poor result.

The unsolved part is genuinely unsolved. High-quality automatic matting on unconstrained photographs remains an active research area, trimap-free methods still lose to trimap-guided ones on hair and transparency — a trimap is a coarse three-region map, definitely subject, definitely background, and an uncertain band between them, drawn by a person or a first-pass model to tell the matting algorithm where to concentrate — and no amount of pipeline work makes chroma-subsampled edge information reappear. A product built on this needs a defined failure path for the tail rather than an assumption that the tail will shrink to zero.

Everything from here is assembly. The job record and the bytes you own, the two moderation checks, the four failure classes, the cost measured per accepted image, references and seeds, the pinned version with its golden set, and an edit path that survives phone photographs are the pieces; the smallest useful first build is one operation, end to end, on twenty real uploads, with the acceptance rate written down before you start.