How background removal actually works for official document photos

Background removal does not trace an outline around you. A segmentation model classifies every pixel as subject or background, and the widely used on-device models make that decision at 256x256 before the mask is scaled back up to your photo, which is why hair is the part that goes wrong ¹. The standard is stricter than it sounds: ISO/IEC 39794-5:2019 asks for a smooth, uniform background texture and colour with the boundary between head and background clearly visible around the entire subject ². Hair, neck, clothing and head coverings are not background, so segmenting the face is the wrong operation to begin with ². And both the US and UK tell applicants not to change the photo with software ³⁴, which is the line worth understanding before you edit anything. On-device tools like SpecSnap run the same class of segmentation model locally, so the photo never leaves the phone ⁵.

It is pixel classification, not cutting around the edges#

The mental model most people carry is scissors. Find the person, cut along the line, drop them onto white.

That is not what happens. A segmentation model looks at the image and assigns a value to every single pixel. Google's MediaPipe Image Segmenter, one of the standard on-device implementations, can return two kinds of output. A category mask is "a segmentation mask as a uint8 image, where each pixel value indicates the winning category value" ¹. A confidence mask is "a segmentation mask as a float value image", where "each float value indicates the level of confidence that it is part of a specific category supported by the model" ¹.

The difference matters more than it looks. A category mask has already committed: pixel 4,192 is you, pixel 4,193 is the wall. A confidence mask preserves the uncertainty, so a pixel that is 60% likely to be hair can be blended rather than forced into one bucket. That blend is what stops the jagged staircase edge along a hairline.

The person segmentation models themselves are deliberately small. MediaPipe ships a square variant at 256x256 input and a landscape variant at 144x256, both quantised to float 16, both outputting two categories: background at index 0 and person at index 1 ¹.

The model is much smaller than your photo#

Read those input sizes again, because they explain most real failures.

Your phone shoots at 12 megapixels or more. The segmentation model decides at 256x256, which is roughly 0.065 megapixels. The mask it produces gets scaled back up to full resolution before it is applied.

Every strand of hair thinner than one model pixel is therefore a guess made at a resolution where that strand does not exist. Scaling the mask up does not recover the detail, it just makes the guess bigger. This is why a background remover handles a clean jawline perfectly and then leaves a soft grey fringe or a chewed edge through wispy hair. The jaw is a large, high contrast, low frequency boundary. Hair is the opposite of all three.

Apple's Vision framework acknowledges the tradeoff by exposing it directly. Person segmentation offers three quality levels, accurate, balanced and fast, with accurate recommended for computational photography where you want the best result and are not racing a video frame clock ⁶. A document photo is exactly that case. You take one picture and you can afford the slower pass.

What the spec counts as "background" is not what you would guess#

Here is the part that quietly breaks naive implementations.

The German Federal Office for Information Security published a detailed review of facial image quality assessment for the EU Entry/Exit System, and it is precise about the definition. The background is "the scenery or the area in the image that is located behind the subject. Thus, hair, neck, parts of the upper body and clothing visible in the image, or head coverings are not part of the background, and consequently, the background is not obtained by a segmentation of the face" ².

That last clause is the whole point. If a tool detects your face and treats everything outside the face box as background, it is wrong by definition. Your hair is you. Your shoulders are you. Your hijab or turban is you. A face detector, which is a different and much easier problem, cannot give you a compliant background on its own. Our face detection explainer covers what that separate step does and does not measure.

The review is explicit that the better approach is whole subject segmentation, or matting, meaning the segmentation of salient parts of the image rather than the face alone ². Matting is the technique that produces soft, partial coverage values instead of a hard in-or-out decision, which is the same idea as MediaPipe's confidence mask.

What "uniform" means, and the number nobody publishes#

You will see confident percentages quoted for background uniformity tolerance. Treat them carefully, because the standards do not work that way.

The actual definition is qualitative. Uniformity means "the brightness and colours in the background do not vary much, in particular, on small spatial scales", and importantly, "small and gradual changes of brightness and colour in one direction over the entire image (e.g. from left to right) do not necessarily conflict with the uniformity requirement" ².

So a gentle gradient across the whole frame can pass. A hard shadow, a skirting board, or a patch of texture near your shoulder is the thing that fails, because it varies on a small spatial scale. That is the practical rule, and it is why our shadow removal guide tells you to stand away from the wall rather than to find a whiter wall.

For the documents that matter, ISO/IEC 39794-5:2019 requires a uniform background only for reference face images for machine readable travel documents, and it asks that the background "should have a smooth, uniform texture/colour and the boundary between the head and the background should be clearly visible around the entire subject" ². The older ISO/IEC 19794-5:2011 recommends for all frontal face images a plain background without texture containing lines or curves ².

That phrase about the boundary being clearly visible is the compliance version of the hair problem. A soft halo is not only ugly, it works against the one thing the standard asks the edge to do.

The measurement is specified. Almost nobody implements it#

There is a defined algorithm for scoring background uniformity, and its weaknesses are instructive.

Under ISO/IEC WD4 29794-5:2022 the method runs a face segmentation, then takes the region from the left edge of the image across to the subject's right ear and from the top down to the vertical centre, plus the mirrored region on the other side, and computes the entropy of the luminance histogram over the join of those two regions ².

The BSI review lists three problems with it. It assumes the ear location is known, but no facial landmark algorithm outputs ear landmarks, largely because hair covers them, so the position has to be interpolated. The regions can swallow voluminous hair or a head covering, which corrupts the score. And the regions only cover the upper half of the image, so non-uniform texture below the ears goes undetected ². We wrote about the same ear estimation problem from a different angle in the ear visibility piece.

Then the blunt finding: "No implementations could be found that assesses the uniformity of the background in facial images" ².

Worth sitting with. The industry has a written method for grading your background and, as of that survey, no shipping software that runs it. What actually judges your photo is a simpler automated check or a human at a desk.

The reason for the white background is partly obsolete#

One more thing the review says out loud. ISO/IEC 19794-5:2011 justified its plain background recommendation by pointing at face detectors getting confused by lines and curves behind the subject. The review's verdict is that "since 2001, the robustness and accuracy of face detectors has greatly improved", detectors now handle challenging in-the-wild images, and "the reasoning of ISO/IEC 19794-5:2011 seems outdated" ².

It still binds you. Conformance to ISO/IEC 19794-5:2011 is required by EU Commission Implementing Decision 2019/329, so the requirement stays live for the Entry/Exit System ². The surviving good reason is human: a uniform background makes it easier for an officer to inspect the reference image and compare it to the person standing there ². Our white background explainer goes further into the biometric side.

The rule that catches people out#

Now the awkward part, and you should read it before you run any background tool.

The US State Department tells applicants: "Do not change your photo using computer software, phone apps or filters, or artificial intelligence," and "Submit the original, unedited photo without filters or digital changes" ⁴. GOV.UK requires a photo "unaltered by computer software" ³.

Read carefully, because the rule is aimed at your face, not the canvas. Cropping to the required size and setting the background to the plain colour the spec demands are mechanical changes to the frame and the backdrop. They do not change how you look, and a plain background is itself part of the specification. What these rules target is retouching: skin smoothing, slimming, relighting, portrait mode blur, beautify filters. Border systems match your photo against a future live capture, so a face that has been smoothed or slimmed can read as a different face.

The practical consequence is that a tool which quietly improves your appearance on the way to removing the background is a liability. Check what your document actually requires if you are unsure, and see why passport photos get rejected for the rest of the list.

The colour itself varies by document. GOV.UK asks for a "plain light-coloured background" for a digital photo, and for printed photos a "plain cream or light grey background" ³⁷. Not white, which surprises people who assume every country wants the same thing.

Where the on-device version of this lands#

Doing the segmentation on the phone changes the privacy maths rather than the algorithm. It is the same class of model, running locally instead of on someone's server.

SpecSnap does it that way. Its site states that "Face detection and background removal use on-device ML models bundled with the app" and that "Face detection, background segmentation, and compliance checks all run locally", with the only network calls being for optional app updates ⁵. The pipeline it describes is the sequence this article has been walking through: face detection locates the eyes, head and chin, background segmentation cleans to white, and the result is cropped to the pixel spec for the document you picked, across 46 document types in 20 countries ⁵.

Two things follow from keeping it local. Your face, at passport resolution, never lands in someone else's storage, which is the argument in our on-device validation piece. And the compliance check happens while you are still holding the camera, so a shadow behind your shoulder that would fail the uniformity requirement shows up as a retake now rather than a rejection later. If you want the comparison across tools rather than the mechanism, the background removal tool roundup covers who does what.

Frequently asked questions#

Why does background removal always mess up hair?

Because the decision is made at low resolution. MediaPipe's person segmentation models take a 256x256 or 144x256 input and output a two category mask, background and person, which is then scaled up to your full size photo ¹. Hair strands finer than one model pixel do not exist at the resolution where the classification happened. A confidence mask, which stores a float per pixel rather than a hard category, softens the edge instead of committing to a hard line ¹.

Does the standard require a specific shade of white?

Not universally, and not as a single number. ISO/IEC 39794-5:2019 asks for a smooth, uniform texture and colour with the head to background boundary clearly visible around the whole subject ². The shade is set by your issuing country. GOV.UK wants a plain light-coloured background digitally, and plain cream or light grey for prints ³⁷.

Will a replaced background get my passport photo rejected?

The rules to read are about altering your appearance. The US prohibits changing the photo with "computer software, phone apps or filters, or artificial intelligence" and asks for the original unedited photo ⁴, and the UK requires it be unaltered by computer software ³. Cropping and setting the backdrop to the required plain colour do not change your face, and the plain background is itself part of the spec. Retouching does change your face, and that is the risk.

Can a tool just detect my face and clear everything else?

No, and this is a genuine technical distinction. Hair, neck, upper body, clothing and head coverings are not background, so "the background is not obtained by a segmentation of the face" ². Getting it right needs segmentation of the whole visible subject, or matting ².

Is there software that checks whether my background is uniform enough?

Less than you would hope. A method exists in ISO/IEC WD4 29794-5:2022, based on the entropy of the luminance histogram over two regions beside the head, but it depends on ear positions that no landmark algorithm reliably produces and it ignores the lower half of the image ². The BSI survey reports that "no implementations could be found that assesses the uniformity of the background in facial images" ². What you get in practice is the narrower check a capture app can actually make: SpecSnap validates the shot against the selected document spec on the device before you export it ⁵, which catches the shadow or the stray edge while a retake is still cheap.

Sources#