Surface Go 2 rear camera detected but black on FydeOS v22.1 – IPU3 CSI-2 timeout

[yiboll]

Sorry for the delayed reply;

The single good capture rules out both of my remaining theories. A miswired lane or a wrong crystal are static faults — neither can produce a correct image one time in ten. Tests 3 and 4 were worth running and both came back clean. The problem is state-dependent at stream start, which is a different and more tractable class of bug.

Answers to your four questions, checked against the source:

1. Correct — those two registers are written only to clear at init and never read. Patch attached, verified to apply to 6.6 and 6.12 unchanged, so 6.18 should be fine.
2. Better than you thought: 0x484d/0x484f are declared in the driver but never written to. You can drive them over I²C during a live capture with no kernel changes at all.
3. The three strings you see are protocol-layer errors. The ones you asked about come from a different register and a different table — and the fact that they aren’t appearing tells us the unrecoverable PHY and ECC bits are not setting.
4. Already answered by your own testing — linux-surface 6.18.7 is upstream for both drivers.

One new lead from your numbers: the binned mode’s default frame rate is exactly 120 fps, and your bad state brackets it. There’s also an inconsistency in the driver where the maximum frame-length value it advertises is wider than the register it writes to, so a large value gets silently truncated. Whether that triggers depends on scene brightness — which would make it intermittent. Worth reading registers 0x3808–0x380f and 0x3500–0x3502 during a good run and a bad run to compare.

Maybe you can validate RAW decoder on the front camera first? If it produces a recognisable image from the working sensor, you can trust it on the rear — that unblocks the boundary you’re trying to establish.

Agreed on stopping FydeOS-specific builds; you have far better tooling where you are now. Several of your measurements have independently reproduced mine, which is reassuring for both of us.

[sawdoctor]

Clean retest using dynamically resolved OV8865 /dev/v4l-subdev7: VTS truncation did not occur, and exposure-overrun is not required for failure. Runs 2–4 had sensor 1632x1224, HTS 1923, VTS 1246, exposure exactly 1238 (= VTS-8), yet processed frames remained empty and arrived around 15 fps. One reproducible anomaly remains: OV8865 HBLANK control reports min=max=291 but current value=624, while hardware HTS confirms actual HBLANK=291. 624 is the previous full-resolution HBLANK value, suggesting stale V4L2 control state across the mode switch. Will do some more tests later

edit
Dedicated startup watch on linux-surface 6.18.7 does not show a gradual/mixed transition: within ~50 ms of a 1920x1080 start the OV8865 was already stable at 3264x2448, HTS 3888, VTS 2470, with control and hardware exposure both exactly 2462, and remained identical across 15 samples. Despite that, repeated 1080p captures produced effectively empty NV12 buffers (~3078 gzip bytes) at ~5.32 fps. So correct sensor geometry/timing/exposure is not sufficient for successful output. VTS wrap and exposure-overrun are not required for the failure. The stale HBLANK control value (624 while range/hardware imply 291 in binned mode) remains reproducible, but actual sensor HTS is correct, so it may be control-state bookkeeping rather than causal.

[yiboll]

That HBLANK anomaly is a real driver bug and I can give you the exact mechanism.

hblank is the only read-only control in the driver that gets routed through the driver’s control handler, and the handler has no case for it, so it falls through to an error return. The range fields get updated first, then the value write fails — leaving min and max at the new 291 while the value keeps the old 624. Exactly what you saw. One-line fix:

+   case V4L2_CID_HBLANK:
+           break;  /* read-only, nothing to program */
    default:
            return -EINVAL;

The interesting part is why it’s inconsistent. That handler returns success early if the sensor is runtime-suspended, before it reaches the failing branch. So the identical call succeeds when the sensor happens to be powered down and fails when it’s powered up — decided by autosuspend timing relative to the previous capture. That is exactly the kind of thing that makes a fault differ between stream starts with no reboot in between, so it’s worth testing whether that one-liner shifts your good/bad ratio.

I don’t think a wrong HBLANK alone explains empty frames, but while it’s stale libcamera computes the sensor’s line length as 2256 instead of 1923 — 17% out — so the exposure logic is working from a wrong model of the sensor whenever it happens.

Two corrections: my frame-length truncation theory was wrong, your VTS of 1246 is well within range and you’ve shown failure without any overrun. And your exposure value of 1238 is exactly the computed maximum for that VTS, so the auto-exposure has simply railed trying to find light in an empty frame — a symptom, as you said, not a cause.

[sawdoctor]

I also identified the audible click when switching to the rear camera. Manually moving the DW9719 focus actuator (focus_absolute=800 then 0) reproduces exactly the same click. A rear capture immediately afterwards remained empty (~1407 gzip bytes), so lens/VCM movement itself does not recover the stream. The click therefore appears to be autofocus actuator movement, not the state transition responsible for the occasional good capture.

[sawdoctor]

I confirmed your HBLANK mechanism in the source, but the runtime-PM part does not appear to explain the intermittent camera behaviour here.

I repeated the test using the correct OV8865 runtime-PM device (/sys/bus/i2c/devices/i2c-INT347A:00).

With power/control=on, all three captures gave:

hblank=624
content=1404

With power/control=auto, I explicitly waited until runtime_status=suspended before starting each of another three captures. All three again gave:

hblank=624
content=1404

So the HBLANK bug itself is definitely real, and the one-line fix is still worth submitting on its own merits, but on this machine HBLANK appears to be unconditionally stale rather than varying according to pre-stream autosuspend state.

That also makes it unlikely that fixing HBLANK alone will change the good/bad capture ratio. Earlier in this same linux-surface 6.18.7 session I obtained a genuinely normal rear-camera frame while HBLANK was still showing the stale 291..291 value=624 state, so stale HBLANK is not sufficient by itself to cause the empty-frame failure.

The six PM tests above all remained empty at ~1404 gzip bytes, so changing the pre-stream PM state had no observable effect on image content either.

At this point the two tests I still think are worth doing are:

1. Try to reproduce the state change after using Plasma Camera, since one genuine good capture occurred after that sequence.
2. Scan every frame of a longer 200-frame capture rather than sampling only a few frames, in case there is a brief good window being missed.

So: your HBLANK diagnosis is confirmed, but the autosuspend explanation for why it would vary between stream starts is ruled out by measurement here.

[yiboll]

One thing that will help your 200-frame scan a lot: the kernel already knows how many bytes each frame actually delivered. The CIO2 driver compares the DMA’s byte count against the expected frame size and prints a warning when they disagree — but then hands the buffer to userspace marked “done” regardless. That’s why cam reports a clean capture even when nothing arrived.

So instead of gzip-testing frames, capture the kernel log alongside the run:

dmesg -w > /tmp/frames.log &

# run your 200-frame capture

grep -c "payload length is" /tmp/frames.log

One line per bad frame, silence for good ones, and the “received” number tells you whether a partial frame arrived or nothing at all. If there’s a good window in that run, this will show you exactly which frames it covers and how long it lasted — with no decoding and no patch needed.

Your VCM finding confirms something useful: the actuator moving means the I²C path and that power rail are genuinely working.

[sawdoctor]

I validated the direct CIO2 RAW harness against the working front camera.

Front OV5693 was configured directly through CIO2 port 1 at 2592x1944 ip3b, bypassing ImgU, using the same capture method as the rear.

All 10 front RAW frames contained dense real data:

frame size: 6469632 bytes
gzip: ~43.7–43.9%
non-zero bytes: ~6,447,200 per frame

So the direct CIO2 RAW capture method itself is definitely sound.

For comparison, the rear OV8865 test gave:

200 frames captured directly from CIO2 port 0
test_pattern=2 (Color bars)
0x5e00 read back as 0x80 twice during the live stream
200/200 RAW buffers completely zero-filled
payload warnings: 91
all 91: received 0

Assuming the dmesg watcher did not lose warnings, the remaining ~109 rear buffers therefore completed with the expected payload length, yet their memory contents were still entirely 0x00.

The front control rules out the RAW harness and generic CIO2 capture method. ImgU is also bypassed in both cases.

This seems to leave either the OV8865 MIPI/transmit side emitting zeros, or something specific to CIO2 receiver port 0/DMA producing zero-filled buffers.

I think your 0x484d MIPI lane-pattern test is now the right discriminator. Could you give me the exact register value/sequence you want used? I don’t want to guess the programming.

[yiboll]

On 0x484d — I can’t give you values for it. The kernel only knows its address; there are no bit definitions in any driver, and I don’t have the datasheet. I’d rather tell you that than hand you a guess to poke into a PHY register.

But your results have made that test less important than we both thought. 109 of your 200 frames delivered the correct payload size. The receiver can only report the exact expected byte count if it decoded the packet headers, line length and line count correctly — a desynced link gives you short or zero payloads, which is exactly what the other 91 did. So the physical link demonstrably works.

And your 0x5e00 readback is the important result: 0x80 is exactly correct for colour bars — enable bit set, pattern select zero. I verified that against the driver’s tables. So the sensor was correctly told to generate a synthetic pattern internally, it transmitted 109 correctly-sized frames, and every byte was zero.

That places the fault inside the sensor, between its pattern generator and its output stage — not the link, not CIO2, not ImgU, not the lens or exposure. Your front-camera control is what allows that conclusion, so validating the harness first was exactly the right move.

The test I’d run next, which I can specify precisely:

write 0x5e00 = 0x81    # enable + RANDOM DATA
write 0x5e00 = 0x83    # enable + BLACK  (negative control)

Random data is impossible to confuse with zeros. If 0x81 produces real data, something specific is zeroing the colour-bar pattern. If 0x81 and 0x83 produce identical output, the pattern selector isn’t being honoured at all and the sensor’s digital output is gated off no matter what we ask for — which would be the cleanest result yet.

Optionally, also read back 0x5000 and 0x5001 during the capture (expect 0x16 and bit 0 set). And if you want one more one-bit experiment, clearing bit 0 of 0x5001 disables black-level correction — I doubt it can zero a full-scale pattern, but it’s reversible.

[sawdoctor]

I did a full set of controlled tests today and there are several important results.

1. Direct rear CIO2 RAW capture confirms the all-zero state is present before ImgU

Current topology on 6.18.7-surface-1:

• rear OV8865: /dev/v4l-subdev7
• rear CIO2 raw node: /dev/video0
• front OV5693 CIO2 raw node: /dev/video1
• /dev/media0 = ipu3-cio2

For the rear I configured:

OV8865:
SBGGR10_1X10 / 1632x1224

CIO2:
1632x1224
Pixel Format: ip3b
Bytes per Line: 2112
Size Image: 2585088

I captured 20 frames directly from /dev/video0, bypassing ImgU completely.

Every RAW frame was exactly 2,585,088 bytes, but every single frame was literally all 0x00:

gzip = 2540 bytes (~0.10%)
non-zero bytes = 0

So ImgU is not the sole source of the black output; in the bad state the data is already empty at the CIO2 RAW node.

2. CIO2 payload accounting in the bad state

In a separate 200-frame rear capture I logged the CIO2 payload warnings.

Result:

104 / 200 frames produced:

payload length is 2585088, received 0

All 104 short frames were exactly received 0; there was no spread of partial payload lengths.

The other 96 frames produced no payload-length mismatch warning.

Because cio2_buffer_done() still returns the buffer as DONE even when the received byte count does not match the expected payload, this explains why userspace can get apparently normal completed buffers even when CIO2 reports zero received bytes.

3. Rear sensor internal colour-bar generator was definitely enabled in hardware, but RAW output stayed completely zero

I then ran a live direct RAW capture from the rear CIO2 node and enabled:

test_pattern = 2 (Color bars)

while the stream was demonstrably still active.

I read the actual OV8865 hardware register twice during that same live stream:

0x5e00 = 0x80
0x5e00 = 0x80

So this was not merely the V4L2 control reporting value 2; the actual sensor register had the correct colour-bar enable value while streaming.

During that exact 200-frame capture:

frames captured: 200
completely zero frames: 200
frames containing non-zero bytes: 0

CIO2 logging for the same capture:

payload warnings: 91
91 received 0

So the OV8865 internal colour-bar generator was genuinely enabled while the sensor was actively streaming, yet all 200 direct rear CIO2 RAW buffers were still literally zero-filled.

Assuming the dmesg watcher did not lose warnings, the remaining 109 buffers completed without a payload-length mismatch, despite their memory contents also being entirely zero.

4. Front-camera control validates the RAW capture harness

I then ran the same direct CIO2 RAW method against the working front OV5693.

Front configuration:

2592x1944
ip3b
Bytes per Line: 3328
Size Image: 6469632

Captured 10 frames directly from the front CIO2 raw node.

All 10 contained dense real data:

gzip: ~43.7–43.9%
non-zero bytes: ~6,447,200 of 6,469,632 per frame

For example:

00: gzip=2833456 (43.80%) nonzero=6447419
01: gzip=2832096 (43.78%) nonzero=6447289
…
09: gzip=2841391 (43.92%) nonzero=6447404

So the direct CIO2 RAW capture method itself is sound. The same harness reads real raw data from the front port and all-zero data from the rear in the bad state.

5. Attempted RANDOM test after reboot was invalid because the register write did not stick

Later I tried the proposed:

0x5e00 = 0x81 (random data)

while streaming.

The stream was active, but immediate hardware readback was:

0x5e00 = 0x00
0x5000 = 0x16
0x5001 = 0x01

So I am not drawing any conclusion from that RANDOM test because the pattern register did not actually change.

6. After a hard reboot, the rear unexpectedly entered a genuine working state again

sudo reboot did not complete cleanly, so I had to hard-power-off and manually select:

6.18.7-surface-1

from GRUB.

After booting, the camera topology was unchanged.

I reconfigured the rear RAW pipeline to 1632x1224 as above.

Something important had changed:

• rear RAW streaming was now running at a normal ~30.06–30.07 fps
• payload warnings during the 100-frame RAW run: 0
• HBLANK now reports the correct binned-mode value:

horizontal_blanking: 291

instead of the stale 624 value I had repeatedly seen in the bad state.

I then ran a normal rear libcamera capture:

cam -c 1 --capture=10 --stream width=1280,height=720,pixelformat=NV12 …

The frames transitioned from startup/partial data to fully populated real image data:

frame 0: 0.00% non-zero
frame 1: 33.33%
frame 2: 33.33%
frame 3: 69.60%
frame 4: 69.58%
frame 5: 69.60%
frame 6: 97.84%
frame 7: 96.72%
frame 8: 69.69%
frame 9: 97.96%

I decoded the frames.

Frame 6:

• coherent real rear-camera image
• strong green cast

Frame 9:

• completely normal coherent colour rear-camera image

I also decoded only the Y/luma plane from frame 6:

• normal coherent greyscale image

For the good frames:

000006:
Y nonzero = 891806 / 921600 = 96.77%
UV nonzero = 460800 / 460800 = 100%

000009:
Y nonzero = 893456 / 921600 = 96.95%
UV nonzero = 460800 / 460800 = 100%

So the rear camera unquestionably returned to a genuine working state on the same kernel/hardware.

7. Known-good OV8865 register snapshot

While the rear was still producing real images, I started a live rear stream to keep the sensor powered and captured a read-only register snapshot.

Known-good controls:

exposure = 1238
horizontal_flip = 0
vertical_flip = 0
vertical_blanking = 22
horizontal_blanking = 291
analogue_gain = 1536
link_frequency = 360000000
pixel_rate = 288000000
test_pattern = 0

Known-good hardware registers:

0x0100 = 0x01

0x0300 = 0x02
0x0301 = 0x00
0x0302 = 0x4b
0x0303 = 0x00
0x0304 = 0x03
0x0305 = 0x01
0x0306 = 0x01
0x0308 = 0x00
0x0309 = 0x01
0x030a = 0x00
0x030b = 0x02
0x030c = 0x00
0x030d = 0x4b
0x030e = 0x00
0x030f = 0x09
0x0310 = 0x01
0x0311 = 0x00
0x0312 = 0x01

0x3018 = 0x72
0x3019 = 0x00
0x3020 = 0x93
0x3021 = 0x23
0x3022 = 0x01
0x3031 = 0x0a
0x3032 = 0x80
0x3033 = 0x24
0x3106 = 0x01

0x3808 = 0x06
0x3809 = 0x60
0x380a = 0x04
0x380b = 0xc8

0x380c = 0x07
0x380d = 0x83

0x380e = 0x04
0x380f = 0xde

0x3820 = 0x00
0x3821 = 0x61

0x4837 = 0x16
0x4850 = 0x10
0x4851 = 0x32

0x5000 = 0x16
0x5001 = 0x01
0x5002 = 0x08
0x5e00 = 0x00

The geometry is internally consistent:

0x0660 = 1632
0x04c8 = 1224
HTS 0x0783 = 1923 → HBLANK 291
VTS 0x04de = 1246 → VBLANK 22

I saved the full known-good register dump together with the proof images.

So at this point the same OV8865, on the same machine and same 6.18.7-surface-1 kernel, has demonstrated both:

• a reproducible bad state where direct rear CIO2 RAW buffers are entirely zero, including while the actual sensor colour-bar register reads back 0x80, and
• a genuine working state where normal coherent rear images are produced.

The front-camera control also proves that the direct RAW capture harness itself is valid.

Because the rear is currently working, I have stopped changing registers.

My plan is to wait for the rear to enter the black/all-zero state again, then take the identical register snapshot before rebooting or changing anything and directly diff GOOD vs BAD.

If there are any additional registers you specifically want included in that bad-state snapshot, let me know and I’ll add them.

[sawdoctor]

One final update before I remove the Mobian USB.

I also captured the rear directly from CIO2 again while it was in the confirmed WORKING state, using exactly the same 1632x1224 ip3b RAW path that previously produced all-zero buffers.

Working-state result, 20 frames:

• payload warnings: 0
• every frame ~95.5% non-zero
• every frame compresses to ~87% of its original size

Example:

00: nonzero=2468932 (95.51%) compressed=2253531 (87.17%)
…
19: nonzero=2468842 (95.50%) compressed=2254332 (87.21%)

So there is now a very clean same-machine/same-kernel/same-CIO2 comparison:

BAD state:
20/20 direct rear RAW frames = literally all 0x00

GOOD state:
20/20 direct rear RAW frames = dense real data (~95.5% non-zero), with zero CIO2 payload warnings

This reinforces that the fault is genuinely state-dependent rather than a problem with the RAW capture harness.

I also tested the front camera in the GUI while both cameras were working.

The front live preview is correctly oriented, but the saved still image comes out upside down. The front sensor reports:

horizontal_flip = 0
vertical_flip = 0
camera_orientation = Front
camera_sensor_rotation = 0

The saved MP4 also contained no rotation metadata (1280x720 only), despite the preview appearing correctly oriented. :contentReference[oaicite:0]{index=0} :contentReference[oaicite:1]{index=1}

There also appears to be a separate video-recording/output issue. The original 7.37-second 1280x720 H.264 recording produced decoder errors including:

• reference picture missing during reorder
• missing reference picture
• mmco unref failures
• non-monotonically increasing DTS

The file is still recoverable: I re-encoded it with ffmpeg and the repaired copy then decoded without errors. :contentReference[oaicite:2]{index=2} :contentReference[oaicite:3]{index=3}

I consider the upside-down saved front image/video and malformed H.264 recording a separate userspace/output issue from the OV8865 rear all-zero problem, but I wanted to record it while I had the Mobian environment available.

I have preserved:

• the known-good OV8865 register dump
• proof images from the working rear camera
• the known-good 20-frame rear CIO2 RAW capture
• its kernel log
• the original problematic video
• a repaired/rotated copy

I now need to remove the live USB, so I am going to stop testing here unless there is anything else you specifically want captured first.

[sawdoctor]

I now have a reproducible suspend/resume trigger and a useful control.

The rear was confirmed working immediately before closing the lid. I closed the lid, let the machine suspend, then resumed. The rear immediately returned to the broken state.

I captured the exact same OV8865 register set while broken and compared it against the known-good dump:

GOOD vs BAD register diff: exit status 0

So every measured OV8865 register is byte-for-byte identical between the proven working state and the proven post-suspend broken state, including PLL, timing, MIPI and ISP registers. HBLANK is also 291 in both states.

Direct rear CIO2 RAW after resume:

20/20 frames:
nonzero = 0 (0.00%)

Kernel payload warnings during the test:
24 x “received 0”

The kernel also repeatedly reports on CIO2 receiver port 0:

• DPHY synchronization error
• single packet header error corrected
• frame sync error

I then captured the FRONT camera directly from CIO2 port 1 without rebooting or changing the suspend state.

Front direct RAW after the exact same suspend/resume:

frame 0: 96.39% non-zero
frame 1: 96.04%
frame 2: 95.98%
frame 3: 96.39%
frame 4: 96.39%

So suspend/resume leaves the front OV5693 / CIO2 port 1 path carrying dense real RAW data, while the rear OV8865 / CIO2 port 0 path is completely zero and reporting CSI-2/DPHY errors.

Current reproducible sequence is therefore:

rear working
→ lid suspend
→ resume
→ rear fails
→ measured OV8865 registers unchanged
→ rear CIO2 port 0 gives all-zero RAW + DPHY/frame-sync errors
→ front CIO2 port 1 remains functional

This seems to narrow the problem considerably to the rear OV8865 / receiver-port-0 suspend-resume path or the sequencing between them, rather than a general camera/ImgU/userspace failure.

I have preserved the good and bad register dumps, direct RAW captures, kernel logs and the post-suspend front control.

[sawdoctor]

Large update from the Surface Go 2 / OV8865 investigation. I think the userspace-side elimination work is now about as complete as I can make it.

Environment:

• Mobian
• linux-surface 6.18.7-surface-1
• libcamera 0.7.2
• rear OV8865 = CSI-2 receiver / CIO2 port 0
• front OV5693 = port 1
• OV7251 IR sensor = port 2

=== 1. A genuinely GOOD rear state was captured ===

Earlier in the same session the OV8865 genuinely worked.

I had coherent real rear-camera images and also captured the rear directly from CIO2 using:

1632x1224
ip3b
BPL 2112
SizeImage 2585088

20/20 direct RAW frames contained dense real data:

~95.5% non-zero bytes per frame
~87.2% compressed size
0 CIO2 payload warnings

So the exact direct-RAW harness is proven to work on this machine/kernel.

During that GOOD state:

HBLANK = 291

and I saved a full OV8865 register snapshot including:

0x0100
0x0300-0x0312
0x3018-0x3019
0x3020-0x3022
0x3031-0x3033
0x3106
0x3808-0x380f
0x3820-0x3821
0x4837
0x4850-0x4851
0x5000-0x5002
0x5e00

Example GOOD timing/config:

0x0100 = 01
0x0302 = 4b
0x3018 = 72
0x3808/09 = 0660 = 1632
0x380a/0b = 04c8 = 1224
0x380c/0d = 0783 = HTS 1923
0x380e/0f = 04de = VTS 1246
0x4837 = 16
0x4850 = 10
0x4851 = 32
0x5000 = 16
0x5001 = 01
0x5002 = 08
0x5e00 = 00

=== 2. Suspend gave me a reproducible GOOD → BAD transition ===

While the rear was confirmed GOOD, I closed the lid and suspended the machine.

After resume the rear immediately returned to the broken state.

A 1280x720 NV12 sample gave:

frame gzip ~1408 bytes

i.e. the familiar effectively-empty output.

I dumped exactly the same OV8865 register set before rebooting or changing anything.

GOOD vs BAD register diff:

diff exit status: 0
good-vs-bad.diff size: 0 bytes

Every measured OV8865 register was byte-for-byte identical between the proven GOOD state and the proven BAD post-suspend state.

HBLANK was also still 291 while completely broken.

So stale HBLANK=624 is not necessary for the fault and cannot be the root cause.

=== 3. BAD direct rear RAW reproduces the original low-level failure ===

While still in that post-suspend BAD state I captured the rear directly from CIO2 again.

20/20 frames:

non-zero bytes = 0
0.00%

Kernel payload warnings during the capture:

24 x “received 0”

I do not assume the 24 warnings correspond one-to-one with the 20 userspace buffers, but every reported mismatch said received=0 and every captured RAW frame was entirely 0x00.

At the same time CIO2 port 0 repeatedly reported:

DPHY synchronization error
single packet header error corrected
frame sync error

=== 4. Front camera remains healthy after the same suspend ===

Without rebooting, while the rear remained broken, I captured the front OV5693 directly through CIO2 port 1.

5 frames:

96.39% non-zero
96.04%
95.98%
96.39%
96.39%

So the same suspend/resume leaves port 1 carrying dense real RAW while port 0 is completely zero.

=== 5. CIO2 port 0 is actually reprogrammed correctly in the BAD state ===

I read the CIO2 MMIO registers while the broken rear stream was definitely still active and PCI runtime state was “active”.

BAD rear / port 0:

ENABLE 0x00000001
NOF_LANES 4
SP_IF_CONFIG 0x00000000
LP_IF_CONFIG 0x00000000
STATUS 0x00000000
DLANE_HS varied between 0x00 and 0x50 in different BAD runs
DLANE_LP 0x000000
CLK settle 1343
D0 settle 1316
D1 settle 1316
D2 settle 1316
D3 settle 1316

So the expected 4-lane configuration and 1343/1316 timing were restored correctly.

Working front / port 1 was:

ENABLE 1
NOF_LANES 2
STATUS 1
CLK settle 1368
D0/D1 settle 1322

Initially STATUS looked interesting, but see the IR control below: a working port can also have STATUS=0, so STATUS is not a GOOD/BAD discriminator.

=== 6. TPS68470 software bookkeeping AND actual hardware registers checked ===

In a proven BAD rear state, Linux reported:

CORE = enabled, 1.200000 V
ANA = enabled, 2.815200 V
VCM = enabled, 2.815200 V
VIO = 1.800600 V, one user
VSIO = enabled, 1.800600 V

tps68470-clk:
rate = 19200000
non-zero prepare/enable counts

Because clk_summary is only framework bookkeeping, I then read the TPS68470’s actual register map through:

/sys/kernel/debug/regmap/i2c-INT3472:05/registers

The real PMIC hardware registers in the BAD state match the driver’s 19.2 MHz clock profile exactly:

06 = 01
07 = 03
08 = 02
09 = 03
0a = aa
0b = 20
0c = 01
0d = d7
0e = 00
0f = 0a
10 = 05

Against the driver’s 19.2 MHz entry:

XTALDIV = 170 → aa
PLLDIV = 32 → 20
POSTDIV = 1
POSTDIV2 = 1
BUCKDIV = 2
BOOSTDIV = 3

PLLCTL also has PLL_EN set, and CLKCFG1/2 match the expected output configuration.

The regulator/power registers also contain the expected programmed values.

=== 7. Rear TPS68470 reset/powerdown state is correct ===

Actual PMIC register:

SGPO 0x22 = 0x05

gpiolib reports:

gpio-7 powerdown out hi ACTIVE LOW
gpio-9 reset out hi ACTIVE LOW

So both rear-camera powerdown and reset are deasserted while the rear is BAD.

The OV8865 is therefore not simply being held in reset or powerdown.

=== 8. Most useful control: the OV7251 IR camera WORKS ===

The OV7251 is the other sensor using the TPS68470 clock provider/PLL, so I enabled its normally-disabled CIO2 port-2 link and captured it directly.

640x480 Y10 / ip3y
10 frames
3,993,600 bytes total

non-zero bytes:
3,688,089 = 92.35%

compression:
45.76%

So this is dense real sensor data.

The only port-2 kernel message was:

“escape mode ultra-low power state exit for clock lane”

The IR path otherwise worked.

I also read CIO2 port 2 while the working IR stream was active:

ENABLE 0x00000001
NOF_LANES 1
STATUS 0x00000000
DLANE_HS 0x00
DLANE_LP 0x000000
CLK settle 1254
D0 settle 1294

This also proves that STATUS=0 and DLANE_HS=0 do not by themselves indicate a failed receiver, because port 2 was producing real data with those values.

=== 9. Current three-port comparison ===

Port 0:
OV8865 rear
BROKEN
all-zero RAW
DPHY / frame-sync / header errors

Port 1:
OV5693 front
WORKING
dense RAW

Port 2:
OV7251 IR
WORKING
92.35% non-zero RAW

So two other CIO2 ports work, including the OV7251 using the TPS68470 clock system, while only the rear OV8865 / port-0 path fails.

=== 10. Recovery is NOT understood ===

The earlier GOOD state was real, but I have not been able to reproduce the transition back to GOOD.

After returning to BAD I tried:

• normal reboot → BAD
• full shutdown/power-off, left off ~1 hour, then cold boot → BAD
• forced power-button reboot → BAD
• different camera application → BAD
• another suspend/resume → BAD

All continue to give the same ~1405-1408 byte gzip empty-frame result.

Therefore the earlier recovery after a hard reboot was coincidental with some other state change; hard reboot should NOT currently be considered a recovery mechanism.

Suspend is established as at least one reproducible GOOD → BAD trigger, but BAD → GOOD is currently unknown.

=== 11. One invalid experiment to explicitly disregard ===

I tried unbinding/rebinding only the OV8865 driver.

The driver reprobed, but its media link did not get reconstructed:

ov8865 2-0010
1 pad
0 link
0 routes

and streaming then failed with:

“Link has been severed”

So that experiment tells us NOTHING about whether a sensor-only reset would recover the failure. Please disregard it as a recovery result.

=== Current elimination map ===

While the rear is definitely BAD:

• measured OV8865 registers: correct
• GOOD vs BAD OV8865 register diff: byte-identical
• HBLANK: correct
• OV8865 is streaming according to 0x0100
• TPS68470 PLL/divider hardware registers: correct
• TPS68470 regulator hardware registers: correct
• rear reset/powerdown GPIO state: correct
• CIO2 port-0 lane count/timing/configuration: correct
• front / CIO2 port 1: working
• IR / CIO2 port 2: working
• IR uses the TPS68470 clock system and transmits real data
• rear / CIO2 port 0: zero payload + CSI-2/DPHY errors

This appears to leave a fairly narrow remaining boundary:

1. hidden/internal OV8865 MIPI transmitter state not represented by the registers we’ve read;
2. the physical OV8865 → CIO2 port-0 CSI-2 link;
3. internal CIO2 port-0/DPHY state not visible in the registers we’ve inspected;
4. or sequencing/timing between those components.

I think I’ve exhausted useful read-only userspace diagnostics at this point.

I have preserved the GOOD RAW capture/register baseline and the BAD sensor registers, CIO2 dumps, PMIC regmap, kernel logs, front control, IR control and GPIO state.

If you want another diagnostic, could you point me at the kernel-side state or instrumentation you think would discriminate between the OV8865 transmitter side and CIO2 port 0? I would rather follow a targeted test now than keep changing state speculatively.

[sawdoctor]

Update on the Surface Go 2 rear OV8865 camera

I’ve done a lot more controlled testing on Mobian with 6.18.7-surface-1, and I think we’ve isolated two separate problems.

The rear camera can initially enter a very bad state where direct CIO2 capture from /dev/video0 at 3264×2448 returns a full-sized 10,340,352-byte buffer containing only 0x00, while the kernel reports hundreds of DPHY synchronization, CRC and packet-header errors. The front OV5693 on receiver 1 works normally on the same boot at ~28–30 fps with zero CSI errors, so this is specific to the rear OV8865 / receiver-0 path.

The interesting part is that I found a reproducible recovery without rebooting. I enabled the OV8865’s internal test pattern (test_pattern=2, colour bars), captured one direct CIO2 RAW frame, then disabled the pattern again. The colour-bar frame was valid and completely clean: 0 DPHY, 0 CRC, 0 header and 0 sync errors.

After that test-pattern/stream cycle, normal rear imaging immediately recovered as well. A 3264×2448 normal RAW frame contained real image data and had zero CSI errors.

I then tested all four combinations back-to-back:

full-normal 3264x2448 size=10340352 nonzero=10049038 uniq=256 csi_errors=0

binned-normal 1632x1224 size= 2585088 nonzero= 2513989 uniq=256 csi_errors=0

full-bars 3264x2448 size=10340352 nonzero= 6558192 uniq=8 csi_errors=0

binned-bars 1632x1224 size= 2585088 nonzero= 1641996 uniq=11 csi_errors=0

So the earlier failure is not simply because 1632×1224/binned mode is broken. Both full and binned modes work cleanly once the sensor has been recovered. Something about the test-pattern/stream cycle (possibly the associated sensor power/runtime-PM transition rather than the pattern register itself) changes the OV8865 from BAD → GOOD.

There also appears to be a second issue. Once the sensor is in the recovered state, libcamera at 1280×720 still produces exactly one frame and then stalls. Importantly, the original DPHY/CRC/header storm is gone, but the kernel reports:

ipu3-cio2 0000:00:14.3: payload length is 2585088, received 2588672

The capture then has to be interrupted manually.

This looks very similar to the OV8865 mode-programming/runtime-PM problem discussed in linux-surface, where set_fmt() can update the software state while the sensor is suspended, while ov8865_s_stream() only resumes the device and clears standby instead of explicitly calling ov8865_mode_configure().

One caveat: earlier direct I2C reads on this Surface did show the 1632×1224 mode registers actually programmed correctly in a failing state, so I don’t think that patch necessarily explains the original zero-data/CSI-error state. It may instead fix the separate one-frame 720p stall.

For FydeOS, my next test would therefore be:

1. Stop cros_camera_service.
2. Identify the OV8865 subdev and CIO2 receiver-0 node.
3. Enable the OV8865 colour-bar test pattern.
4. Perform one direct RAW capture.
5. Disable the pattern.
6. Start the FydeOS camera service again and test the rear camera.

If the same BAD → GOOD recovery happens on FydeOS, we may have a practical workaround and a much narrower place to look for the real initialization/power-state bug. Separately, the ov8865_s_stream() mode-programming patch looks worth testing for the lower-resolution one-frame stall.

The key new finding is that the sensor, MIPI transmitter, physical link and CIO2 receiver 0 can all carry completely clean real image data. The failure is state-dependent and recoverable, rather than the rear hardware simply being unsupported or dead.

[sawdoctor]

I’ve done some low-level testing of the Surface Go 2 rear OV8865 camera on FydeOS and can reproduce the failure without opening the ChromeOS Camera app.

FydeOS correctly detects the OV8865 on IPU3 CSI-2 receiver 0. On the first rear-camera stream after boot, libcamera cam configures 1280x720 NV12 and reports completed frames, but the resulting image buffer is entirely zero-filled. At the same time the kernel logs continuous ipu3-cio2 receiver-0 errors: DPHY synchronization errors, multiple packet-header errors and frame-sync errors.

Interestingly, setting libcamera TestPatternMode=2 causes the CSI error storm to stop completely. Switching the test pattern back off leaves the CSI link clean, but the camera image remains black.

I also discovered that this FydeOS libcamera build can expose RAW when requested alongside a processed stream:

1280x720-NV12 + 3264x2448-SBGGR10_IPU3

The native RAW buffer is the expected 10,340,352 bytes, but every byte is 0x00. There are no CSI errors once the camera has entered the cleaned-up state.

I repeated this with TestPatternMode=2; libcamera reports that the control is applied, but the 3264x2448 RAW buffer is still completely zero-filled.

So there appear to be two observable states:

initial state: zero RAW + severe CSI-2 errors

after stream/test-pattern cycling: zero RAW + clean CSI-2

I’m doing one final test now by setting the OV8865 V4L2 test-pattern control directly on the sensor subdevice before starting libcamera, to determine whether the libcamera TestPatternMode request is actually reaching the hardware.

Rear sensor is /dev/v4l-subdev7; CIO2 rear capture is /dev/video0.

I can provide the full cam and dmesg logs if useful.

[sawdoctor]

Further Surface Go 2 rear-camera investigation update (OV8865 / IPU3 CIO2):

I have now isolated the failure well below the Camera app/libcamera level.

Hardware/topology:

• Rear sensor: OV8865, i2c-INT347A:00
• Sensor node: /dev/v4l-subdev7
• CSI-2 receiver 0: /dev/v4l-subdev2
• CIO2 RAW node: /dev/video0
• Native RAW format: 3264x2448 SBGGR10_IPU3 / ip3b
• Native buffer size: 10,340,352 bytes

Direct V4L2/CIO2 capture:
I built a static helper which bypasses libcamera and configures:
OV8865 → CSI2 receiver 0 → /dev/video0.

STREAMON succeeds and full-sized RAW buffers dequeue normally.

However, every RAW frame is entirely 0x00:

• bytes = 10,340,352
• nonzero bytes = 0
• unique byte values = 1

Initially this direct capture also produced a large receiver-0 error storm:

• DPHY synchronization error
• multiple packet header errors
• frame sync error

Runtime-PM finding:
The actual OV8865 sysfs device is:

/sys/devices/pci0000:00/0000:00:15.2/i2c_designware.2/i2c-2/i2c-INT347A:00

Normally:
power/control = auto
runtime_status = suspended

Forcing the OV8865 to:
power/control = on
runtime_status = active

completely eliminates the DPHY/header/frame-sync errors.

However, RAW remains completely zero.

I then also forced the CIO2 PCI device 0000:00:14.3 to power/control=on.
With both OV8865 and CIO2 held active:

• no new CSI errors
• RAW still entirely zero

Register-level OV8865 verification:
I read the physical sensor registers over /dev/i2c-2 while streaming.

Chip ID:
0x300a = 0x00
0x300b = 0x88
0x300c = 0x65

Native mode is physically programmed:
0x3808 = 0x0c
0x3809 = 0xc0
0x380a = 0x09
0x380b = 0x90
=> 3264x2448

Timing:
HTS = 3888
VTS = 2470

MIPI:
0x3018 = 0x72
0x3019 = 0x00
0x3022 = 0x01
0x3031 = 0x0a
0x4837 = 0x16

Hardware colour bars:
V4L2 Test Pattern = 2
Physical sensor register 0x5e00 = 0x80

Streaming:
Before STREAMON: 0x0100 = 0x00
During STREAMON: 0x0100 = 0x01
After STREAMOFF: 0x0100 = 0x00

So the sensor really is being told to stream, native mode is genuinely in the silicon, and the hardware test-pattern generator is genuinely enabled. Nevertheless CIO2 RAW remains all-zero.

Full FydeOS libcamera control test:
With both OV8865 and CIO2 held runtime-active, libcamera dual-stream capture:

1280x720 NV12 + 3264x2448 SBGGR10_IPU3

also produced:

• processed NV12: known black frame
• RAW: completely zero, SHA256
  04573d9ad0f5d6d5f4580b7cda45fb9ef3b23b40a29301acd932aa37b22ff485
• no new CSI errors

Therefore this is not specific to my direct V4L2 helper, ImgU, or the normal Camera application.

Power/regulator debug:
Rear OV8865 suppliers show:

• i2c-INT347A:00-dvdd enable count 1
• i2c-INT347A:00-avdd enable count 1
• i2c-INT347A:00-dovdd enable count 1

The kernel clk_summary also shows a clock consumer for:
i2c-INT347A:00 / no_connection_id

I am now extracting the clock’s actual parent row/rate/enable counts.

Current interpretation:

1. Runtime PM clearly affects the bad CSI/DPHY state: keeping OV8865 active removes the transport errors.
2. It does not restore pixel data.
3. Keeping CIO2 active as well does not restore pixel data.
4. Basic mode programming, STREAMON and test-pattern programming are physically present in the OV8865 registers.
5. The remaining area appears to be sensor clock/PLL/MIPI transmitter/power sequencing or another lower-level OV8865/INT3472/CIO2 initialisation issue.

The known OV8865 mode_configure()/s_stream() issue may still matter for non-native modes, but it does not explain this native 3264x2448 failure because the physical mode registers are already correct.

I can provide the full raw capture/register logs if useful.

[sawdoctor]

Rear OV8865 is intermittent on Surface Go 2. The FydeOS Camera app can display a normal live rear image, but direct native RAW tests can enter a state where capture buffers remain completely untouched even though frame sequence numbers advance. This affects both MMAP and USERPTR, so it is not specific to the V4L2 memory model. A runtime suspend/resume of the sensor does not recover it and can produce CSI-2 receiver port 0: DPHY synchronization error. During a known working Camera-app state, OV8865 PLL/MIPI registers match the previously captured bad baseline; recent CIO2 logs also contain repeated inter-frame long packet discarded messages. The issue therefore appears intermittent in the rear OV8865/CSI-2 startup/link path rather than a fixed incorrect PLL setting.

[yiboll]

This is exceptional work, and I think you’ve already found the answer without quite naming it.

First, don’t chase SP_IF_CONFIG / LP_IF_CONFIG — the driver never writes those, so zero is normal.

The important line is inter-frame long packet discarded. That message only appears when the receiver has successfully decoded a packet of pixel data and then discarded it because it didn’t think a frame was in progress. So the link is working: signal locked, bytes aligned, headers parsed, checksums passed. What’s failing is the frame markers, not the data.

That one mechanism explains both of your contradictory symptoms. If the frame-start marker is missed, every line of pixel data arrives “outside” a frame and is discarded — nothing is written, which is why your buffers come back untouched rather than zeroed while the frame counter still advances. If the frame-end marker is missed, the next frame’s data spills into the current buffer — which is your received 2588672 overrun. Underrun-to-nothing and overrun are the same fault seen from two sides.

So the question is no longer sensor-versus-receiver. It’s short-packet handling on receiver 0.

For your bad-state snapshot, please add the MIPI backend block — nobody has looked at it. Offsets from the port base (port 0 = 0x000, port 1 = 0x400, port 2 = 0x800):

0x174 short-packet acceptance for VC0 expect 1
0x184 long-packet routing entry 0 expect 0x1580
0x16c packet stall status
0x104 backend status
0x124 backend interrupt status

Capture those on port 0 while broken, and on ports 1 and 2 in the same boot. You have two working receivers on the same chip, which makes that the strongest comparison available.

And your test-pattern recovery is the most practically useful thing here. The difference between your two environments suggests the recovery comes from the stream/power cycle rather than the pattern itself — on FydeOS the camera service probably still holds the device, so the cycle never completes. Stopping the service first, as you planned, is the right test. If it reproduces, we have a workaround we can ship while the real fix is found.

Thank you for the front and IR controls especially. Proving two other receivers work on the same silicon, in the same boot, after the same suspend, is what makes any of this conclusive.

[sawdoctor]

I tested the FBPT/DMA hypothesis during another confirmed BAD direct rear stream.

CDMAC0 stayed 0x4c00021f throughout: DMA_EN=1, DMA_HALTED=0.

CDMARI was not frozen. Ten samples showed the FBPT RP progressing:

0, 0, 1, 1, 1, 1, 2, 2, 2, 3

So the DMA engine is enabled, not halted, and actively advancing through FBPT entries.

The same capture remained completely BAD: all eight MMAP buffers were dequeued with advancing sequence numbers, but every 10,340,352-byte userspace buffer remained 100% unchanged sentinel data.

There were also no payload length ... received ... warnings in the kernel delta for that run.

Separately, the MIPIBE snapshot during a fresh BAD direct stream was fully correct: SP0=0x1, LP0=0x1580, MIPIBE_EN=1, status/IRQ/stall all zero.

So we can now rule out a halted/frozen FBPT read pointer. The interesting remaining question seems to be whether the FBPT/LOP entry being consumed actually points at the userspace buffer the driver associates with it, versus a dummy/stale/incorrect DMA target.

[yiboll]

That FBPT test was exactly the right one, and the result is the sharpest thing we have. DMA enabled, not halted, read pointer advancing, buffers untouched — and no payload warnings, meaning the hardware reported transferring the full expected byte count. Those can only both be true if the transfer went somewhere other than your mapped buffer, or never happened while the bookkeeping says it did. Your instinct about the LOP target is the right question.

One thing to skip: if you look at cio2_fbpt_entry_init_buf() you’ll notice it calls the “enable” helper with a pointer that the fill loop has already advanced, so the VALID bit appears to land on the wrong entry. I chased that and it’s fine — the dummy-initialisation path sets VALID on the correct entry beforehand and the buffer slot inherits it. Same code in every kernel version, and your working front camera uses it too.

There is a field that settles your question outright. Each FBPT entry has cur_line_num and frame_num, both written by the DMA engine itself, and the driver never reads either. So:

• if cur_line_num is 0, the DMA genuinely transferred nothing and the completion is fiction;

• if it’s around 1224 or 2448, the DMA wrote a whole frame somewhere — and your theory is confirmed.

Attached patch dumps those, plus the byte/page counts and the first physical page the DMA was pointed at. Comparing that address between a good run and a bad run tells us if the target moved. Applies to 6.6 and 6.12 unchanged, so 6.18 should be fine. One log line per frame, so keep the captures short.

Also worth noting: this run looked different from your earlier ones — clean MIPIBE, no payload warnings, untouched buffers, whereas before you had received 0 with a DPHY error storm. We may be looking at two separate faults that we’ve been treating as one. This instrumentation should tell them apart.
9006-DEBUG-cio2-dump-fbpt-dma-fields.patch (1.5 KB)

[sawdoctor]

I can’t see the attachment, but I checked the proposed instrumentation against the 6.18 source before reconstructing it. Two questions before I build:
entry[0].lop_page_addr appears to be the PFN of the LOP table, rather than the first destination data page; the first data-page PFN looks to be b->lop[0][0], while PFN_DOWN(b->lop_bus_addr[0]) is what entry[0].lop_page_addr should contain.
Also, cio2_fbpt_entry_init_dummy() / cio2_fbpt_entry_init_buf() don’t appear to clear cur_line_num, frame_num or num_of_bytes when an FBPT slot is reused. Would a completion-only dump therefore risk reading stale DMA counters? Does your patch also snapshot those fields when the buffer is queued, so we can prove they changed during that particular frame?
Could you repost the exact patch? I’d rather test your intended instrumentation than accidentally change the meaning while reconstructing it.