Netty IllegalReferenceCountException causes false upload failure despite successful upload

[RosiKyu]

problem

Related Issue
Related to #12486

Summary
When uploading templates/ISOs from local file via the UI, the upload completes successfully on the SSVM (file is written to disk, SSVM returns HTTP 200 "upload successful"), but a Netty IllegalReferenceCountException occurs during response handling. This causes the UI to display an error to the user even though the upload actually succeeded. The template eventually becomes "Ready" after background processing completes.

versions

CloudStack Version 4.20.2.0

The steps to reproduce the bug

1. Navigate to Images → Templates
2. Click "Upload Template from local"
3. Fill in the form and submit
4. Watch browser Network tab - upload returns 200 OK
5. Check browser Console - may show error despite 200 OK
6. Check SSVM logs - see Netty IllegalReferenceCountException
7. Wait - template eventually becomes Ready

Note: This issue is intermittent.

Expected Behavior
When the upload succeeds (HTTP 200, file written to disk), the UI should show success to the user.

Actual Behavior

• Upload succeeds (HTTP 200, "upload successful")
• File is written to disk correctly
• Netty throws IllegalReferenceCountException during cleanup
• UI may show error to user (HTTP 432 in some cases)
• Template eventually becomes Ready after background processing

Impact

• User confusion: User sees error but upload actually worked
• Duplicate uploads: User may retry, creating duplicate templates
• Wasted time: User troubleshoots non-existent problem

Root Cause Analysis
The IllegalReferenceCountException occurs in HttpUploadServerHandler.channelUnregistered() when trying to destroy the HttpPostRequestDecoder. The decoder's reference count has already reached 0, indicating it was already released elsewhere.
This is a reference counting bug in the Netty upload handler - the decoder is being released twice:

• Once during normal processing
• Again in channelUnregistered() cleanup

Evidence

Browser Network Tab - Upload Succeeded (HTTP 200)
Request Headers (all correct):

X-Expires: 2026-01-21T16:46:06.391Z
X-Metadata: 7d9tasmSFbD/i/a+7cIPVHUKLB5oZUTfHOf6nIwEo6gdgJ1IHcgXfSHe1TouKFGRARpVRJJyyfVX5fqGcsv14oYurocRB4KGOkuW6zfW3GZC3hxhM2vA5OjK5Q497cxlc/TkgD/BOYoyx3VS32w4jFFxfVty9YSrc2tx1/bo9kE4MFXbFiuGNHI799xK8WhEXiTuR/7iyxRqLM17+ql/JYxlPLSP7DWbVOMFwm7HACbmSSaUNyJhi2OiaOmmU0TC5jpezvLxXhVmDsl9Fe9/uhTG2PDtmd/d2L+QtQz8+zNqxwgrS8EJqJKsk7q7tPzoDFKWc7Qp0uYpPvMqpyd377kXA2qhKnphEKrB6rBvIHV0OaX071UMNSysfrYC+GOIi3smUK6p5x939GlNlVFI6kYM5pkVZ9AsuNzGii4SMDZc8UGYrYmhazT10+ye0IlAVw5cFu5hcwVmeGnK8O2xDQkHo1z5wFYxiqCzZSuVdrhk0jdV0ktZ0CAcQeZzjBblu4DWFCcdqRDHl+8SkeAwtUjV3ZDht0u4D1lQdgV4G/NeDPCPhkm61WhKcR4AHvq5pb1+bs9h/8+b2BkuhfNbdNCd/gjW2kMsl4KcbhQwB+3uQsxZ9srg+hKacPB2Cyd8kgdLzuC4zYBq8mv8pRg4gpj5fbVC5Vs7pYbzpDlfoeIAuCZS7pDJM4Jontlae/QoGGo5
X-Signature: zw+vnaAEiYNsjUfHov44KNZ/GBw=
Content-Length: 400621794

• Response:

Status Code: 200 OK
Body: upload successful.

• SSVM Logs (/var/log/cloud.log) - Upload Received and Processed

2026-01-21T15:46:06,551 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: SIGNATURE=zw+vnaAEiYNsjUfHov44KNZ/GBw=
2026-01-21T15:46:06,552 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: METADATA=7d9tasmSFbD/i/a+7cIPVHUKLB5oZUT...
2026-01-21T15:46:06,552 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: EXPIRES=2026-01-21T16:46:06.391Z
2026-01-21T15:46:06,552 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: HOST=10.0.56.181
2026-01-21T15:46:06,553 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) HEADER: CONTENT_LENGTH=400621794
2026-01-21T15:46:06,554 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) URI: uuid=6c80606a-8826-460f-88a8-c15ed3606a6c
2026-01-21T15:46:06,555 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) base directory: /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203
2026-01-21T15:46:06,611 INFO  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) BODY: FileUpload: name="file"; filename="linux-debian-11-x86_64-gen2-v1.qcow2"

• SSVM Logs - Netty Exception During Cleanup

2026-01-21T15:47:30,572 WARN  [storage.resource.HttpUploadServerHandler] (nioEventLoopGroup-3-6:[]) 
Decoder already destroyed 
io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:140)
    at io.netty.util.AbstractReferenceCounted.release(AbstractReferenceCounted.java:79)
    at io.netty.handler.codec.http.multipart.HttpPostRequestDecoder.destroy(HttpPostRequestDecoder.java:293)
    at org.apache.cloudstack.storage.resource.HttpUploadServerHandler.channelUnregistered(HttpUploadServerHandler.java:169)

• File Successfully Written to SSVM

root@s-1-VM:~# ls -la /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/
total 391236
drwxrwxrwx 2 root root        95 Jan 21  2026 .
drwxr-xr-x 3 root root        25 Jan 21  2026 ..
-rw-rw-rw- 1 root root 400621568 Jan 21  2026 c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
-rw-rw-rw- 1 root root       340 Jan 21  2026 template.properties

• Template Properties File Created

root@s-1-VM:~# cat /mnt/SecStorage/5f75abe3-3803-3595-89ad-52d23b3dea97/template/tmpl/2/203/template.properties
#
#Wed Jan 21 15:47:30 UTC 2026
uniquename=203-2-0cb72ddc-d73a-3f2c-9f1f-24af1d033391
qcow2.filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
qcow2.virtualsize=5368709120
description=test
qcow2.size=400621568
filename=c620128e-31ef-31ec-a04a-24104a89fdf0.qcow2
size=400621568
public=true
id=203
hvm=true
virtualsize=5368709120
qcow2=true

• Despite the error shown to the user, the template eventually became Ready:

{
  "id": "6c80606a-8826-460f-88a8-c15ed3606a6c",
  "name": "Debian 11",
  "isready": true,
  "status": "Download Complete",
  "downloaddetails": [
    {
      "datastore": "NFS://10.0.32.4/acs/secondary/ref-trl-10677-k-Mr8-rositsa-kyuchukova/ref-trl-10677-k-Mr8-rositsa-kyuchukova-sec1",
      "downloadPercent": "100",
      "downloadState": "DOWNLOADED"
    }
  ],
  "physicalsize": 400621568,
  "size": 5368709120
}

What to do about it?

In HttpUploadServerHandler.java, add a null check or try-catch around the decoder destruction in channelUnregistered()

Workaround

Ignore the error in the UI and wait - template will eventually become Ready
Refresh the template list after a minute to see actual status