Apparently, just upgrading the container version for Nextcloud is not enough. Over time, you might end up with an error like this:-

This version of Nextcloud is not compatible with > PHP 8.0

You need to be upgrading nextcloud as well within the container along with the image updates. You’ll need to exec into the container and run updater.phar 

Once you’re done with the update, re-run updater.phar until you get to the latest version of Nextcloud.

If you want to upgrade the PostgreSQL version, export the DB and then import it into the newer version of PostgreSQL.

There’s also a all-in-one container in the works that handles all upgrading internally: https://hub.docker.com/r/nextcloud/all-in-one

I will revisit the all-in-one container in a few days, but atleast for now I have successfully switched over to Nextcloud v26.0.1, and I love the new UI ! Thanks to everyone at Nextcloud for building such a great piece of tech.