You have the basics of the story.
I don't know CASA's current views but, going back to the 80s/90s the airworthiness engineers were dead against fuel ballast for light aircraft operations on the basis that the risk of operational necessity's pushing the operation into using the ballast fuel was too high. They had no qualms regarding heavy aircraft operations which always made me shake my head just a little.
I’m grappling with all this fuel as ballast not counting as ZFW stuff...
Nothing to grapple with .. fuel has naught to do with ZFW limits (unless the OEM were to include fuselage fuel in the ZFW. The only likely situation where this might occur is where an aircraft has an STC fuselage tank added and the designer/owner saw no purpose in revisiting the ZFW limitation considerations. I can recall one modification of my design, many years ago, where we did just that for a fuselage located auxiliary tank). Whether you are carrying fuel intending to burn it, carrying it for reserves, tankering to save on fuel purchase costs, or carrying for loading ballast .. the fuel has nothing to do with ZFW limits.
I can add 50 kg of ballast fuel, which say, pushes me above my ZFW, but that’s ok, because I’m still under my ZFW without fuel
Correct .. the aircraft has no idea what you are carrying whatever fuel load for and, I suggest, absolutely no interest, so that's fine. The fuel load doesn't come into consideration for ZFW.
but then this ballast fuel is not to be included as my fuel available for the flight.?
Absolutely the case. If you needed that bit of fuel to get/keep you inside the CG envelope, then you need it from start to finish for the flight. There is one case I can think of where this wouldn't be the situation and that is where there are multiple tanks and the CG goes out of limits and then back in again during flight in the absence of ballast fuel. In that case you might have a situation where the ballast fuel could be burnt at a later stage of the flight but that's not a common situation needing to be considered.
So in effect, the say 50kg of fuel I’ve loaded to balance my plane, will still be there when I land...
... unless you want to land outside your CG limits. That conceivably could cause you handling problems and, very definitely, could cause you some censure should CASA find out or, should you have an incident of note, cause you some legal difficulty/embarrassment at the subsequent Inquiry
