I suspect a little confusion here. This is the only reference I can find to a viaduct:
Australian Inn, Accrington - another lost pub
My guess is the date is slightly out and the viaduct in question is none other than the railway viaduct and the rubble is what was left after construction. Or alternatively, our ex-penal colonist was an early ***** (previous word censored - substitute 'traveller') - who reverted to type and nicked the material off the viaduct site as it was being constructed in order to build his house of inebriation.
I wonder if he subsequently went into further property development and built Melbourne St, Sydney St and Adelaide St? There's definatly a surplus of Aussie linked names around central Accy.