Hilary Duff was spotting heading into an apartment complex