Own Your Weights or Rent Them?