Taylor Swift says her fans are the only reason why people care about her music