Disclaimer: I use rbenv rather than rvm for installing ruby and have not tested the suggested setup locally.
Rather than install bundler via apt, install it via rubygems such that it uses your rvm installed version of ruby.
(after `sudo apt purge bundler`)
```
gem install bundler
```
(don't run it with sudo)
This should set bundler up in such a way that the gems are installed to your rvm ruby location, where the current user should have writing permissions.
If this does not fix the problem, perhaps you can run `gem env` and add its output to your question such that we can see which version of ruby is being used and what it was installed with.