Create 2 custom registration forms
In one form set the user role as Student and in another form set user role as instructor : http://prntscr.com/n43cmo
Now, create a page using elementor or visual composer. Create two sections or tabs in the page and in one section add the Student registration form shortcode and in another set the Instructor registration form shortcode.
Publish the page.
Now, disable BuddyPress registration page from WP admin – Settings – General – anyone can register.
Now set the above createed page in WP admin – WPLMS – BuddyPress – Registration page.
Now, when users click on signup, they land up on the page which has registration forms for both student and instructor.
To use WPLMS, please make sure you are running WordPress 5.1. 1 or higher, PHP 7.1 or higher, and MySQL 5.6 or higher (for legacy environments also PHP versions 5.3. x will work but it is recommended to be on at least PHP 5.6 and preferably PHP 7.0
https://docs.wplms.io/installation/installing-wplms-on-a-fresh-wp-install#hosting_requirements