Hi,
Please make sure your site and it's plugins are updated to the latest version is 2.8.1.
1. By default if a non logged in user goes on to start a FREE course then, on clicking the start course button the login popup is triggered automatically.
2. For the paid courses, GO to wp-admin->LMS Settings->
ENABLE IT : Open Login popup for non logged in users when they click on take this course button
Please refer to this screenshot : http://prntscr.com/fgw601
So, When a non logged in user goes to a course page and clicks on to buy that course, the login popup will appear. After logging in he can checkout from the cart.
So, this will help you in achieving your requirement in another way.
Regards,
Vibethemes Support.