Mathjax not working in options of the questions

yes you are right the mathjax is not working with wplms .this plugin is doing something different than other plugins to render its shortcodes .

We would recommend you to using the beautiful math by jectpack : https://wplms.io/support/knowledge-base/math-equations-in-wplms/

or wp quicklatex plugin .

we have integrated with these plugins coz these work on the_content filter of wordpress .

Well beautiful math from jetpack is working fine on my local test site :

http://prntscr.com/k6d3bt

you need to use the latex in shortcode form rather than writing it in raw format :

for example :

[latex]i\hbar\frac{\partial}{\partial t}\left|\Psi(t)\right>=H\left|\Psi(t)\right>[/latex]

refer : http://prntscr.com/k6d3r6