Your best and easiest bet is to just download a standalone client off the forums that comes packaged with .bat or .sh files to load the client with your own custom JVM settings. As it stands, Ender's VIA steam (or most, clients for that matter on the workshop) would require a fair bit of work to get running with custom JVM options. In Ender's scenario his JVM arguments are loaded in through his own launcher file (
https://github.com/EnderWiggin/hafen-cl ... otstrap.hl ->
https://enderwiggin.github.io/hafen-steam/hafen.hl ). You'd have to modify his bootstrap file and have it load your own hafen.hl file with the custom JVM arguments. When loading through steam we end up having a launcher that loads a launcher and the JVM arguments will get lost along the way, so they need to be specified in a launcher file ahead of time.
EDIT :: if you're tech savvy enough and comfortable with java you can kind of get an understanding on what is going on here
viewtopic.php?f=49&t=76318