I cannot speak on Windows, but for the official JE OSRS client from the official Old School download page.
For mac you may simply open up a terminal and run:
Code:
sudo open -n -a PATH_APPLICATION
Drag the application to the terminal window to get the absolute path, that's usually easier. do that when I need another client open, for anything such as Runelite or the official client. I do this after I usually have one opened already.
I know sometimes runelite get buggy with me sometimes. Have you tried right clicking the client, clicking duplicate naming it osrs1 or something then opening that way? That SHOULD work