Note: This was inspired by this wiki page, which I used as a base for this guide, which has a lot of additional information for attempting to use Vortex in a Windows VM with Linux. As of 5/2/2022, an error was also fixed after publication after an error with linking instructions was pointed out by the user who wrote the below wiki page, for which they have my gratitude and I apologize to any readers for the error.
Vortex Mod Manager is a well-known modding program that Linux users find does not work well via Wine, and since it's one of the very few general mod managers, this can make gaming on Linux a pain.
However, thanks to the use of virtual machines, one can work around this by installing Vortex in a Windows virtual machine, connecting the host Linux machine to the guest Windows VM via shared folders, and be able to, with a few limits, manage mods with no hassle.
This process will require the following:
1. Any version of Linux with VirtualBox installed. Other VM software with similar features is also possible, but this guide is written with the use of VirtualBox in mind and presumes basic knowledge of both it and Linux. I used EndeavourOS (based on Arch Linux, but this will work fine in any major supported GUI-based Linux distribution). It is unknown at the time of this writing if this will work on a Steam Deck.
2. Any copy of Windows 7, 8/8.1, 10, or 11 in 64-bit for the virtual machine. They do NOT need to be registered, they will work fine even after a 30-day grace period except for a minor reminder to activate and disabling wallpaper changing. Since all we need Windows for is to run Vortex and a web browser, this is not a problem. Windows 7 or 10 64-bit is recommended; the former bare minimum because Vortex does not support versions of Windows below this. Vortex is a 64-bit application, so the Windows 7 or higher must be 64-bit, regardless of version or registration. It is perfectly legal to acquire a nonmodified Windows ISO of any of the above to run in a VM without activation, hence why this guide presumes you will not be bothering to activate them. This guide will NOT support cracked versions of any of the above since we have no intention of supporting piracy.
3. Any game supported by Vortex. We will be using Daggerfall Unity for the example. Vortex supports the Windows versions of all games by default but can be made to work with games with Linux native versions like DFU with some work, which this guide will explain.
4. The default browser in 7/8/8.1/10/11 is usually Internet Explorer/Edge. I recommend Firefox or any Chrome-based browser for use with the Nexus Mods website (Google Chrome, Brave, etc), and this guide will presume their use in the VM, as IE especially has issues with working properly.
First off, the VM and host operating system should be in good working order. The VM (especially if using Windows 7) need not be updated (Windows Update no longer works for Windows 7 anyway), just a basic install with Vortex installed. Since the only website we are deliberately visiting in the VM is the Nexus Mods site, there should be only a little security risk at most. If you are downloading mods from other sites, and you have reason to believe it would be inadvisable to download them in the VM, then acquire them via your host system and place them where appropriate later.
Next, once it is confirmed both are installed, go to the Shared Folders setup in VirtualBox and link the /home/(username) folder to the VM (not the /home directory itself, link to whatever your username subdirectory is). This is the ideal directory for Steam games on Linux (it has both .steam and the .wine directories in it as well) and modding of non-Steam games since it has generally open user permissions without needing root access. Confirm the Windows VM is reading your home folder by accessing "This PC" and opening the linked shared folder (which should appear below the C drive). If it opens like a Windows drive and is readable, it works.
NOTE: BE CAREFUL, ANY EDITS DONE TO THE LINKED LINUX FOLDER IN THE VM WILL BE DONE IMMEDIATELY TO THE LINUX HOST AS IF YOU ARE A ROOT USER.
Now, if you want to manage a Windows game with Vortex, with the above in place, just select the game to manage and have it find the folder manually for the game you wish to manage. The shared folder will be mounted at Drive Z: so navigate to the proper file path and set it.
A Linux game version is a bit different. Vortex is a Windows program, so using the Linux version of Daggerfall Unity (my copy was at Z:/Downloads/DF Unity) requires a bit of sleight of hand.
Specifically, it will look for Daggerfall Unity.exe like the Windows version. As long as you have some file named this in the directory of the Linux host (it can be a copy of the Linux run file renamed as an EXE), it will manage the game properly. You will NOT run the game from the VM, but you need to register an EXE file for the mod manager to know which folders to target with mods.
With this in mind, download mods via Nexus Mods in the VM and add them to the Vortex Mod Manager, which will require your account on NM to be logged in on the Windows VM. Once the mods are downloaded, activate them with the manager and make sure they activate by checking the Linux host folder where the mods should install. If it worked, you will see the installed mod files and can then run the game on Linux itself with the mods activated.
Some notes. You will need admin permissions/an admin account to use Vortex for sharing folders with Linux. Windows 7 VM users may get a warning this is dangerous from Vortex, but ignore this, I find it often fails to make a symlink connection to the Linux drive properly unless you are logged in as an admin on the Windows VM.
Another thing to keep in mind is that you must RUN the game itself in Linux (you could do so in the VM technically, but this is not advised for performance and stability reasons). For Bethesda games like Skyrim and Fallout 4 with the SKSE/F4SE loaders, you must run those in Linux via Steam, look up how to do so via Wine/Proton, but so long as all the needed files are there they and other DLL loaders should work fine. The mod manager, while it should not be used to run the game through the VM, it can be used to install the DLL loaders.
You can turn off the VM after handling all the mod management issues you need to, and while in the VM you should run middleware tools like the Bodyslide for Skyrim/Fallout to generate the needed files for the Linux install. I would recommend allocating as many resources as possible to the VM for the more intensive modding tool operations.
In conclusion, this guide is a workaround for using Vortex with as few hassles as possible while gaming on Linux until it works natively (when or if that happens). It is advised one be very careful and back up anything important on the Linux host prior. This may not work for all types of mods but should work for the mass majority that is otherwise platform-agnostic to install.