After the success of Noodle Extensions and Chroma, I still wanted to continue pushing the envelope. For that, the next step was to move beyond manipulating the in-game assets and instead allow mappers to import their own assets.
Thus, I began developing Vivify, a mod that allows mappers to compile an AssetBundle then load, instantiate, and manipulate custom made assets using in-game events. These assets include post-processing shaders, 3D models, audio files, textures, and pretty much anything else that can be compiled into an asset bundle.
By leveraging the power of asset bundles, Vivify grants mappers complete control over all the visuals in their map. Vivify also uses my Heck library allowing for animating using either the Unity editor or in-game events.
Github Repo(Currently private.)
When rendering post-processing effects to the camera, there can be many complications with different resolutions, switching between VR and desktop, handling multiple active shaders, and creating and using buffer textures. One of the main features of Vivify is its post-processing system which resolves many of these problems.
The main goal of the system was versatility. To that end, one can declare screen textures and Vivify will automatically create a texture with appropriate resolution and depth for either VR or desktop cameras. Any amount of shaders can then be applied with a priority system to copy color data around different textures (or alternatively no buffer textures) while modified by the shader.
To assist with these effects, there is also a system for defining the properties on the camera. An example of this is to enable the depth texture which is commonly used in many shaders. As a final bonus, Vivify also adds a system for declaring additional cameras that will cull any objects declared whose texture can be accessed from a global variable.
Complications
The first complication I came across was how to actually deliver the asset bundle files to the player when the map is loaded. All custom maps are hosted on the website Beat Saver, and many third-party applications depend on it's API such as in-game leaderboards and automatic map downloaders.
Unfortunately, the website does not allow maps to be uploaded with unspecified files within i.e. the asset files could not be packaged with the map. The solution to this problem was to have the map files alone be hosted on Beat Saver while the asset files would be hosted on a different server and automatically be downloaded when required.
To implement this, I added an auto-downloader into Vivify with multiple protections such as a checksum test to verify the correct file is being downloaded. With this, I can upload the assets to my server and verify that the player gets the required asset files while still having all the perks of using Beat Saver.
Image courtesy of Unity
Because Beat Saber is a VR game, that means that shaders need to compensate for this rendering method. In particular, Beat Saber (v1.29.1 and less) originally used the Single Pass rendering method and thus all shaders needed to be written to be compatible with this.
To further complicate this, Beat Saber (v1.29.4 and greater) updated to Unity 2020.3.16f1 which forced Single Pass Instanced rendering. This meant that all shaders now needed to support GPU instancing in order to correctly render. To even further complicate matters, the engine upgrade created a large divide in the user base. That means to satisfy all users, Vivify had to release to multiple game versions and additionally, all shaders needed to support BOTH rendering methods.
Luckily, Unity comes with many macros when writing shaders that can easily swap between variants for desktop, single pass, and single pass instanced. Although this added overhead, the end result are assets that can work across multiple versions of Beat Saber.
Beat Saber AudioLink
The VRChat communityhas countless talented Unity asset developers, and using Vivify, many of their works can now be leveraged in Beat Saber. Many assets use a system called AudioLink for audio reactive shaders.
Wanting this tech for Beat Saber, I ported the system as a standalone mod in Beat Saber. Other than being a straight-up port, my version also fetches some Beat Saber specific information like player username and theme colors.
While installed, any asset from any mod can make use of AudioLink and make their asset audio reactive. The mod itself only provides the globally accessible texture that drives the entire system.