Okay, I've partially fixed the issue; I've found an easy way to compile OBS from source on Linux with hardware acceleration (NVENC) enabled, and now streaming (even at high CPU usage) at 1250kbps seems to no longer overload the system, although as NVENC works better at higher bitrates, I'll probably up it to 2000kbps and see what happens. In any case, I do now have a new upgrade path for OBS, which isn't too much more difficult than downloading and running the file from the official site anyway.
On the flipside, however, we still have the high CPU usage with the browser plugin, even if I unload the browser elements upon hiding them, but I have at least lowered the CPU consumption for OBS overall, which will help me in the long run for any other encoder overload issues. As far as things go, the software issue is mostly addressed with the bleeding edge version of OBS with NVENC enabled, but we still have the browser plugin consuming CPU like crazy, even after being compiled fresh along with everything else!
I haven't tested streaming a game as of yet but at least OBS doesn't get overloaded any more; my 1080 Ti should be powerful enough to stream Doom, Isaac, and any other low demand games relatively easily, but the issue still remains as to why all of a sudden, the plugin consumes my CPU so much, when it was fine before... I've got no idea how to even tackle that issue at present, especially as without using a browser source, I can't actually have overlays, alerts, etc. which really messes up my stream presentation!
Granted, I could just use Windows, but I'm honestly not a fan of Windows 10, which is why I'm on Linux in the first place! It does look like I don't have any options at this point, however, besides looking at what's going on with the sources, and also the browser plugin, because it is the sole reason OBS is even acting up at this point!
I'm going to completely remove the browser plugin and build OBS from scratch again to see what happens, just in case it just needs some sort of fresh install. One of the other things that using the build script also compiles with the browser and hardware encoding is WebSocket support, although I currently have no use for that. I'm hoping though that a new browser plugin update will eventually happen because right now, that issue is literally what's been stopping me from streaming this past week or so (since after my admin break), although I am hoping that I might be able to stream less demanding games from this point on.
Given the issues I've been having with doing a build of OBS on top of a current install, I'm now taking the more extreme step of compiling and installing OBS fresh to see if that fixes anything, although the disadvantage of that is that I have to build my scenes again, but equally it does allow me to examine every step of the process to see if anything is happening. I'm as of yet unsure how this will affect the addition of the browser plugin but at the same time, a fresh install can occasionally work wonders! If, for some reason this still doesn't work, I'm going to have to scale back my streaming even further for the time being, until I've given my PC a cleanout, upgraded the CPU and added some better thermal paste, as well as perhaps replacing the cooler, to see if that does anything. I also think that a fresh install of Windows 10 and Linux Mint 19.1 wouldn't go astray either.
I'm still at least able to stream Titan Quest and Isaac via my PS4, so it may be that all I stream for the next while is Titan Quest on Monday, and Isaac on Friday via my PS4, which at least means I can do twice a week, and it means that I have more time to think about the issue. Even so, the option of NVENC (hardware) encoding to OBS is very much welcomed because it will make certain things easier going forward, especially as I can reduce the strain of streaming on my CPU which isn't something that should be understated.
Unfortunately though, from what I can see, all that seems to have been done with the compilation of OBS is adding NVENC encoding, as there is an error with the Linux Browser plugin, although equally, I'm still not at square one as I do at least have NVENC, and technically speaking, the browser plugin wouldn't've been different to the current release anyway, so what I may do with my PC is at least do some testing streams for Barony, Doom, Isaac, and Baldur's Gate (granted, I'm not playing it on stream for now, but it's a good test of a slightly more demanding game).
Of course, if I push OBS through its paces on other games whilst outputting to the test stream, we can see if anything is affected, but I still think that cleaning the inside of the PC, as well as improving airflow, and upgrading the CPU is a path I need to go through! It does, however, seem that the WebSocket compilation does work. In any case, things will need to happen re my PC setup because it's clear that it's suffering from bottlenecks, but at least I have a better version of OBS that will pay off dividends in the future, though it does now mean that I'll always have to build from source as a result, as it's not too much more complicated than what I previously did.
The next stage now is to bring the browser stuff back in and see if things get fixed as a result of that; I'm not sure if there will be any difference to the quality of my stream overall, but I'm hoping it will increase something somewhere; I'm still stuck at streaming at 720p however, so don't expect a larger resolution just yet. After this, I'll be looking at finding a CPU to upgrade to at a decent price, along with a better cooler, and also getting some decent thermal paste; I'm going to at least redo the thermal paste on my CPU after also cleaning up my current heatsink to see if I can do something in the interim to reduce the CPU usage; NVENC definitely helps in that regard though, and when I get a better graphics card, we should really see things starting to pay off!
I'll next update on this issue at least a few days from now, as I want to consider my options moving forward, and providing near-instant updates are going to affect things, especially as I'm blogging as I'm doing things! Either way, I'm hoping that if nothing else, I can at least make better use of my hardware before I start upgrading more of it! RAM thankfully is one of the last things I need to worry about right now, as I need to get a new CPU, a larger hard drive (not an SSHD this time, especially as they max out at 2TB!), and eventually a new graphics card, but that's pretty far on the horizon at present; it may be that I get a new CPU/motherboard/RAM combo before my GPU upgrade happens, but the Gen 4 Intel platform still has many years left in it, especially if I get the i7-4790K at a good enough deal; it's certainly better than the Gen 1 Intel I was running with before!