The exact same trends go round and round in web design too (and now apps).
At first things were square (because that was all the technology could do) then in the 2000s CSS exploded and everything went colour gradients and rounded corners, just because people could, then that became old-hat and everything went flat and square again, and then rounded came back (but without so many gradients)
Everything is cyclical.
They are incentivised because showing accurate results for what you asked for isn’t necessarily the best way to keep people on the platform.
By pushing certain types of videos, such as opinionated content or loud shouty videos for low attention spans, YouTube hopes to keep you engaged for longer than they would by being accurate.
There’s also a direct advertising reason to funnel certain types of video. YouTube creators earn different amounts of money for the same number of views depeding on what category (e.g. financial, gaming, writing advice, cookery etc) YT has auto-categorised your video as. We can infer from this that advertisers are willing to pay more money for ads in some categories than others, and therefore YT is directly incentivised to push those more lucrative categories in search results, even if they aren’t what you wanted.
Plenty of reasons why they want to mess with results.