It matters, but not as much as you think. Is a very tricky thing, this whole SEO scoring game.
The first thing you have to understand is, as I have said many times on here, "Something is only special as long as it is rare." The thing about great SEO is, it is NOT rare. You and about ten million other Tubers have SEO scores of 100. I have videos with scores of 100 that have almost no views at all. You see, once something becomes commonplace, it loses the special appeal and scale becomes balanced and there is no advantage in it.
Let's pretend that SEO score is a runner that can run the 100 in 10 seconds and everyone else can only do it in 15. Well, that runner is special and will always win. But now, he has set a standard and everyone else trains until they can all run the 100 in 10 seconds. Now the race is even and he is no longer special. This is how it is with good SEO. Thanks to tools like TubeBuddy, now everyone has good SEO. The race is even again.
What I am saying is that good SEO is just the beginning of competing in a market where everyone has an SEO score of 100. It may not help you win, but without it, you are definitely losing.