What you are describing is very important. Here is how. Contrary to what most people here think, YouTube is NOT a search index. YouTube is a Human Index. YouTube cares absolutely zip, zilch, nada about your video or how good your SEO is. What YouTube is watching is one thing, this is, how Humans interact with your video. When you first put a video out, YouTube does not know enough about how humans will interact with your video, and they will give it a little time to start gathering data by suggesting it to people they think might be interested in it. This is where your initial views come from. Once humans start interacting with your video, YouTube starts to analyze the data and decides whether your video deserves more promotion or less. In your case, it is less. This tells you that the Human Index is pushing your video down.
There are really only three things YT cares about. Here they are.
1. The Click.
2. Tha Stay.
3. The Return.
The Click is when people click on your video. The click is good, but it by itself is of little worth. The Stay is how long the clicker stays on your video. If the clicker is immediately clicking off, that is actually worse for you than if they did not click at all. The Return, is when after watching your video, the viewer then clicks on another one of your videos that has been suggested from your library.
Get people to click and watch a good portion of your video and want to watch more of your content. Do this, and nothing else you do will matter.