It depends at what stage you are into your channel. If you're already established and have a few thousand subscribers and you don't communicate the name change to your current subscribers, then yes, you'll probably see a lot of impact negatively. However, if your channel is still new and beginning, and the title of the channel doesn't convey the channel niche or theme properly, and you change the name to something that conveys the niche better, it will more than likely help improve things.
A long channel name that's hard to remember can negatively impact your channel when growing, changing it to a shorter, easier to remember, and easier to brand name, will help the channel long term I believe.
And it's always a good idea to make a video explaining the change to make the transition a lot easier to work with and adapt with long term.
Draw With Jazza is a prime example. He started as Draw with Jazza, but over time his channel became less about drawing, and more about his creative personality. He was already known as Jazza, so he changed his name and it doesn't seem to have impacted him negatively.
View: https://youtu.be/JpzTuS4Vta8
(you can use this video as a bit of a case study)
But at the end of the day, you need to think long term, and If you're already thinking about it, it's pretty evident your original channel name probably doesn't fit you, and will fit you even less in the future. If the change helps communicate your future channel goal and niche better, is easier to read and remember, I'd personally suggest changing earlier, to get it out of the way so it's less of a transition later on. Losing 50 subscribers due to a name change (probably less than that realistically) is less of an impact, then losing 500 because of a channel name change in 5years time when people are wondering "why am I following this channel?"
It WILL definitely impact your metadata though, so that's the part I'd be concerned about.
