I’ve been interested for a while in the intersection of knowledge and confidence.
I’ve noticed that some people present themselves as experts, but on investigation really don’t know that much about the subject they are ‘expert’ in. Initially, I thought these people were over inflating themselves and a little dishonest, but having met some of them, and realised their genuine belief in their expertise, I think something more subtle is sometimes going on. And, for the avoidance of doubt, I believe I do this as well when it comes to subjects on which I have passing knowledge.
In contrast, I’ve also come across people with relatively deep subject knowledge, but little confidence in that knowledge and sometimes an inability to see the important ‘wood’ for the trees. I think I do this, too.
I think what may be happening is that the more aware you are of what there is to know about a subject, the more you know you don’t know, and the less confident in your knowledge of the subject you become. Obviously, the only way you can know the breadth of information about a subject, and therefore lack confidence in your small comparative knowledge, is by being fairly well versed in it – already on the way to a good level of knowledge! Compare this, though, with someone with a cursory knowledge of the subject, and no awareness of how much more there is to know. That person will be comparatively confident in their knowledge, and maybe even profess to be something of an expert.
How to overcome this? Probably, simply, by having a decent way to measure your knowledge. Surround yourself with people who know bits about it, and compare your knowledge with theirs. This is why scientific papers and conferences work, and why learning about random topics on the internet is problematic (it provides both lots of ‘experts’ and a very low barrier of entry to ‘expertise’).
I think this is also why you often find people running things and speaking at events who seem to lack deep knowledge. They have high confidence. And why it can be hard to find and assess actual experts.
The flip side of this is that sometimes, possibly even often, you don’t need deep knowledge. You really just need ‘enough’ knowledge. And ‘enough’ might not be very much if you’re presenting to an audience that knows even less. And ‘enough’ may be the amount needed to have the confidence to actually DO something. The lack of confidence related to the awareness of the knowledge you don’t have can be paralysing, and in this situation those with less knowledge and more confidence could be more effective. Often, especially in business, there is no ‘right’ answer, no matter how well you know a subject, so confidence can be the crucial and winning factor.
See also this very interesting summary of the Dunning Kruger effect.