I understand exactly what you're saying and where you're going with this. You are correct.
James is, for some reason, conflating the two separate points you are making.
We humans do make a distinction between sins. Some are worse than others and our punishments for those sins vary in degrees.
But, God makes no such distinction (according to the Bible). All sin, even the least of it, is ****ing to the soul.