With you don't need to worry about case when comparing strings. That can be a big benefit, and avoid converting everything to lower ( or upper) case to comapre for string equality.
The other place where this plays a part is sorting of strings.
will sort like the file list in Windows, but will sort like a Unix file list (all the upper case file names appear before the lower-case file names).
After reading the comments and the other answer, and thinking a bit more, I'd say is the way to go from point of view of consistency with the rest of the .Net Framework.
If dictionary keys etc. are case-sensitive regardless of the setting then using binary comparisons by default throughout your code is just being consistent.
VBA String Comparisons
All you then need to worry about is if, for a particular comparison, you need it to be case-insensitive and code for that.
If you go with then not only do you need to worry about whether or not you need a particular comparison to be case-(in)sensitive you also need to be aware of the default behaviour in the current context.
It then becomes an argument not of consitency with other languages, but of consistency with the framework you're developing to.