Usually, when .net is really slow, the cause has been that I'm using a Microsoft library 'wrong'. The solution has been to learn how that library works internally, and work around it, rather than runtime level optimizations.
Although this is all good stuff to know, if you are here trying to fix your applications performance problem, this is probably the wrong tree to bark up.
Are you talking about things like using List<T>.Contains() instead of HashSet<T>.Contains()? I've been doing .net for some time, and your comment is not consistent with my experience.
That's totally true. For instance, I (almost) never think about garbage collection. For some kinds of development, I suppose not accounting for it can cause a complete failure.
If you're looking to improve performance, the best thing you can do is learn how to use Apache JMeter. Run it locally against your solution, with VS running the performance profiler (which is under the Debug menu). This will give you a nice click through graphical break down showing you which parts of your code are taking the longest.
MS SQL Server for instance comes with its own profiler.
You can look for slow queries, then try to understand why by getting the query execution plan. You can alternatively also look for queries with a high standard deviation, which can hint of queries that get slower as there are more rows associated with a user.
Once you have the execution plan, you can either optimize the query, optimize the schema (add indices), or simply optimize the application (e.g: narrow the scope of the query, reduce call count). That however, won't allow you to horizontally scale.
The only realistic way to horizontally scale is sharding. Replication will slow down your writes as more machines are added and that simply does not scale. Using the same database for everything and everyone is a lousy way to go. Eventually you will need to keep things on different machines at some point.
Then, don't put large blobs into a database and store just the blob IDs in your database. Use another data store for that. Analytics, logging, etc doesn't belong into an OLTP database. Use another database for that not your OLTP one.
I tried analyzing execution plans for a fairly large stored procedure. I only needed to do it a few times. Everytime resulting in the DBA informing me that they've already analyzed that before and it was understandable why I couldn't make any real headway.
Would you consider logging user actions, for reporting in website, as something that should be stored else where? In one case I can think of the table had about 7 million records, but nobody seemed concerned about it.
miniprofiler seems like it's only good when you want to target a very specific area and try to report while in production. I don't like the idea of mixing profiling in with the code.
Glimpse looks awesome though and I'm surprised they're not charging for it.
One tricky thing about databases is that the performance characteristics of a database running on local dev hardware versus production are quite different.
Therefore it's important to capture and analyze performance metrics _from production_. A really good commercial tool is SQL Sentry (http://www.sqlsentry.com/). You can, of course, patch together your own tooling from open source and homegrown solutions.
I had a quick look at that page but it mostly seems to refer to 'general' resources. I somewhat expected a guide to something breaking like hiphop was for PHP. Or maybe a CLR-machine-bridge or something.
On a only tangentially related note, I was surprised at just how little the big data/machine learning/data science fields intersect with the .NET ecosystem.
I don't know how many .NET developers do what I do, but I think a lot of us are doing Enterprise development, typically working with huge databases that have been around for decades. My company does a lot of work with DB2 on IBMi servers, replacing old green-screen applications with new WPF and web applications. It's not the big data you're taking about, but it's the old-school version of big data.
Although this is all good stuff to know, if you are here trying to fix your applications performance problem, this is probably the wrong tree to bark up.