Ayende @ Rahien

It's a girl

Profiler Usage Analysis

I have been doing some studying of how people are using the profiler, and it shows some interesting results.

  • Typical profiler session is :
    • NH Prof : 1:15 hours
    • Hibernate Profiler: 1:05 hours
    • EF Prof: 42 minutes
    • L2S Prof: 50 minutes
  • 83% of the profiler users have used it more than once. In fact, here is the # of usages:
    image
    So we have over 50% that use it regularly.
  • Most people use it predominately to view the statements executed:
    image
    This means that the reports are getting comparatively little attention.
  • The results per geographical location are also interesting:
    image

Comments

Set
08/17/2010 09:27 AM by
Set

I am surprised to see China and Belgium so "high" but I expected to see France in a lower position.

It is really interesting.

Louis Haußknecht
08/17/2010 09:38 AM by
Louis Haußknecht

Are you using a 3rd party tools such as EQATEC Analytics in the Profiler, or did you build your own analytics tool?

chrissie1
08/17/2010 09:43 AM by
chrissie1

Us Belgians do well for such a small country. Perhaps you should do a stat where you divide that number by the number of citizens ;-).

Davy Brion
08/17/2010 10:10 AM by
Davy Brion

I think it's a good idea to collect this information, since it's a good way to see where it makes to most sense to put effort into the product

but i was a bit surprised by this since i had no idea that all of this data was being collected... i suppose it's mentioned in the EULA somewhere (though nobody reads that), but i think it might be a good idea to make this fact a bit more explicit to people when they first start the profiler. Including an option to disable the collection of this data would be nice as well...

Benny Michielsen
08/17/2010 10:17 AM by
Benny Michielsen

Belgians sure love NHProf. I've used the profiler several times, including a demo for students.

Erik van Brakel
08/17/2010 10:56 AM by
Erik van Brakel

@Davy Brion: I was surprised as well. Not that I mind, but I didn't notice this anywhere either. Good practice would be to explicitly state this on first startup I guess.

Either way, interesting stuff! It's spot on for the way I use the profiler. Mostly for looking at the SQL being executed, to easily spot weird queries and select N + 1.

I'd say a report would be useful in larger (corporate?) environments with automated builds that are checked by another department/team. Easy to spot weird behaviour. Do you have any indication on what kind of developers use the profiler? I've got a feeling it's more single developers than companies.

Jak
08/17/2010 10:56 AM by
Jak

Completely agree with Davy. I'm not comfortable with this data being collected

James L
08/17/2010 01:41 PM by
James L

What's the most common database password?

Dmitry
08/17/2010 02:06 PM by
Dmitry

What James said. How do we know you are not capturing connection strings or query results?

jdn
08/17/2010 02:26 PM by
jdn

I agree with Jak and Davy. I'm glad I saw this before purchasing, will hold off until there is some clarification.

Are you doing this with RavenDB as well?

Abdu
08/17/2010 04:25 PM by
Abdu

Use a good firewall which asks for your permission whenever an app requests a connection to the outside. I use the free Comodo. This way you know when an app is calling out (and wonder why).

João P. Bragança
08/17/2010 04:39 PM by
João P. Bragança

Are these all usages? Or just those with paid licenses? I would like to see the breakdown of what demo vs paid customers are doing.

viscious
08/17/2010 09:40 PM by
viscious

I was also surprised you are collecting this info.

Even the "evil" MS asks permission before they collect this kind of data.

I think it would be sensible to explicitly state what you are collecting and why and give the user to turn it off the first time they run your app.

@abdu, lots of people in a corp. type environment where you can't change firewall policies.

Steve Py
08/18/2010 12:32 AM by
Steve Py

@viscious:

Even MS doesn't always ask permission before calling home. I remember when I was running a fairly intrusive/selective firewall I noticed after a Windows Update my MS Mouse Driver all of the sudden wanted to establish a link to call home.

(I did quite a bit of online gaming at the time and it was interesting to see what kind of "chatter" would be coming from community gaming servers. Most of it was 3rd party ladder/stats tracking packages, some of it wasn't :)

Still, I agree. Automatically collecting information about usage is somewhat evil, and in some countries may be illegal. The issue is that while you may defend that the information is harmless (and anonymous) the user cannot verify that and choose to opt out of sending such information. If you haven't already Oren, check with a lawyer.

z
08/18/2010 01:08 AM by
z

How about a little common courtesy before sniffing my backside?

Martin
08/18/2010 05:51 AM by
Martin

Wow! I'm surprised! This is not ok!

Registered User
08/18/2010 05:25 PM by
Registered User

Ayende, can you explain why do you do that without explicitely informing about that ?

I think it's really not ok. We pay you for your software, but not for these statistics. Tell us if this is being done only for trial users or for paid also !?

Also - I think you should make it an explicit option in configuration so the user can agree or not for this non-cool practice.

You just lost a bit of respect in my and mine colleagues eyes. Remember that it takes a time and a lot of work to gain some and it's extremely easy to loose it!

Registere User
08/18/2010 05:33 PM by
Registere User

Ok, I just read the licence agreement again.

Can you explaoin us WHERE is the information about spying on us being shown? I'm pretty sure that it wasn't included or attached when I was installing the profiler.

Abdu
08/18/2010 06:48 PM by
Abdu

@viscious - I meant if you can install a firewall on your own machine.. beside the corp's firewall.

Davy Brion
08/18/2010 07:32 PM by
Davy Brion

if it's not even mentioned in the EULA, then that is just plain wrong... i mean, mentioning it in the EULA, and not explicitly notifying the user of it is bad enough already. But if this data is being collected without any notification whatsoever, than that is just plain wrong and it needs to be changed.

Again, i realize how valuable and interesting this data is. But you simply do not collect data from a user with software that was installed on the user's computer without any acknowledgment of this happening.

If this is not fixed, i would like my testimonial for NHProf to be retracted from the nhprof.com website. I know that doesn't make any difference for your sales, but i don't want my name linked to this kind of behavior in any way.

Bob
08/18/2010 09:04 PM by
Bob

FYI - for those who dont get it - get with the program! Ayende is a wimp and is never going to respond to any of these comments. He only responds when it is in his best interest. I cant tell you how many times this has been the case across a broad spectrum of topics.

dadebaser
08/18/2010 09:30 PM by
dadebaser

How far can you push commercialization ?

Anon for now
08/18/2010 09:59 PM by
Anon for now

As a fellow ISV I can understand what motivates Ayende to do this. For a moment, consider that web applications can collect as much usage data as they want to - user have zero visibility into that process. Note that 'usage' data is very different to 'information' data. Knowing what parts of an application is used the most, usage scenarios etc are extremely useful to the vendor.

I agree that the user should be aware that data is about to be sent, and should also have the ability to see what is about to be sent, but....they shouldn't be able to stop it, apart from blocking the firewall or killing the process. Why.....well most users will always choose not to send based on a stupid knee-jerk reaction. How about helping the vendor improve their product?

I've seen similar behaviour in error reporting: allowing the user to choose not to send reports results in > 95% of errors never getting reported. This has an impact on product quality. Forcing users to send the report....zero users have complained because they understand what is happening. Maybe the same approach should be used for usage data collection: pop up a message showing the user what is included in the report and telling the user to click 'ok' to send the report...no cancel button. I'm sure users would be happy with that - their fears about what data is included are addressed, and if they have any intelligence they'll see the benefit in providing this data to the vendor.

Steve Py
08/18/2010 10:16 PM by
Steve Py

@Anon

The issue is that is trusting a third-party on faith. The line between "usage" and "information" is blurry, and the legal distinction changes country to country. For instance, if NHProf captures example SQL statements (I.e. to assess what kinds of queries Prof was issuing warnings on) and transmits those, that reveals schema information and I'm sure a lot of commercial businesses would have an issue with that. Yeah, that's a big "if" and I doubt Oren would not have captured that, but the problem is that the user has no easy way of verifying that, and regardless they should (and may legally be entitled to) be a) informed up front that this information is collected, and b) be given the option to opt out.

People's knee-jerk reaction to sending error reports is more often than not perfectly justified. It's good that MS offers to display that information, but if the information isn't purely plain text (I.e. does not contain any HEX etc.) I wouldn't advise people to send it in, and even then I would e-mail it in, not use any automatic service because that way I control exactly what gets transmitted. Does that package of information identify whether or not any of your software is unlicensed? Sure, companies like Microsoft are not that interested in individual license offenders, Today. Does it include any content from that Word Document or VS project that just crashed?

Davy Brion
08/18/2010 10:21 PM by
Davy Brion

@Anon

there is a difference between providing sensitive data through internet applications vs apps that are installed locally on your system. You typically expect that local apps do not spy on you without your approval. Whatever you do with web apps can be stored by the web app or pretty much everyone who's interested in it. But that is simply something you should know and accept in advance. A local application spying on you without your consent, well... that's just not done, plain and simple.

Grandma COBOL
08/19/2010 12:33 PM by
Grandma COBOL

It's easier to ask forgiveness than it is to get permission.

Anon Right Now
08/19/2010 01:44 PM by
Anon Right Now

What the fuck?

This is seriously messed up. What kind of information is this sending?

A NHProf Customer
08/19/2010 01:54 PM by
A NHProf Customer

Very very disappointed at this. Under no circumstances should applications send data back without the explicit consent of the user.

Alex Simkin
08/19/2010 02:12 PM by
Alex Simkin

NHProf doesn't send anything anywhere. It just opens back door and sits quietly until Ayende seize control of your computer and gets all information that he needs for his reports.

Ayende Rahien
08/19/2010 04:08 PM by
Ayende Rahien

Joao,

I don't have the breakdown of demo vs. paid, that data isn't being tracked.

Ayende Rahien
08/19/2010 04:13 PM by
Ayende Rahien

@Steve Py,

Um.. no. See the next post for exactly what I am tracking, and how I am tracking it.

Ayende Rahien
08/19/2010 04:33 PM by
Ayende Rahien

Offensive comment by a guy named "Bob" removed.

Jim
08/19/2010 05:36 PM by
Jim

Come on people. You learned of this because Ayende told you. You didn't discover it via your strict network monitoring which you would do if this were REALLY important. Maybe some of you just like to complain - you think?. Chill out. geez.

Rob Kent
08/21/2010 09:59 AM by
Rob Kent

@Steve Py, "Does it include any content from that Word Document or VS project that just crashed?"

By coincidence, Android apps now automatically upload crash stats and a stack trace to their server when your app crashes on someone's phone. My app is a speed dialer and in one stack trace I saw the information of the contact it was trying to dial.

In this case, it was merely somebody's name but let your imagination roam and you can imagine any kind of information ending up on Google's server and being visible to the developer (and, I'm guessing, Google's support staff). I don't think Google have thought through the implications of this automatic crash policy.

For example, a malign developer could write an app that deliberately crashed at some point and scooped up information to include in the exception. Without even asking the user for Internet permissions, they would get their data uploaded to the Android developer's console. They would only have to do it once per user and then turn the feature off.

Patrick Smacchia
08/24/2010 03:17 PM by
Patrick Smacchia

Nobody noticed that the big missing IT country is Japan?

I would say that the Japanese market seems to be very specific, because japanese developers are no good in English.

The question is: is it financially worth investing in a localized Japanese version of the site (like DevExpress does)

Comments have been closed on this topic.