ÜberProf and Continuous Integration

One of the features that keep popping up for ÜberProf is that people want to use that in CI scenarios, usually to be able to programmatically check that they don’t have things like SELECT N+1 popping up, etc.

With build 562 of ÜberProf, this is now possible. How does this works? We now have a command line interface for ÜberProf, with the following options:

/CmdLineMode[+|-]         (short form /C)
/File:<string> (short form /F)
/ReportFormat:{Xml|Html} (short form /R)
/Port:<int> (short form /P)
/Shutdown[+|-] (short form /S)

Starting up with no options will result in the usual UI showing up, but you have a new mode available for you. Let us say that you want to output the results of your integration tests into a format that you can easily work with programmatically. Here is how it can be done:

nhprof.exe /CmdLineMode /File:Output.xml /ReportFormat:Xml <-- starts listening to applications
xunit.console.exe Northwind.IntegrationTests.dll
nhprof.exe /Shutdown <-- stop listening to applications and output the report

The example is using NH Prof, but the same holds for all the other variants.

The way it works is very simple and should be pretty easy to integrate into your CI process. The XML output can give you programmatic access to the report, while the HTML version is human readable.

One thing that you might want to be aware of, writing the report file is done in an async manner, so the shutdown command may return before writing the file is done. If you need to process the file as part of your build process, you need to wait until the first profiler instance is completed. Using PowerShell, this is done like this:

nhprof.exe /CmdLineMode /File:Output.xml /ReportFormat:Xml
xunit.console.exe Northwind.IntegrationTests.dll
nhprof.exe /Shutdown
get-Process nhprof | where {$_.WaitForExit() } <-- wait until the report export is completed

Please note that from a licensing perspective, the CI mode is the same as the normal GUI mode. On one hand, it means that you don’t need to do anything if you have the profiler already. On the other, if you want to run it on a CI machine, you would need an additional license for that.

Print | posted on Thursday, December 03, 2009 8:11 PM

Feedback


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 8:31 PM Krzysztof Kozmic

is there a way to fail a build if XProf finds some issues were introduced?


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 8:35 PM Ayende Rahien

Krzysztof,
You get the XML, you can read it and decide to abort if the profiler has alerts to report.


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 8:36 PM Krzysztof Kozmic

ah ok.

I was thinking rather along the lines of non-zero return code


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 8:41 PM Ayende Rahien

Krzysztof,
No, I explicitly do not want to do that. There is too much subjectivity in there.


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 8:55 PM Ted Elliott

Just curious, how did you set up your project to perform as both a WPF or WinForm app as well as a console app? I've been wanting to do the same thing, but from what I've read it's almost impossible to do correctly in .NET.


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 9:07 PM Ayende Rahien

Ted,
It _isn't_ possible.
The issue relates to how windows decides how to execute, each EXE has a flag in its header that decide how this is done.
What I did was hook into the console subsystem and attach myself to the parent console. Not ideal, but it works.


Gravatar

# re: ÜberProf and Continuous Integration 12/3/2009 9:27 PM Jeff Brown

Sounds like it would be worth adding a /Wait argument to be used alongside shutdown. You might need to build a separate nhprof.console.exe application for that though, depending on how the exe is configured.

(I hope the nhprof startup isn't also asynchronous since that will create a potential for race conditions...)

The behavior that @Krzysztof wants could be provided by a separate report processing utility that has the option of returning different exit codes based on observed results. Alternately it would be useful to have some MSBuild or NAnt integration that returns statistics as optional output properties for subsequent handling in the build script.


Gravatar

# re: ÜberProf and Continuous Integration 12/5/2009 1:16 PM Frans Bouma

attaching to the parent console is indeed the only way to do it, but it does have side issues (like it's just emitting output to the console, the user can still interact with the console as if nothing happens, like entering commands and pressing enter, which made me decide to go for a console variant of my .exe instead of use the hack). Isn't it easy to do btw? your main logic is in a centralized dll anyway, not in the ui, so using a console variant isn't that hard and avoids the mess with attaching to a parent console.


Gravatar

# re: ÜberProf and Continuous Integration 12/6/2009 12:00 PM Ayende Rahien

Frans,
The main problem is that the nature of the work is async by nature, you have to create a process that starts listening, doing this as a console app is tricky.


Gravatar

# re: ÜberProf and Continuous Integration 12/9/2009 1:12 AM gunteman

Does it have to be async in all scenarios?

In an integration test, specifically designed to check for unwanted query behavior, I wouldn't mind something like...

using (var context=ProfilingContext.SetupFor.....)
{
//Do stuff
Assert.That(context.NumberOfQueries,Is.EqualTo(5));
}


Gravatar

# re: ÜberProf and Continuous Integration 12/14/2009 12:58 PM Ayende Rahien

Please check:
ayende.com/.../...re-programmatic-integration.aspx

Comments have been closed on this topic.