Ayende @ Rahien

Scott Mitchell commented on Analyzing a performance problem – Is a prisoner dangerous?

Mon, 03 Aug 2009 18:13:25 GMT

We use a similar pattern in one of the apps I work on, only it's for hospitals and not prisons. There are dozens of complex rules for billing - some of the jargon is "group code," "authorized sessions," "service code," and so on - all of which are assigned based on many other variables of a particular service that may exist in the main Activities table or through a number of other one to many tables. We have a batch job that runs once a night and goes through all of the (non-invoiced) activities and computes or updates these values, if needed. In the case where a user needs to bill an activity in a one-off fashion, we perform these rules for that specific activity. This approach has worked well for us given that there are dozens of rules and that several rules change every year (or new ones are added, or old ones removed).

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 30 Jun 2009 05:34:10 GMT

Command pattern

Jon commented on Analyzing a performance problem – Is a prisoner dangerous?

Mon, 29 Jun 2009 14:02:50 GMT

I like the IPrisonerDangerousEvaluator interface, and the way you use it! Is this a more general pattern?

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Thu, 11 Jun 2009 17:50:19 GMT

Futures are cool, but 40+ queries are still slow no matter how you are doing things

Dennis commented on Analyzing a performance problem – Is a prisoner dangerous?

Thu, 11 Jun 2009 15:44:36 GMT

Sure but you can Future them using a two stage approach, and thus only make a single roundtrip to the DB. Will you get super optimal speed, no. Will you get something that is decent, probably. Will you get the same developer efficiency as the first solution, close to it. I do agree with you that putting this on the writer side might be a better idea. 40+ rules is just nasty.

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Thu, 11 Jun 2009 13:09:44 GMT

You end up with a lot of queries

Dennis commented on Analyzing a performance problem – Is a prisoner dangerous?

Thu, 11 Jun 2009 06:54:28 GMT

My solution for this is usually to force load of the lazy evaluated properties for the whole data set. So instead of having SELECT 1 + N(1+1+2+2) you have select 1 + 1 + 1 + 2 + 2 Ending up with queries like: SELECT DisciplinaryNotices.Prisoner FROM DisciplinaryNotices WHERE DisciplinaryNotices.Prisoner in ( getdate()-60 >

Suiden commented on Analyzing a performance problem – Is a prisoner dangerous?

Wed, 10 Jun 2009 17:39:54 GMT

Sorry, some browser issue with my post. The return type of method Eval() is Expression < Func < Prisoner, bool > >

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Wed, 10 Jun 2009 17:37:59 GMT

Suiden, You missed the part of having tens of them?

Suiden commented on Analyzing a performance problem – Is a prisoner dangerous?

Wed, 10 Jun 2009 17:33:50 GMT

How about this implementation (kind of Alex's way except queries are separated from entities): public interface IPrisonerDangerousEvaluator { Expression > Eval(); } public class DisciplinaryNoticeInLast2Months : IPrisonerDangerousEvaluator { public Expression > Eval() { return p => p.DisciplinaryNotices .Any(x => x.Date > DateTime.Today.AddMonths(-2)); } } .. and so on. The expressions would be translated to SQL by a LINQ provider, there is still an OO model, queries are separated from entities, LINQ is used as a primary query language instead of Criteria so queries are strongly-typed...but still no better than the 8 joins right? And I can't even imagine how hard it is to write the LINQ provider. >

JoseFR commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 17:30:08 GMT

Ok, now I see the problem (hehe). One more question Ayende... Will be redundant to say that is a dangerous property?

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 17:22:33 GMT

JoseFR, An 8 table join is going to produce a horrendous Cartesian product. Let us say that I want to show the first 20 prisoners. Using this: from prisoners inner join disciplinarynotices -- most dangerous inmates have 2 - 5 inner join charges -- most dangerous inmates have 3 inner join medicalrecord -- most dangerous inmates have at least 10 inner join withdrawals -- most dangerous inmates have at least one inner join request -- most dangerous inmates have 4 inner join status -- most dangerous inmates have 6 This means that to load 20 prisoners data, you have to load 43,200 rows.

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 17:19:13 GMT

Alex, I am saying that, yes, for the problem that we have here, we have lots of props to check. About calculated methods, no, NH won't do that. If you need that, go and write query, that would be a much cleaner solution

JoseFR commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 17:17:10 GMT

How _many times_ do you need to see _all_ the prisoner dangerous property? Correct me If I wrong; there will be various tipically places where the application will need to dangerous property: 1-You need to know if a prisoner is dangerous. 2-You need to display Prisoners with certain characteristics. 3-You need to display ALL the Prisoners in a PAGINATED grid. 4-You need to process all the Prisoners of the database .... 5-You need all the dangerous or you need all the quiet. For 1,2,3 is ok because the query is limited. The database will pick the better execution plan and if you don't like it, probabillly you could change (I never need to change one). 4 is supposed to be a long time task... 5 is the WORSE case.

Alex Yakunin commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 17:11:52 GMT

Concerning 8-table join - obviously, this won't work ;) > You are going to have a LOT of properties like that. Did I say this? Actually these are your words ;) > won't touch the caching issue, but generally I don't like that, it violates the single state principle. That's obvious: use of caching normally implies this. And all depends on the costs & real business case here - e.g. if getting precise value is important, caching won't work. But in case with article versions it's normally ok to get cached value. > You can do this with NHibernate using query only formulas. Really? I.e. mapping of calculated values is pretty obvious feature, but: - Can I do the same with methods? Like GetActiveVersionFor(DateTime time)? - Can I cache such a value separately - e.g. with similar expiration policy?

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 17:00:17 GMT

JoserFR, Do you have any idea what this will look like with the resulting Cartesian product?

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 16:58:58 GMT

Alex, You are going to have a LOT of properties like that. I won't touch the caching issue, but generally I don't like that, it violates the single state principle. You can do this with NHibernate using query only formulas.

JoseFR commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 16:56:58 GMT

How many prisoners do you need to manage at the time? What about fetching all and them go with the beautifully Object Oriented approach. from prisoners inner join disciplinarynotices inner join charges inner join medicalrecord inner join withdrawals inner join request inner join status 8-table join?, The problem is not hitting 8 tables, give me more details about the volume of data.

Alex Yakunin commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 16:54:49 GMT

Well, obviously everything depends on a particular case. As I've mentioned, exactly here I'd prefer background recalculation of persistent property value. But in many other cases I'd prefer this approach. E.g. I could implement Article.PublishedVersion and ArticleVersion.IsPublished by this way, and use both properties in queries. ArticleVersion.IsPublished is true when some simple condition is satisfied for it, e.g. its publication date is greater than now and it's the latest version of it. Benefits: - Simpler queries - Normally result can easily be cached in BLL for e.g. 1 minute. And... Actually I didn't understood the part about executing hundreds of queries. In above example a single query will be executed once per day for a particular Prisoner.Key on accessing the property; if this property (ChargedWithViolentCrime) will be used in queries, it also won't lead to any additional ones.

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 16:51:51 GMT

Joao, There is too much data, in too many tables

João P. Bragança commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 16:34:48 GMT

Why can't the UI force an eager load (via context) of the data it needs? Is that bad from a design standpoint?

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 16:14:08 GMT

Alex, there are 40 such things, of varying complexity, they change often (based on intelligence work) and are of critical importance to the app. Do you really think that execute hundreds of queries would be a viable option here?

Alex Yakunin commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 15:08:11 GMT

I didn't say this must be done right in Prisoner - I wrote "Prisoner or service methods", and if you love PI, you should obviously use the second way. Concerning performance: obviously, this may bring a dramatic improvement only if after all substitutions you get a query with the lower complexity. In exactly this case this is obviously not true. But there is another benefit of such an approach: you're getting more human readable queries. UDFs and views solve nearly the same problem, but in SQL rather than LINQ. You example is just a good illistration of this: I could write e.g.: from prisoner in Prisoner where prisoner.ChargedWithViolentCrime select prisoner and get a result by a single query. Btw, I even could automatically cache the result of such call. An example with caching should look like (ok, lef me forget about PI here): public class Prisoner { ... public bool ChargedWithViolentCrime { get { return CachedQuery.Execute( new CachingPolicy(TimeSpan.FromDays(1)), () => from prisoner in Query.All(of Prisoner) where prisoner==this and prisoner.ChargedWithViolentCrime select prisoner); } } [Translator("ChargedWithViolentCrime", MemberType.PropertyGetter)] private static Expression(of Func(of Prisoner,bool)) TranslateChargedWithViolentCrime { return _this => _this.Charges.Any(p=>p.ChargeStatus.IsViolent); } ... } So the effect of this example: - Query result is cached for 1 day. Cache key varies by all query parameters by default ("this" in this case) - Query uses ChargedWithViolentCrime getter, for which LINQ translator is provided. Obviously it, as well as ChargedWithViolentCrime method itself can be implemented externally. Opinions?

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 14:23:39 GMT

Alex, That violates PI in a pretty significant ways. It also still leaves you with the complexity of the queries and the performance hit that you are going to get.

Ayende Rahien commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 14:04:34 GMT

K, They don't have Arab names, the names are Hebrew words.

K commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 12:01:45 GMT

Any particular reason that the prisoner names are all Arab names?!

Richard Dingwall commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 10:44:50 GMT

We have just embarked on a vaguely-similar project at work, and this post has answered a lot of questions for me about NHibernate, specifications, complex querying and eventual consistency. Thank you!

Think before coding commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 09:15:19 GMT

The less business logic in your database / SQL, the better. For two reasons : Architectural reason : In the command part, you evaluate the business logic in code, not in the database. Stay consistant. Performance reason : You usually update data far less often than you query it. If the result of a query cannot change between updates, compute it and store it on updates ! You can then query it at will for free. And there is no more business logic on the query part. So no triggers for me. No linq or NH for me. No big Sql script for me. It's even simpler this way. I follow you on this one Ayende.

Mischa Kroon commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 07:39:04 GMT

I would probably go with triggers + daily job for the date dependant rules.

leonard commented on Analyzing a performance problem – Is a prisoner dangerous?

Tue, 09 Jun 2009 05:16:24 GMT

@Igor "But notice that we don’t have Prisoner.Dangerous column on the result. That is because there is no such column in the table. So what does our good developer do?" So the problem is to resolve this without a new table, because we don't have access to database, I think.