Ayende @ Rahien

Refunds available at head office

Modeling exercise: The grocery store’s checkout model process approach

I posted about the grocery store checkout process exercise before. Now I want to see if I can do a short outline on how I would handle this.

The key aspect from my perspective is that we need to separate the notion of the data we have and the processing of the data. That means that we are going to have the following model:

public class ShoppingCart
{
   public List<ProductInShoppingCart> Products {get;set;}
   public List<Discount> Discounts { get;set; }
}

public class ProductInShoppingCart
{
   public string ProductId;
   public Discount Discount;
}

Note that we explicitly do not have a quantity field here. If we purchase 6 bottles of milk, that would appear three times in the cart. Why is that?

Let us assume that we have a sale for 2 bottles of milk for 20% discount or a 3 +1 bottles of milk offer. Consider the kind of code you would have to write in the offer code:

  • Find all products that have this offer and have 4 items without discount.
  • Add the discount to those products.
  • After searching for products without discount, need to search for products with a discount, but that we can apply this to and get a better option.

In this case, we start by doing:

  • Add bottle of milk
  • Add bottle of milk – 2 for 20% discount is triggered.
  • Add bottle of milk
  • Add bottle of milk – 3+1 offer is triggered, removing the previous discount.

Because this is likely going to be complex, I’m going to be writing this once. A set of offers and the kind of rules that we want. Then we will give the users the ability to define those rules.

Note that we keep the raw data (products) and the transformations (discounts) separate, so we can always reapply everything without losing any data.

Career planning: Mine

I got some really good questions about my career. Which caused me to reflect back and snort at my own attempts to make a sense of what happened.

Here is a rough timeline:

  • 1999 – Start of actually considering myself to be a professional developer. This is actually the start of a great one year course I took to learn C++, right out of high school.
  • 2001 – Joined the army, was sent to the Military Police, and spent 4 years in prison. Roles ranged from a prison guard, XO of big prison, teacher in officer training course and concluded with about a year as a small prison commander.
  • 2004 – Opened my blogged and started writing about the kind of stuff that I was doing, first version of Rhino Mocks.
  • 2005 – Joined the Castle Comitter’s team, Left the army, joined We!, worked as a consultant.
  • 2006 – My first international conference – DevTeach.
  • 2008 – Left We!, started working as an independent consultant.
  • 2009 – NHibernate Profiler beta release.
  • 2010 – DSLs in Book book is published, Entity Framework Profiler, Linq to SQL Profiler, RavenDB.
  • 2011 – Hiring of first full employee.
  • 2014 – Writing this blog post.

A lot of my history doesn’t make sense without a deeper understanding. In the army, there was a long time in which I wasn’t actually able to do anything with computers. That meant that on vacations, I would do stuff voraciously. At that time, I already read a lot of university level books (dinosaurs book, Tanenbaum’s book, TCP/IP, DDD book and a lot of other stuff like that). At some point, I got an actual office and had some free time, so I could play with things. I wrote code, a lot. Nothing that was actually very interesting. It was anything from mouse tracking software (that I never actually used) to writing custom application to track medical data for inmates. Mostly I played around and learned how to do stuff.

When I got to be a prison commander, I also got myself a laptop, and start doing more serious stuff. I wasn’t able to afford any professional software, so it was mostly Open Source work. I started working on NHibernate Query Analzyer, just to see what I can do about it. That thought me a lot about reflection and NHibernate itself. I then got frustrated with the state of mocking tools in the .NET framework, and I decided to write my own. Around that time, I also started to blog.

What eventually became Rhino Mocks was a pretty big thing. Still one of the best pieces of software that I have written, it required that I’ll have a deep understanding of a lot of things. From IL generation to how classes are composed by the runtime to AppDomains to pretty much everything.

Looking back, Rhino Mocks was pretty huge in terms of pushing my career. It was very widely used, and it got me a lot of attention. After that, I was using NHibernate and talking about that a lot, so I got a lot of reputation points in that arena as well. But the first thing that I did after starting as an independent consultant was to actually work on SvnBridge. A component that would allow an SVN client to talk to a Team Foundation Server. That was something that I had very little experience with, but I think that I did a pretty good job there.

Following that, I mostly did consulting and training on NHibernate. I was pretty busy. So busy that at some point I actually have a six week business trip that took me to five countries and three continents. I came back home and didn’t leave my bed for about a week. For two weeks following that, I would feel physically ill if I sat in front of the computer for more than a few minutes.

That was a pretty big wakeup call for me. I knew that I had to do something about it. That is when I actually sat down and thought about what I wanted to do. I knew that I wanted to stay in development, and that I couldn’t continue being a road warrior without burning out again. I decided that my route would be to continue to do consulting, but on a much reduced frequency, and to start focusing on creating products. Stuff that I could work on while at home, and hopefully get paid for. That is how the NHibernate Profiler was born.

From there, it was a matter of working more on that and continuing to other areas, such as Entity Framework, Linq to SQL, etc. RavenDB came because I got tired of fixing the same old issues over and over again, even with the profilers to help me. And that actually had a business plan, we were going to invest so much money and time to get it out, and it far exceeded our expectations.

Looking back, there were several points that were of great influence. Writing my blog. Writing Rhinos Mocks, joining open source projects such as Boo or Castle. Working and blogging with NHibernate. Going to conferences and speaking there. All of those gave me a lot of experience and got me out there, building reputation and getting to know people.

That was very helpful down the road, when I was looking for consultancy jobs, or doing training. Or when it came the time to actually market our software.

In terms of the future, Hibernating Rhinos is growing at a modest rate. Which is the goal. I don’t want to double every six months, that is very disturbing to one’s peace of mind. I would much rather have a slow & steady approach. We are working on cool software, we are going home early and for the most part, there isn’t a “sky is falling” culture. The idea is that this is going to be a place that you can spend a decade or four in. I’ll get back to you on that when I retire.

Career planning: Disaster recovery

One of the more important things that you have to remember is that you should always be ready for failure. As developers, we are used to thinking about stuff like that in our code, but this is true for real life as well.

I’m going to leave aside things like personal disasters for this post (things like car accidents, getting seriously sick, etc), because there are some ways to mitigate those (insurance, family, etc) and they really isn’t anything special in development to say about those. Instead, I want to talk about professional disasters.

Those can be things like:

  • Company closing (nicely or otherwise).
  • Getting fired.
  • Product going under.
  • Product doing badly.
  • Reputation smear.
  • High profile failure.

Let me try take them in turn. The easiest one to handle is probably a company closing down, there is very little blame attached here, so there shouldn’t be an issue of having a new job. This is also the time to consider if you want to move tracks to be an independent or entrepreneur. Getting fired is a bit harder, but assuming that you weren’t fired for cause (such as negligence of criminal behavior), the old “everyone is downsizing” is going to work.

Even in a so-so economy, there is still a lot of jobs out there for software developers, but getting a good one might require you to polish your skills, and getting good idea of what is marketable today. Note that there is a big difference between what is popular and what is marketable (as in, will land you a job). Node.JS seems to be the buzzword of the day, but knowing Java very well is probably a much better path for quick employment.

This comes back to what kind of approach you want to take. For now, I’m going to assume that the fallback position for a good developer is to get hired in some fashion, it can be a short term contract, or just be gainfully employed writing software. This is important when we consider the other things that can happen, those are the kind of disasters that strike when you are more than just an employee. If you are entrepreneur and your product is just loosing too much money, for example, what is your next path?

The easy thing is when you know that you can’t go on. Maybe a competitor is pricing you out of the market, or the bank is closing the credit line or you can’t get more client or any of a hundred reasons. You are done, and you are well aware of that. A much harder issue is when you are just doing badly. So you do make sales, but not enough to cover expenses, or just getting by. Not enough to bankrupt you immediately, but you can see it happening. Unless  something changes… So you have the option of pulling the cord or trying to get it to work, with the chance of going to actual failure.

For a startup, you usually don’t have to deal with those details, but you might just show up and the company is closing down.  In those cases, there is usually not much that can be done by you (unless you are the founder, in which case, there is a wealth of information on that issue out there).

The last issue that you need to take into account is how to deal with reputation damage or a high profile failure. That depend on what the actual issue is. If it is a high profile arrest for doing coke, it might be hard to get / retain clients. If it is a big failure that cost a customer a lot of money, you might be dealing with legal consequences as well as the actual damage with other customers.

We can simplify how we look at this if we treat it all as the same thing, just a basic setback to zero (or negative). The issue is how to recover and move on. At this point, the issue is what sort of future you want? Setbacks like that are a great reason to do some thinking about where you want to go and what you want to actually do.

The conservative choice would be to find a job as a full time developer of some kind, since that at least give you a steady paycheck for the duration. More complex can be the decision to do contracting, either on a short term (at worst you can be a Word Press consultant and install that to people) or longer term projects (which require you to actually sale yourself). Hopefully you won’t be doing someone’s else homework, at least not for long.

Note that actually being able to recover from a disaster properly require prior planning. Do you have resources to survive a duration with no money? Can you handle (mentally) being out of work? Are you running on the razor’s edge of a single disruption in money flow causes an utter collapse. If that is the case, your disaster planning is going to focus on just getting reserves to handle any hiccups, versus actually managing an actual disaster.

Oh, and of course, you need to consider the cost of disaster planning. It is all very well to build a bunker to survive atomic war and the zombie rampage, but it isn’t that good if it also bankrupt you on its own.

The general recommendation is to stay current, so it would be easier to hire you, and have some idea about what to do if you wake up one day, and for whatever reason, showing up for work is not going to happen.

Career planning: What is your path?

I got a lot of really great answers about my “Where do old developers go?”, I’m feeling much better about this now Smile.

Now let turn this question around, instead of asking what is going on in the industry, let’s check what is going on with you. In particular, do you have a career plan at all?

An easy way to check that is asking: “What are you going to do in 3 years, in 7 years and in 20 years from now?”

Of course, best laid plans of mice and men will often go awry, plans for the futures are written in sand on a stormy beach and other stuff like this. Any future planning has to include the caveats that they are just plans, with reality and life getting in the way.

For lawyers*, the career path might be: Trainee, associate, senior associate, junior partner, partner, named partner. (* This is based solely on seeing some legal TV shows, not actual knowledge.) Most lawyers don’t actually become named partners, obviously, but that is what you are planning for.

As discussed in the previous post, a lot of developers move to management positions at some point in their careers, mostly because salaries and benefits tend to flat line after about ten years or so for most people in the development track. Others decide that going independent and becoming consultants or contractors is a better way to increase their income. Another path is to rise in the technical track in a company that recognize technical excellence, those are usually pure tech companies, it is rare to have such positions in non technical companies. Yet another track that seems to be available is the architect route, this one is available in non tech companies, especially big ones. You have the startup route, and the Get Rich Burning Your Twenties mode, but that is a high risk / high reward, and people who think about career planning tend to avoid such things unless carefully considered.

It is advisable to actually consider those options, try to decide what options you’ll want to have available for you in the next 5 – 15 years, and take steps accordingly. For example, if you want to go in the management track, you’ll want to work on thinks like people’s skill, be able to fluently converse with the business in their own terms and learn to play golf. You’ll want to try to have leadership positions from a relatively early start, so team lead is a stepping stone you’ll want to get to, for example. There is a lot of material on this path, so I’m not going to cover this in details.

If you want to go with the Technical Expert mode, that means that you probably need to grow a beard (there is nothing like stroking a beard in quite contemplation  to impress people). More seriously, you’ll want to get a deep level of knowledge in several fields, preferably ones that you can tie together into a cohesive package. For example, networks expert would be able to understand how TCP/IP work and be able to actually make use of that when optimize an HTML5 app. Crucial at this point is also the ability to actually transfer that knowledge to other people. If you are working within a company, that increase the overall value you have, but a lot of the time, technical experts would be consultants. Focusing on a relatively narrow field gives you a lot more value, but narrow your utility. Remember that updating your knowledge is very important. But the good news is that if you have a good grasp of basics, you can get to grips with new technology very easily.

The old timer mode fits people who work in big companies and who believe that they can carve a niche in that company based on their knowledge of the company’s business and how things actually work. This isn’t necessarily the one year experience, repeated 20 times, although in many cases, that seems to be what happens. Instead, it is a steady job with reasonable hours, and you know the business well enough and the environment in which you are working with, that you can just get things done, without a lot of fussing around. Change is hard, however, because those places tend to be very conservative. Then again, you can do new systems in whatever technology you want, at a certain point (you tend to become the owner of certain systems, you’ve been around longer than the people who are actually using the system). That does carry a risk, however. You can be fired for whatever reason (merger, downsizing, etc) and you’ll have hard time finding equivalent position.

The entrepreneur mode is for people who want to build something. That can be a tool or a platform, and they create a business selling that. A lot of the time, it involve a lot of technical work, but there is a huge amount of stuff that needs to be done that is non technical. Marketing and sales, insurance and taxes, hiring people, etc. The good thing about this is that you usually don’t have to have a very big investment in your product before you can start selling it. We are talking about roughly 3 – 6 months for most things, for 1 – 3 people. That isn’t a big leap, and in many cases, you can handle this by eating some savings, or moonlighting. Note that this can completely swallow your life, but you are your own boss, and there is a great deal of satisfaction around building a product around your vision. Be aware that you need to have contingency plans around for failure and success. If your product become successful, you need to make sure that you can handle the load (hire more people, train them, etc).

The startup mode is very different than the entrepreneur mode. In startup, you are focused on getting investments, and the scope is usually much bigger. There is less of a risk financially (you usually have investors for that), but there is a much higher risk of failure, and there is usually a culture that consider throwing yourself on hand grenade advisable. The idea is that you are going to burn yourself on both ends for two to four years, and in return, you’ll have enough money to maybe stop working altogether. I consider this foolish, given the success rates, but there are a lot of people who consider that to be the only way worth doing. The benefits usually include a nice environment, both physically and professionally, but  it comes with the expectation that you’ll stay there for so many hours, it is your second home.

There are other modes and career paths, but now I’ve to return to my job Smile.

Modeling exercise: The grocery store’s checkout model

I went to the super market yesterday, and I forgot to get out of work mode, so here is this posts. imageThe grocery store checkout model exercise deals with the following scenario. You have a customer that is scanning products in a self checkout lane, and you need to process the order.

In terms of external environment, you have:

  • ProductScanned ( ProductId: string ) event
  • Complete Order command
  • Products ( Product Id –> Name, Price ) dataset

So far, this is easy, however, you also need to take into account:

  • Sales (1+1, 2+1, 5% off for store brands, 10% off for store brands for loyalty card holders).
  • Purchase of items by weight (apples, bananas, etc).
  • Per customer discount for 5 items.
  • Rules such as alcohol can only be purchased after store clerk authorization.
  • Purchase limits (can only purchase up to 6 items of the same type, except for specific common products)

The nice thing about such an exercise is that it forces you to see how many things you have to juggle for such a seemingly simple scenario.

A result of this would be to see how you would handle relatively complex rules. Given the number of rules we already have, it should be obvious that there are going to be more, and that they are going to be changing on a fairly frequent basis. A better model would be to actually do this over time. So you start with just getting the first part, then you start streaming the other requirements, but what you actually see is the changes in the code over time. So each new requirement causes you to make modifications and accommodate the new behavior.

The end result might be a Git repository that allows you to see the full approach that was used and how it changed over time. Ideally, you should see a lot of churn in the beginning, but then you’ll have a lot less work to do as your architecture settles down.

Career planning: Where do old devs go to?

We are pretty much always looking for new people, what is holding us back from expanding even more rapidly is the time that it takes to get to grips with our codebases and what we do here. But that also means that we usually have at least one outstanding job offer available, because it takes a long time to fill it. But that isn’t the topic for this post.

I started programming in school, I was around 14 / 15 at the time, and I picked up a copy of VB 3.0 and did some fairly unimpressive stuff with it. I count my time as a professional since around 1999 or so. That is the time when I started actually dedicating myself to learning programming as something beyond a hobby. That was 15 years ago.

When we started doing a lot of interviews, I noticed that we had the following pattern regarding developers’ availabilities:

image

That sort of made sense, some people got into software development for the money and left because it didn’t interest them. From the history of Hibernating Rhinos, one of our developers left and is now co-owner in a restaurant, another is a salesman for lasers and other lab stuff.

However, what doesn’t make sense is the ratio that I’m seeing. Where are the people who have been doing software development for decades?

Out of the hundreds of CVs that I have seen, there has been less than 10 that had people over the age of 40. I don’t recall anyone over the age of 50. Note that I’m somewhat biased to hire people with longer experience, because that often means that they don’t need to learn what goes under the hood, they already know.

In fact, looking at the salary tables, there actually isn’t a level of higher than 5 years. After that, you have a team leader position, and then you move into middle management, and then you are basically gone as a developer, I’m guessing.

What is the career path you have as a developer? And note that I’m explicitly throwing out management positions. It seems that those are very rare in our industry.

Microsoft has the notion of Distinguished Engineer and Technical Fellow, for people who actually have decades of experience. In my head, I have this image of a SWAT team that you throw at the hard problems Smile.

Outside of very big companies, those seem to be very rare.  And that is kind of sad.

In Hibernating Rhinos, we plan to actually have those kind of long career paths, but you’ll need to ask me in 10 – 20 years how that turned out to be.

Tags:

Published at

Originally posted at

Comments (74)

The holidays, plans & what is next

So, we are done with the holidays here. The last month was basically very little work, because a lot of our people were out for the holidays.

Internally, we are gearing up to finish the website for RavenDB 3.0, while another part of the team is focused on stability and performance. We just hired another new guy, and he is going to be working pretty much on distribution from now on. I’ll report more on that in a few weeks.

Looking at the blog, I’ve mostly been talking about RavenDB, and I want to do a small shift and talk about other topics, so I’m declaring the next two weeks to be RavenDB free weeks. I’m going to continue to blog regularly, of course, but I’m going to be talking about other topics for a change.

Don’t worry, we haven’t stopped working on RavenDB, it is just that it is pretty boring to hear about things like test clusters, or how we work on issues from the people trying out the RC builds.

Published at

Originally posted at

Release preps, and my mobile cluster

I just took this picture on my desk. This is a set of machines running a whole set of tests for RavenDB 3.0. On the bottom right, you can see one of our new toys (more below).

image

This new toy is a NUC (i5, 16 GB, 180 GB SSD). We have a couple of those (and will likely purchase more).

They have a very small form factor, and they are pretty cool machines. We got a few of them so we can have easier time testing distributed systems that are really distributed.

It also has a very nice effect of actually being able to carry around a full cluster and “deploy” it in a few minutes.

Tags:

Published at

Originally posted at

Comments (4)

Looking at nopCommerce

I’m preparing for a talk about using multiple databases in a single application, and I wanted to show how this looks like in a real application. I wanted to take a look at nopCommerce as the target application.

To be fair, I haven’t really worked with enterprise applications in a while, but I found some interesting stuff there. Note that I’m not doing anything like a full review. My goal is to show the concept, not explore the full implications of nopCommerce. The code in questions is from commit 1286b4f8d4c0ed2a5d6441db7cbd5398821d32f2.

nopCommerce is using Entity Framework for accessing SQL Server, so it was a simple matter to install the Entity Framework Profiler and take a peek into what was going on there. I then did the simplest thing I could think of, and searched for gift in the site:

image

Here is what happened:

image

Now, there are 25 queries, and a total of 40KB of SQL executed. Here is one such query:

SELECT [Project4].[Id]                             AS [Id],
[Project4].[Name] AS [Name],
[Project4].[Description] AS [Description],
[Project4].[CategoryTemplateId] AS [CategoryTemplateId],
[Project4].[MetaKeywords] AS [MetaKeywords],
[Project4].[MetaDescription] AS [MetaDescription],
[Project4].[MetaTitle] AS [MetaTitle],
[Project4].[ParentCategoryId] AS [ParentCategoryId],
[Project4].[PictureId] AS [PictureId],
[Project4].[PageSize] AS [PageSize],
[Project4].[AllowCustomersToSelectPageSize] AS [AllowCustomersToSelectPageSize],
[Project4].[PageSizeOptions] AS [PageSizeOptions],
[Project4].[PriceRanges] AS [PriceRanges],
[Project4].[ShowOnHomePage] AS [ShowOnHomePage],
[Project4].[IncludeInTopMenu] AS [IncludeInTopMenu],
[Project4].[HasDiscountsApplied] AS [HasDiscountsApplied],
[Project4].[SubjectToAcl] AS [SubjectToAcl],
[Project4].[LimitedToStores] AS [LimitedToStores],
[Project4].[Published] AS [Published],
[Project4].[Deleted] AS [Deleted],
[Project4].[DisplayOrder] AS [DisplayOrder],
[Project4].[CreatedOnUtc] AS [CreatedOnUtc],
[Project4].[UpdatedOnUtc] AS [UpdatedOnUtc]
FROM (SELECT [Limit1].[Id] AS [Id],
[Limit1].[Name] AS [Name],
[Limit1].[Description] AS [Description],
[Limit1].[CategoryTemplateId] AS [CategoryTemplateId],
[Limit1].[MetaKeywords] AS [MetaKeywords],
[Limit1].[MetaDescription] AS [MetaDescription],
[Limit1].[MetaTitle] AS [MetaTitle],
[Limit1].[ParentCategoryId] AS [ParentCategoryId],
[Limit1].[PictureId] AS [PictureId],
[Limit1].[PageSize] AS [PageSize],
[Limit1].[AllowCustomersToSelectPageSize] AS [AllowCustomersToSelectPageSize],
[Limit1].[PageSizeOptions] AS [PageSizeOptions],
[Limit1].[PriceRanges] AS [PriceRanges],
[Limit1].[ShowOnHomePage] AS [ShowOnHomePage],
[Limit1].[IncludeInTopMenu] AS [IncludeInTopMenu],
[Limit1].[HasDiscountsApplied] AS [HasDiscountsApplied],
[Limit1].[SubjectToAcl] AS [SubjectToAcl],
[Limit1].[LimitedToStores] AS [LimitedToStores],
[Limit1].[Published] AS [Published],
[Limit1].[Deleted] AS [Deleted],
[Limit1].[DisplayOrder] AS [DisplayOrder],
[Limit1].[CreatedOnUtc] AS [CreatedOnUtc],
[Limit1].[UpdatedOnUtc] AS [UpdatedOnUtc]
FROM (SELECT [Distinct1].[Id] AS [Id]
FROM (SELECT DISTINCT [Extent1].[Id] AS [Id]
FROM [dbo].[Category] AS [Extent1]
LEFT OUTER JOIN [dbo].[AclRecord] AS [Extent2]
ON ([Extent1].[Id] = [Extent2].[EntityId])
AND (N'Category' = [Extent2].[EntityName])
LEFT OUTER JOIN [dbo].[StoreMapping] AS [Extent3]
ON ([Extent1].[Id] = [Extent3].[EntityId])
AND (N'Category' = [Extent3].[EntityName])
WHERE ([Extent1].[Published] = 1)
AND ([Extent1].[Deleted] <> cast(1 as bit))
AND (([Extent1].[SubjectToAcl] <> cast(1 as bit))
OR (([Extent2].[CustomerRoleId] IN (4))
AND ([Extent2].[CustomerRoleId] IS NOT NULL)))
AND (([Extent1].[LimitedToStores] <> cast(1 as bit))
OR (1 /* @p__linq__0 */ = [Extent3].[StoreId]))) AS [Distinct1]) AS [Project2]
OUTER APPLY (SELECT TOP (1) [Filter2].[Id1] AS [Id],
[Filter2].[Name] AS [Name],
[Filter2].[Description] AS [Description],
[Filter2].[CategoryTemplateId] AS [CategoryTemplateId],
[Filter2].[MetaKeywords] AS [MetaKeywords],
[Filter2].[MetaDescription] AS [MetaDescription],
[Filter2].[MetaTitle] AS [MetaTitle],
[Filter2].[ParentCategoryId] AS [ParentCategoryId],
[Filter2].[PictureId] AS [PictureId],
[Filter2].[PageSize] AS [PageSize],
[Filter2].[AllowCustomersToSelectPageSize] AS [AllowCustomersToSelectPageSize],
[Filter2].[PageSizeOptions] AS [PageSizeOptions],
[Filter2].[PriceRanges] AS [PriceRanges],
[Filter2].[ShowOnHomePage] AS [ShowOnHomePage],
[Filter2].[IncludeInTopMenu] AS [IncludeInTopMenu],
[Filter2].[HasDiscountsApplied] AS [HasDiscountsApplied],
[Filter2].[SubjectToAcl] AS [SubjectToAcl],
[Filter2].[LimitedToStores] AS [LimitedToStores],
[Filter2].[Published] AS [Published],
[Filter2].[Deleted] AS [Deleted],
[Filter2].[DisplayOrder] AS [DisplayOrder],
[Filter2].[CreatedOnUtc] AS [CreatedOnUtc],
[Filter2].[UpdatedOnUtc] AS [UpdatedOnUtc]
FROM (SELECT [Extent4].[Id] AS [Id1],
[Extent4].[Name] AS [Name],
[Extent4].[Description] AS [Description],
[Extent4].[CategoryTemplateId] AS [CategoryTemplateId],
[Extent4].[MetaKeywords] AS [MetaKeywords],
[Extent4].[MetaDescription] AS [MetaDescription],
[Extent4].[MetaTitle] AS [MetaTitle],
[Extent4].[ParentCategoryId] AS [ParentCategoryId],
[Extent4].[PictureId] AS [PictureId],
[Extent4].[PageSize] AS [PageSize],
[Extent4].[AllowCustomersToSelectPageSize] AS [AllowCustomersToSelectPageSize],
[Extent4].[PageSizeOptions] AS [PageSizeOptions],
[Extent4].[PriceRanges] AS [PriceRanges],
[Extent4].[ShowOnHomePage] AS [ShowOnHomePage],
[Extent4].[IncludeInTopMenu] AS [IncludeInTopMenu],
[Extent4].[HasDiscountsApplied] AS [HasDiscountsApplied],
[Extent4].[SubjectToAcl] AS [SubjectToAcl],
[Extent4].[LimitedToStores] AS [LimitedToStores],
[Extent4].[Published] AS [Published],
[Extent4].[Deleted] AS [Deleted],
[Extent4].[DisplayOrder] AS [DisplayOrder],
[Extent4].[CreatedOnUtc] AS [CreatedOnUtc],
[Extent4].[UpdatedOnUtc] AS [UpdatedOnUtc]
FROM [dbo].[Category] AS [Extent4]
LEFT OUTER JOIN [dbo].[AclRecord] AS [Extent5]
ON ([Extent4].[Id] = [Extent5].[EntityId])
AND (N'Category' = [Extent5].[EntityName])
WHERE ([Extent4].[Published] = 1)
AND ([Extent4].[Deleted] <> cast(1 as bit))
AND (([Extent4].[SubjectToAcl] <> cast(1 as bit))
OR (([Extent5].[CustomerRoleId] IN (4))
AND ([Extent5].[CustomerRoleId] IS NOT NULL)))) AS [Filter2]
LEFT OUTER JOIN [dbo].[StoreMapping] AS [Extent6]
ON ([Filter2].[Id1] = [Extent6].[EntityId])
AND (N'Category' = [Extent6].[EntityName])
WHERE (([Filter2].[LimitedToStores] <> cast(1 as bit))
OR (1 /* @p__linq__0 */ = [Extent6].[StoreId]))
AND ([Project2].[Id] = [Filter2].[Id1])) AS [Limit1]) AS [Project4]
ORDER BY [Project4].[ParentCategoryId] ASC,
[Project4].[DisplayOrder] ASC

For reference, here is the query plan for this:

image

 

Note that all of this is generated for the products is not done via EntityFramework. That is done via the ProductLoadAllPaged stored procedure. That is 620 lines of code, includes dynamic SQL generation and several temp tables.

At that point, I actually looked at the code, and it isn’t something that I can actually make easy modifications to, at least not without spending way too long trying to understand what is going on for it to be worth it for a single presentation. So I’m going to leave this aside and look at another app. Probably a sample site, nopCommerce has 44 projects and Orchard has 77 projects. That is too big to actually be able to talk about concisely in a talk.

Tags:

Published at

Originally posted at

Comments (14)

RavenFS and NServiceBus’ Data Bus

The NServiceBus data bus allows you to send very large messages by putting them on a shared resource and sending the reference to it. An obvious use case for this is using RavenFS. I took a few moments and wrote an implementation for that*.

public class RavenFSDataBus : IDataBus, IDisposable
{
private readonly FilesStore _filesStore;
private Timer _timer;


private object _locker = new object();
private void RunExpiration(object state)
{
bool lockTaken = false;
try
{
Monitor.TryEnter(_locker, ref lockTaken);
if (lockTaken == false)
return;

using (var session = _filesStore.OpenAsyncSession())
{
var files = session.Query()
.WhereLessThan("Time-To-Be-Received", DateTime.UtcNow.ToString("O"))
.OrderBy("Time-To-Be-Received")
.ToListAsync();

files.Wait();

foreach (var fileHeader in files.Result)
{
session.RegisterFileDeletion(fileHeader);
}

session.SaveChangesAsync().Wait();
}
}
finally
{
if (lockTaken)
Monitor.Exit(_locker);
}
}

public RavenFSDataBus(string connectionString)
{
_filesStore = new FilesStore
{
ConnectionStringName = connectionString
};
}

public RavenFSDataBus(FilesStore filesStore)
{
_filesStore = filesStore;
}

public Stream Get(string key)
{
return _filesStore.AsyncFilesCommands.DownloadAsync(key).Result;
}

public string Put(Stream stream, TimeSpan timeToBeReceived)
{
var key = "/data-bus/" + Guid.NewGuid();
_filesStore.AsyncFilesCommands.UploadAsync(key, stream, new RavenJObject
{
{"Time-To-Be-Received", DateTime.UtcNow.Add(timeToBeReceived).ToString("O")}
}).Wait();

return key;
}

public void Start()
{
_filesStore.Initialize(ensureFileSystemExists: true);
_timer = new Timer(RunExpiration);
_timer.Change(TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1));
}

public void Dispose()
{
if (_timer != null)
_timer.Dispose();
if (_filesStore != null)
_filesStore.Dispose();
}
}

* This is written to check it out, hasn’t been tested very well yet.

Tags:

Published at

Originally posted at

Complex nested structures in RavenDB

This started out as a question in the mailing list. Consider the following (highly simplified) model:

   public class Building
   {
       public string Name { get; set; }
       public List<Floor> Floors { get; set; }        
   }
   
   public class Floor
   {
       public int Number { get; set; }
       public List<Apartment> Apartments { get; set; }
   }
 
   public class Apartment
   {
       public string ApartmentNumber { get; set; }
       public int SquareFeet { get; set; }
   }

And here you can see an actual document:

{
    "Name": "Corex's Building - Herzliya",
    "Floors": [
        {
            "Number": 1,
            "Apartments": [
                {
                    "ApartmentNumber": 102,
                    "SquareFeet": 260
                },
                {
                    "ApartmentNumber": 104,
                    "SquareFeet": 260
                },
                {
                    "ApartmentNumber": 107,
                    "SquareFeet": 460
                }
            ]
        },
        {
            "Number": 2,
            "Apartments": [
                {
                    "ApartmentNumber": 201,
                    "SquareFeet": 260
                },
                {
                    "ApartmentNumber": 203,
                    "SquareFeet": 660
                }
            ]
        }
    ]
}

Usually the user is working with the Building document. But every now an then, they need to show just a specific apartment.

Normally, I would tell them that they can just load the relevant document and extract the inner information on the client, that is very cheap to do. And that is still the recommendation. But I thought that I would use this opportunity to show off some features that don’t get their due exposure.

We then define the following index:

image

Note that we can use the Query() method to fetch the query specific parameter from the user. Then we just search the data for the relevant item.

From the client code, this will look like:

var q = session.Query<Building>()
    .Where(b =>/* some query for building */)
    .TransformWith<SingleApartment, Apartment>()
    .AddTransformerParameter("apartmentNumber", 201)
    .ToList();


var apartment = session.Load<SingleApartment, Apartment>("building/123",
        configuration => configuration.AddTransformerParameter("apartmentNumber", 102));

And that is all there is to it.

Tags:

Published at

Originally posted at

Comments (8)

Mono frustrations

I’m porting Voron to Mono (currently testing on Ubuntu). I’m using Mono 3.2.8, FWIW, and working in MonoDevelop.

So far, I have run into the following tangles that are annoying.  Attempting to write to memory that is write protected results in null reference exception, instead of access violation exception. I suspect that this is done because NRE is generated on any SIGSEGV, but that led me to a very different path of discovery.

Also, consider the following code:

using System.IO;
using System.IO.Compression;

namespace HelloWorld
{
class MainClass
{

public static void Main (string[] args)
{
new ZipArchive (new MemoryStream ());
}
}
}

This results in the following error:

Unhandled Exception:
    System.NotImplementedException: The requested feature is not implemented.
        at HelloWorld.MainClass.Main (System.String[] args) [0x00006] in /home/ayende/HelloWorld/HelloWorld/Program.cs:11
        [ERROR] FATAL UNHANDLED EXCEPTION: System.NotImplementedException: The requested feature is not implemented.
        at HelloWorld.MainClass.Main (System.String[] args) [0x00006] in /home/ayende/HelloWorld/HelloWorld/Program.cs:11

This is annoying in that it isn’t implemented, but worse from my point of view is that I don’t see any ZipArchive in the stack trace. That made me think that it was my code that was throwing this.

Tags:

Published at

Originally posted at

Comments (16)

RavenDB 3.0–Release Candidate & Go Live

Update: We delayed the RC release by a week or two because we wanted to finish the new website. But I decided that it doesn't make sense to at least give you the RC bits so you can play with them. You can look at the new website at http://beta.ravendb.net, it should be done in about a week (we are in a holidays period right now, which slow things down). 

I’m taking a break from explaining what is new in RavenDB 3.0 because we have more important news. This is still release candidate, because we want to get more feedback from the field before we can say that this is a final version. The plan is to give the RC a few weeks to mature, and then make a full release. This also comes with Go Live version, so this is fully support for production (and much easier to deal with on production).

This release also include a new website for RavenDB, as well as the updated licensing. Note that we provide a 20% discount for purchases during the RC period. For customers that purchased a RavenDB license since 1 Jul 2014, can upgrade (for no cost) to a RavenDB 3.0 release.

You can go to our site to see how things changes.

image

Tags:

Published at

Originally posted at

Comments (16)

Working on Voron…

This took a bit less than what I expected, but…

image

And yes, it works. And this is running on Ubuntu.

And no, it isn’t ready.

Tags:

Published at

Originally posted at

Comments (5)

Google Domains won’t take my money

This is frustrating. I’m trying to use the new Google Domains (if only because I’m sick of GoDaddy’s aggressive upsale approach).

This is what happened when I tried it.

image

That sucks, but I’m actually a business user for Google Apps, so I called their support. I’m currently on the 32th minute of the call, and so far they explained to me numerous times that this is not possible to do, and are still unable to tell me why they are refusing to take my card.

I can understand, beta service and all of that, but using Wallet isn’t beta, and not being able to tell me what is wrong is not cool.

Tags:

Published at

Originally posted at

Comments (8)

RavenDB Recognized in DZone’s 2014 Guide to Big Data

DZR_BigData_VendorButton

I’m excited to get to tell you that RavenDB is a  featured vendor in DZone’s 2014 Guide to Big Data. The guide includes expert opinions and tips, industry knowledge, and data platform and database comparisons. And it would give you a good background information about the different NoSQL solutions that are currently available.

Readers can download a free copy of the guide here.

Tags:

Published at

Originally posted at

Comments (5)

Optimizing event processing

During the RavenDB Days conference, I got a lot of questions from customers. Here is one of them.

There is a migration process that deals with event sourcing system. So we have 10,000,000 commits with 5 – 50 events per commit. Each event result in a property update to an entity.

That gives us roughly 300,000,000 events to process. The trivial way to solve this would be:

foreach(var commit in YieldAllCommits())
{
using(var session = docStore.OpenSession())
{
foreach(var evnt in commit.Events)
{
var entity = evnt.Load<Customer>(evnt.EntityId);
evnt.Apply(entity);
}
session.SaveChanges();
}
}

That works, but it tends to be slow. Worse case here would result in 310,000,000 requests to the server.

Note that this has the nice property that all the changes in a commit are saved in a single commit. We’re going to relax this behavior, and use something better here.

We’ll take the implementation of this LRU cache and add an event for dropping from the cache and iteration.

usging(var bulk = docStore.BulkInsert(allowUpdates: true))
{
var cache = new LeastRecentlyUsedCache<string, Customer>(capacity: 10 * 1000);
cache.OnEvict = c => bulk.Store(c);
foreach(var commit in YieldAllCommits())
{
using(var session = docStore.OpenSession())
{
foreach(var evnt in commit.Events)
{
Customer entity;
if(cache.TryGetValue(evnt.EventId, out entity) == false)
{
using(var session = docStore.OpenSession())
{
entity = session.Load<Customer>(evnt.EventId);
cache.Set(evnt.EventId, entity);
}
}
evnt.Apply(evnt);
}
}
}
foreach(var kvp in cache){
bulk.Store(kvp.Value);
}
}

Here we are using a cache of 10,000 items. With the assumption that we are going to have clustering for events on entities, so a lot of changes on an entity will happen on roughly the same time. We take advantage of that to try to only load each document once. We use bulk insert to flush those changes to the server when needed. This code will handle the case where we flushed out a document from the cache then we get events for it again, but he assumption is that this scenario is much lower.