Ayende @ Rahien

Refunds available at head office

Challenge: Find the bug

This piece of code caused a crashing bug on a QA server ( and we are lucky it didn’t go to production ).

image

Can you spot the bug?

I will give you a hint, it is the code that isn’t there.

Posted By: Ayende Rahien

Published at

Originally posted at

Comments

configurator
05/27/2009 12:25 AM by
configurator

Is a transaction or some other synchronization method missing here? That is, it seeks the values, then deletes them. The data could change during the deletions.

Gloubidou
05/27/2009 05:17 AM by
Gloubidou

It's in the loop... ?

Vadi
05/27/2009 06:16 AM by
Vadi

I think, you're missing a check before deleting it ...

Dave
05/27/2009 07:25 AM by
Dave

Where does 'data' comes from? (The ApplyToKeyAndActiveVersions line).

Rafal
05/27/2009 08:04 AM by
Rafal

Ayende, where's your sweet spot in the Esent database? Do you miss good ol' days of WINAPI programming, or is Esent exceptionally performant/stable/powerful/whatever? Maybe there was a reason Microsoft has hidden it?

Ayende Rahien
05/27/2009 09:43 AM by
Ayende Rahien

No, that is handled elsewhere

Ayende Rahien
05/27/2009 09:48 AM by
Ayende Rahien

Dave,

Another table that we join against.

Ayende Rahien
05/27/2009 09:50 AM by
Ayende Rahien

Gloubidou,

You are close...

Vadi,

No, that is not it.

Rafal,

I use it when I need a local DB. It is really easy to use when you get how it is working.

It helps that it is the only local DB with true threading support.

Alexander
05/27/2009 09:53 AM by
Alexander

maybe issue is related with closure of the session and data variables here: v=>Api.JetDelete(session, data)

Ayende Rahien
05/27/2009 09:55 AM by
Ayende Rahien

Alexander,

No, that is not it

meo
05/27/2009 10:13 AM by
meo

Maybe "v => Api.JetDelete(session, data)" can't be run thru delegate for some reason?

Ayende Rahien
05/27/2009 10:52 AM by
Ayende Rahien

meo,

No, that is not it. It works just fine.

Stephen
05/27/2009 11:01 AM by
Stephen

I take it the crash is a stack overflow from an infinite loop and that you are either missing a breaking condition or not doing something that would should effect the 'position'..

Basically no idea, is this something that will probably only make sense when illustrated and explained?

Gerke
05/27/2009 11:17 AM by
Gerke

What happens to the table cursor after Api.JetDelete()?

And does Api.JetDelete() immediately remove the table row or only mark it as deleted?

Otherwise scenario could be:

  • Api.JetDelete moves cursor forward

  • Api.TryMovePrevious moves cursor backwards (to same row)

=> infinite loop

configurator
05/27/2009 11:48 AM by
configurator

Stephen, That would hang, not crash...

Stephen
05/27/2009 12:25 PM by
Stephen

Ah, true ;)

(btw ayende, having problems submitting comments from ie7/8- not sure if its just me, hitting submit doesn't do anything it just sits there).

Ayende Rahien
05/27/2009 12:41 PM by
Ayende Rahien

Stephen,

No stack overflow, no, and not an infinite loop, but you should concentrate on the loop

Ayende Rahien
05/27/2009 12:42 PM by
Ayende Rahien

Gerke,

It is marking that for deletion in the end of the current transaction, TryMovePrevious is not the issue.

The loop will terminate at the expected time.

gloubidou
05/27/2009 12:58 PM by
gloubidou

I guess that you are trying to clean expired values (stuff we don't care about anymore).. thus having a big result set would pose a problem... am I right?

aaron
05/27/2009 01:23 PM by
aaron

do we need to know the contract of the jet api to find the problem?

DuvallBuck
05/27/2009 02:23 PM by
DuvallBuck

If the Api.RetrieveColumnAsString gets the first one then the Api.TryMovePrevious would never go to the rest but stop when the first one is deleted.

Ayende Rahien
05/27/2009 03:22 PM by
Ayende Rahien

gloubidou,

Tada, you are very close.

aaron,

No, you don't.

Ayende Rahien
05/27/2009 03:24 PM by
Ayende Rahien

DuvallBuck,

No, that is not the behavior we have

Itamar
05/27/2009 04:38 PM by
Itamar

How often do values expire?

Michael Morton
05/27/2009 04:53 PM by
Michael Morton

You're calling the function on a fixed timer and it's being executed again, before the previous execution has finished. Considering it results in a crash, it's probably stacking a good number of times.

Ayende Rahien
05/27/2009 05:04 PM by
Ayende Rahien

That is a user defined value. In the system that we are talking abut, 24 hours

Ayende Rahien
05/27/2009 05:05 PM by
Ayende Rahien

Michael,

Huh?

What timer? There isn't any external code involved that affect the bug in this function

Sander Rijken
05/27/2009 06:33 PM by
Sander Rijken

The call to Api.JetDelete should probably be called with session and key, not session and keyS (deleting everything)?

Ayende Rahien
05/27/2009 06:59 PM by
Ayende Rahien

Sander,

keys is the variable holding the table name, not a collection. It is instruction to delete the current row.

Paulo Köch
05/27/2009 08:14 PM by
Paulo Köch

Two Api.JetDelete calls?

Ayende Rahien
05/27/2009 08:18 PM by
Ayende Rahien

Everyone,

Notice what I said, what is the code that IS NOT THERE?!

Neal Blomfield
05/27/2009 08:47 PM by
Neal Blomfield

You get the version but do not check it has expired?

TBH I am guessing based on what I would expect to see rather than understanding the code / api that this is using but it seems like the most logical thing.

Ayende Rahien
05/27/2009 09:07 PM by
Ayende Rahien

Neal,

No, that is not that.

And you don't need to understand anything about the API to see the problem.

The problem is that until you see the problem, you don't know what it is.

I am actually encouraged that no one managed to find it.

It means that I am not that stupid

gloubidou
05/27/2009 09:11 PM by
gloubidou

Commit size will become huge in this case... Since you say that it is about code that isn't there, all I am able to come with is fixing a maximum of expired values to be processed in the loop and process the remaining later. That's similar to ORA-01555 snapshot too old!

Ayende Rahien
05/27/2009 09:13 PM by
Ayende Rahien

gloubidou,

TADA! You got it!

When you hit it exactly right, you may get a LOT of expired values.

Those expired values can be big enough to hit the commit limit, and cause an error!

So I was stupid, I thought so! :-)

firefly
05/27/2009 10:51 PM by
firefly

Wow... gloubidou I am impress.

Oren, how did you uncover the bug? If the application throw an exception somewhere then it's probably not that hard to trace down the culprit. Did you catch it through stress testing?

configurator
05/28/2009 12:07 AM by
configurator

Silly question: why is there a commit limit?

Ayende Rahien
05/28/2009 12:28 AM by
Ayende Rahien

As I said, we run into this error in QA.

Once I had the stack trace, and the error, it was really obvious what was wrong

Comments have been closed on this topic.