Ayende @ Rahien

Hi!
My name is Oren Eini
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

, @ Q c

Posts: 18 | Comments: 79

filter by tags archive

ChallengeFind the bug

time to read 1 min | 54 words

This piece of code caused a crashing bug on a QA server ( and we are lucky it didn’t go to production ).

image

Can you spot the bug?

I will give you a hint, it is the code that isn’t there.

More posts in "Challenge" series:

  1. (28 Apr 2015) What is the meaning of this change?
  2. (26 Sep 2013) Spot the bug
  3. (27 May 2013) The problem of locking down tasks…
  4. (17 Oct 2011) Minimum number of round trips
  5. (23 Aug 2011) Recent Comments with Future Posts
  6. (02 Aug 2011) Modifying execution approaches
  7. (29 Apr 2011) Stop the leaks
  8. (23 Dec 2010) This code should never hit production
  9. (17 Dec 2010) Your own ThreadLocal
  10. (03 Dec 2010) Querying relative information with RavenDB
  11. (29 Jun 2010) Find the bug
  12. (23 Jun 2010) Dynamically dynamic
  13. (28 Apr 2010) What killed the application?
  14. (19 Mar 2010) What does this code do?
  15. (04 Mar 2010) Robust enumeration over external code
  16. (16 Feb 2010) Premature optimization, and all of that…
  17. (12 Feb 2010) Efficient querying
  18. (10 Feb 2010) Find the resource leak
  19. (21 Oct 2009) Can you spot the bug?
  20. (18 Oct 2009) Why is this wrong?
  21. (17 Oct 2009) Write the check in comment
  22. (15 Sep 2009) NH Prof Exporting Reports
  23. (02 Sep 2009) The lazy loaded inheritance many to one association OR/M conundrum
  24. (01 Sep 2009) Why isn’t select broken?
  25. (06 Aug 2009) Find the bug fixes
  26. (26 May 2009) Find the bug
  27. (14 May 2009) multi threaded test failure
  28. (11 May 2009) The regex that doesn’t match
  29. (24 Mar 2009) probability based selection
  30. (13 Mar 2009) C# Rewriting
  31. (18 Feb 2009) write a self extracting program
  32. (04 Sep 2008) Don't stop with the first DSL abstraction
  33. (02 Aug 2008) What is the problem?
  34. (28 Jul 2008) What does this code do?
  35. (26 Jul 2008) Find the bug fix
  36. (05 Jul 2008) Find the deadlock
  37. (03 Jul 2008) Find the bug
  38. (02 Jul 2008) What is wrong with this code
  39. (05 Jun 2008) why did the tests fail?
  40. (27 May 2008) Striving for better syntax
  41. (13 Apr 2008) calling generics without the generic type
  42. (12 Apr 2008) The directory tree
  43. (24 Mar 2008) Find the version
  44. (21 Jan 2008) Strongly typing weakly typed code
  45. (28 Jun 2007) Windsor Null Object Dependency Facility

Comments

configurator

Is a transaction or some other synchronization method missing here? That is, it seeks the values, then deletes them. The data could change during the deletions.

Gloubidou

It's in the loop... ?

Vadi

I think, you're missing a check before deleting it ...

Dave

Where does 'data' comes from? (The ApplyToKeyAndActiveVersions line).

Rafal

Ayende, where's your sweet spot in the Esent database? Do you miss good ol' days of WINAPI programming, or is Esent exceptionally performant/stable/powerful/whatever? Maybe there was a reason Microsoft has hidden it?

Ayende Rahien

No, that is handled elsewhere

Ayende Rahien

Dave,

Another table that we join against.

Ayende Rahien

Gloubidou,

You are close...

Vadi,

No, that is not it.

Rafal,

I use it when I need a local DB. It is really easy to use when you get how it is working.

It helps that it is the only local DB with true threading support.

Alexander

maybe issue is related with closure of the session and data variables here: v=>Api.JetDelete(session, data)

Ayende Rahien

Alexander,

No, that is not it

meo
meo

Maybe "v => Api.JetDelete(session, data)" can't be run thru delegate for some reason?

Ayende Rahien

meo,

No, that is not it. It works just fine.

Stephen

I take it the crash is a stack overflow from an infinite loop and that you are either missing a breaking condition or not doing something that would should effect the 'position'..

Basically no idea, is this something that will probably only make sense when illustrated and explained?

Gerke

What happens to the table cursor after Api.JetDelete()?

And does Api.JetDelete() immediately remove the table row or only mark it as deleted?

Otherwise scenario could be:

  • Api.JetDelete moves cursor forward

  • Api.TryMovePrevious moves cursor backwards (to same row)

=> infinite loop

configurator

Stephen, That would hang, not crash...

Stephen

Ah, true ;)

(btw ayende, having problems submitting comments from ie7/8- not sure if its just me, hitting submit doesn't do anything it just sits there).

Ayende Rahien

Stephen,

No stack overflow, no, and not an infinite loop, but you should concentrate on the loop

Ayende Rahien

Gerke,

It is marking that for deletion in the end of the current transaction, TryMovePrevious is not the issue.

The loop will terminate at the expected time.

gloubidou

I guess that you are trying to clean expired values (stuff we don't care about anymore).. thus having a big result set would pose a problem... am I right?

aaron

do we need to know the contract of the jet api to find the problem?

DuvallBuck

If the Api.RetrieveColumnAsString gets the first one then the Api.TryMovePrevious would never go to the rest but stop when the first one is deleted.

Ayende Rahien

gloubidou,

Tada, you are very close.

aaron,

No, you don't.

Ayende Rahien

DuvallBuck,

No, that is not the behavior we have

Itamar

How often do values expire?

Michael Morton

You're calling the function on a fixed timer and it's being executed again, before the previous execution has finished. Considering it results in a crash, it's probably stacking a good number of times.

Ayende Rahien

That is a user defined value. In the system that we are talking abut, 24 hours

Ayende Rahien

Michael,

Huh?

What timer? There isn't any external code involved that affect the bug in this function

Sander Rijken

The call to Api.JetDelete should probably be called with session and key, not session and keyS (deleting everything)?

Ayende Rahien

Sander,

keys is the variable holding the table name, not a collection. It is instruction to delete the current row.

Ayende Rahien

Everyone,

Notice what I said, what is the code that IS NOT THERE?!

Neal Blomfield

You get the version but do not check it has expired?

TBH I am guessing based on what I would expect to see rather than understanding the code / api that this is using but it seems like the most logical thing.

Ayende Rahien

Neal,

No, that is not that.

And you don't need to understand anything about the API to see the problem.

The problem is that until you see the problem, you don't know what it is.

I am actually encouraged that no one managed to find it.

It means that I am not that stupid

gloubidou

Commit size will become huge in this case... Since you say that it is about code that isn't there, all I am able to come with is fixing a maximum of expired values to be processed in the loop and process the remaining later. That's similar to ORA-01555 snapshot too old!

Ayende Rahien

gloubidou,

TADA! You got it!

When you hit it exactly right, you may get a LOT of expired values.

Those expired values can be big enough to hit the commit limit, and cause an error!

So I was stupid, I thought so! :-)

firefly

Wow... gloubidou I am impress.

Oren, how did you uncover the bug? If the application throw an exception somewhere then it's probably not that hard to trace down the culprit. Did you catch it through stress testing?

configurator

Silly question: why is there a commit limit?

Ayende Rahien

As I said, we run into this error in QA.

Once I had the stack trace, and the error, it was really obvious what was wrong

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

  1. Production postmortem: The industry at large - 5 hours from now
  2. The insidious cost of allocations - about one day from now
  3. Buffer allocation strategies: A possible solution - 4 days from now
  4. Buffer allocation strategies: Explaining the solution - 5 days from now
  5. Buffer allocation strategies: Bad usage patterns - 6 days from now

And 2 more posts are pending...

There are posts all the way to Sep 11, 2015

RECENT SERIES

  1. Find the bug (5):
    20 Apr 2011 - Why do I get a Null Reference Exception?
  2. Production postmortem (10):
    01 Sep 2015 - The case of the lying configuration file
  3. What is new in RavenDB 3.5 (7):
    12 Aug 2015 - Monitoring support
  4. Career planning (6):
    24 Jul 2015 - The immortal choices aren't
View all series

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats