ChallengeRobust enumeration over external code
Here is an interesting little problem:
public class Program { private static void Main() { foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc)) { Console.WriteLine(i); } } public static IEnumerable<T> RobustEnumerating<T>( IEnumerable<T> input,Func<IEnumerable<T>, IEnumerable<T>> func) { // how to do this? return func(input); } public static IEnumerable<int> FaultyFunc(IEnumerable<int> source) { foreach (int i in source) { yield return i/(i%2); } } }
This code should not throw, but print:
1
3
5
7
9
Can you make this happen? You can only change the RobustEnumerating method, nothing else in the code
More posts in "Challenge" series:
- (01 Jul 2024) Efficient snapshotable state
- (13 Oct 2023) Fastest node selection metastable error state–answer
- (12 Oct 2023) Fastest node selection metastable error state
- (19 Sep 2023) Spot the bug
- (04 Jan 2023) what does this code print?
- (14 Dec 2022) What does this code print?
- (01 Jul 2022) Find the stack smash bug… – answer
- (30 Jun 2022) Find the stack smash bug…
- (03 Jun 2022) Spot the data corruption
- (06 May 2022) Spot the optimization–solution
- (05 May 2022) Spot the optimization
- (06 Apr 2022) Why is this code broken?
- (16 Dec 2021) Find the slow down–answer
- (15 Dec 2021) Find the slow down
- (03 Nov 2021) The code review bug that gives me nightmares–The fix
- (02 Nov 2021) The code review bug that gives me nightmares–the issue
- (01 Nov 2021) The code review bug that gives me nightmares
- (16 Jun 2021) Detecting livelihood in a distributed cluster
- (21 Apr 2020) Generate matching shard id–answer
- (20 Apr 2020) Generate matching shard id
- (02 Jan 2020) Spot the bug in the stream
- (28 Sep 2018) The loop that leaks–Answer
- (27 Sep 2018) The loop that leaks
- (03 Apr 2018) The invisible concurrency bug–Answer
- (02 Apr 2018) The invisible concurrency bug
- (31 Jan 2018) Find the bug in the fix–answer
- (30 Jan 2018) Find the bug in the fix
- (19 Jan 2017) What does this code do?
- (26 Jul 2016) The race condition in the TCP stack, answer
- (25 Jul 2016) The race condition in the TCP stack
- (28 Apr 2015) What is the meaning of this change?
- (26 Sep 2013) Spot the bug
- (27 May 2013) The problem of locking down tasks…
- (17 Oct 2011) Minimum number of round trips
- (23 Aug 2011) Recent Comments with Future Posts
- (02 Aug 2011) Modifying execution approaches
- (29 Apr 2011) Stop the leaks
- (23 Dec 2010) This code should never hit production
- (17 Dec 2010) Your own ThreadLocal
- (03 Dec 2010) Querying relative information with RavenDB
- (29 Jun 2010) Find the bug
- (23 Jun 2010) Dynamically dynamic
- (28 Apr 2010) What killed the application?
- (19 Mar 2010) What does this code do?
- (04 Mar 2010) Robust enumeration over external code
- (16 Feb 2010) Premature optimization, and all of that…
- (12 Feb 2010) Efficient querying
- (10 Feb 2010) Find the resource leak
- (21 Oct 2009) Can you spot the bug?
- (18 Oct 2009) Why is this wrong?
- (17 Oct 2009) Write the check in comment
- (15 Sep 2009) NH Prof Exporting Reports
- (02 Sep 2009) The lazy loaded inheritance many to one association OR/M conundrum
- (01 Sep 2009) Why isn’t select broken?
- (06 Aug 2009) Find the bug fixes
- (26 May 2009) Find the bug
- (14 May 2009) multi threaded test failure
- (11 May 2009) The regex that doesn’t match
- (24 Mar 2009) probability based selection
- (13 Mar 2009) C# Rewriting
- (18 Feb 2009) write a self extracting program
- (04 Sep 2008) Don't stop with the first DSL abstraction
- (02 Aug 2008) What is the problem?
- (28 Jul 2008) What does this code do?
- (26 Jul 2008) Find the bug fix
- (05 Jul 2008) Find the deadlock
- (03 Jul 2008) Find the bug
- (02 Jul 2008) What is wrong with this code
- (05 Jun 2008) why did the tests fail?
- (27 May 2008) Striving for better syntax
- (13 Apr 2008) calling generics without the generic type
- (12 Apr 2008) The directory tree
- (24 Mar 2008) Find the version
- (21 Jan 2008) Strongly typing weakly typed code
- (28 Jun 2007) Windsor Null Object Dependency Facility
Comments
public static IEnumerable <t RobustEnumerating <t(
<t input,Func <ienumerable<t, IEnumerable <t> func)
Juan,
That code doesn't compiles
Not tested and ugly and possibly cheating, but:
public static IEnumerable <t RobustEnumerating <t(
<t input,Func <ienumerable<t, IEnumerable <t> func)
<t();
It doesn't work either, because the exception occurs at MoveNext method of the enumerator. After that, the enumerator becomes unusable.
I don't think there is a solution. =/
Michael,
Your code won't compile
Stealing Michael's idea:
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
The problem is that the whole IEnumerable is evaluated upon the first call to the MoveNext, therefore it is not possible to have a try/catch around each.
I give up :(
This is terrible, but the only way I could get this to work:
public static IEnumerable <t RobustEnumerating <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
{
<t holder = new List <t();
}
David,
That is an workable implementation, yes. It has some bugs (there isn't 1:1 mapping between result and reply)
Now try doing this without invoking the enumerator for every single item (initialization cost may be expensive).
There are at least two additional ways of doing it
This works.. And I learned something about enumerables and exceptions today.
<t RobustEnumerating <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
<t enumerator = input.GetEnumerator();
<int FaultyFunc(IEnumerable <int source)
Dennis,
If it isn't a lazy enumerator, you can't do anything, right.
But the code I have shown IS lazy
vsjones,
Your approach won't work with infinite enumerators, can you make it lazy?
pb,
What happens if a function returns two output values for every input value?
David, that wont work.
What if the FaultyFuncsaid
public static IEnumerable <int FaultyFunc(IEnumerable <int source)
You cannot presume the function is without sideeffects
It ain't pretty, but it works. :)
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
var e = func(input).GetEnumerator();
while(true) {
}
Not pretty also, but...
<t RobustEnumerating <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
<t output = new List <t();
<t)output);
dam people reply fast. :) Same as pb.
Hehe, your original challenge was to make the example work. ;P
Should satisfy Ayende's adjustment:
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t results = new List <t();
Should work - only calls func once so if it has side effects we are fine - works with infinite enumerations :)
public class Program {
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func) {
<t enumerable = func(input);
<t enumerator = null;
<int FaultyFunc(IEnumerable <int source) {
Since Steve copied me I'll say ditto to his... :)
@mike
You stolez my codez! ;)
@oren
This isn't possible if you think about it. If MoveNext throws you have NO idea what the state of the enumerator is. What if it's a custom Ienumerator implementation? The c# codegened ones fail gracefully, but that's not guaranteed if I write one myself.
public static IEnumerable <t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
{
if (!input.Any()) return Enumerable.Empty <t();
var result = new List <t();
try
{
}
catch
{
}
}
Nice Mike. :)
Never thought on reseting the state of the enumerator. This sounds a bit of hack, but it works :)
Steve Py:
you can remove the bool match flag.. as the list will be empty in case of exception...
What about something like below?
It works, but I feel there has to be a cheaper way to do it.
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t();
lazy, concise, expressive.
it'd be even better if I had an immutable list
return
<t(),
@Jual Lopez yeah, just a "little" hack :)
Oops. Change
.Add(......Single())
to
.AddRange(.....)
to handle multiple returns for a single input
public static IEnumerable <t RobustEnumerating <t(
<t input,Func <ienumerable<t, IEnumerable <t> func)
{
<t();
}
@nb != copy. :) When I started the current post was from Ayende about Juan's not compiling. I was shocked after posting to see some 10 other responses including virtually the same solution.
@yh Yeah, could just check for the empty list... Hacking that together on my lunch break. :)
Final attempt.
it turns out Aggregate isn't lazy.
The only trick here is the .ToList on the func result to force the error to occur inside the try catch rather than later when selectmany is flattening lists.
return
<t();
@mike,
Your code only works assuming the enumerator has 2 states.
Take the following, for example:
<int FaultyFunc2(IEnumerable <int source) {
This will 5 different states, and your implementation sends it into an infinite loop.
Also, Mike's code fails when the enumerator disposes of resources in a finally block. The code below sends it into an infinite loop even though there are only 2 states and state 2 is the "get me the next item" state. Attempting to continue iteration at that point leads to an ObjectDisposedException, which promptly gets swallowed and "iteration" continues.
My code-sense is telling me that unless you have an Oracle for the Halting Problem, this can't be done in such a manner that you only initialize the enumerator one time, or in a way that handles every possible enumerator implementation. You will have to impose constrains upon what "types" of enumerators you will consume. Or, better yet, remove this requirement altogether by changing the assumptions/requirements in whatever code needs this.
<int FaultyFunc3(IEnumerable <int source) {
I gave up reading the suggestions, its just too annoying to read code that isn't html encoded and formatted properly. It would be better if people were posting codepaste/github gist/etc links.
It ain't pretty.
public static IEnumerable <t RobustEnumerating <t( IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
<t();
How about this? (never said we couldn't add methods...)
public class Program
{
public static void Main(string[] args)
{
}
public static IEnumerable <t RobustEnumerating <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
{
}
private static IEnumerable <t RecursiveEnumerator <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
{
<t();
}
public static IEnumerable <int FaultyFunc(IEnumerable <int source)
{
}
}
Something something functional ;)
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t.Nothing).
<t One <t(T value)
<r TryApply <t,> (T input, Func <t,> func)
<r.Just(func(input));
<r.Nothing;
<t
{
<t Nothing = new Maybe <t();
<t Just(T value)
<t {Value = value};
try the following code
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
It's a little bit ugly but it compiles, returns what you need and doing it in lazy manner:
public static IEnumerable <t Wrap <t(IEnumerator <t input)
<t RobustEnumerating <t(
<t input,Func <ienumerable<t, IEnumerable <t> func)
<t(IEnumerator <t en)
What about this?
public static IEnumerable <t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t { el }.AsEnumerable()).FirstOrDefault();
<t { el }.AsEnumerable()).FirstOrDefault();
Since Ayende kept saying input & output need not be 1:1 here is a solution that keeps track of the input enumerator and tries to minimize initialization of faulty func.
Am sure he has some super beautiful functionally composable elegant solution up his sleeve though :-)
<t RobustEnumerating <t(
<t input,Func <ienumerable<t, IEnumerable <t> func)
Also there might be a way to keep track of the current & next in the input sequence (and avoid the Skip) but am algorithmically challenged.
Ajai
Sorry for formatting :(
<t RobustEnumerating <t (
<t input, Func <ienumerable<t , IEnumerable <t > func)
<t .Nothing).
<t One <t (T value)
<r TryApply <t,> (T input, Func <t,> func)
<r .Just(func(input));
<r .Nothing;
<t
{
<t Nothing = new Maybe <t ();
<t Just(T value)
<t {Value = value};
argggh, the site engine ate template parameters in my functions so it wont compile straight away but I'm sure it's easy to fix.
Here is a version that works with multiple results:
My attempt.
The code is lazy. In my code, I've duplicated the yield return line and I've got this:
1
1
3
3
5
5
7
7
9
9
http://codepaste.net/gric4t
The original problem is easy to solve with interactive extensions from Rx Framework:
Instead of:
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc))
use:
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc).OnErrorResumeNext())
I have a feeling I know why you need this...
I'm guessing that this relates to building indexes, and that you want to be able to iterate over the LINQ query that generates the index, skipping over items that (for some reason or another) cause an exception to be thrown...
If that's right, then you might try this: http://pastie.org/853146
Basically, I just reimplemented the most basic LINQ operators to swallow exceptions when enumerating. If you ditch System.Linq and just include your own implementation of those methods when you compile the index LINQ queries, you should be able to accomplish this.
@Alex
I don't think that's what OnErrorResumeNext does. OnErrorResumeNext takes an enumerable of enumerables, and iterates over each one of their members in sequence. If one of enumerables throws, it moves to the next enumerable and starts iterating over its items. That doesn't allow you to iterate over a single enumerable in the way Oren described...
Steve,
What if the func return a different number of elements?
You code with either lose some elements or fail itself when there are no elements
Justice,
It doesn't compile
timoconnell ,
Huh?
What happen if I give you a data source over dates?
Mike,
Nice!
Now what happen if I pass in an enumerable that isn't using yield?
Or a different enumerable where state might be something else?
Nick,
I have three different implementations that work :-)
Mike,
What if you can only traverse the enumerable once?
Jay,
This doesn't work with the function provided.
Those enumerables are stateful, and will stop on the first exception
Venu,
The function cannot have any knowledge of the methods I pass to it.
Nick,
Very good observation!
You can still do a lot with it, though.
See my next few posts
Bobby,
You iterate over the source enumerable more than once
Demid,
You hit my second implementation right on the money, but did it much more elegantly!
Nice work
Ajai,
You are iterating over the input enum more than once
Nick,
You are correct about how I run into this.
Wait for the next post on the subject, though, I'll give it full coverage.
And it looks like Linq actually make this easier
Oren,
I bet for each one of your implementations, I can devise an enumerator implementation for which they fail ;)
I really don't think this can work in a perfectly general sense (that is, for any enumerator implementation); assumptions have to be made as to the behavior of the enumerator under certain conditions.
But, if you're expecting to get compiler-generated enumerators, or enumerators from LINQ chains, then you can, indeed, make assumptions about how they work.
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
Pretty sure this is not what you want but it works:
return func(input.Cast <int().Where(i => i % 2 == 1).Cast <t());
Kenneth,
You can't assume the same enumerating function.
Ugly code that needs some tuning.
public static IEnumerable <t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t results = new List <t();
<t {a}));
@ Nick Aceves
You're right - I really made a mistake. Btw, such OnErrorResumeNext is fitting very well on their sequence concept (throw terminates a sequence).
So now it looks like a tricky case to implement this fully on default LINQ combinators and Rx.
My generic type parameters weren't automatically HTML encoded so here's a second attempt encoding them myself:
return func(input.Cast<int>().Where(i => i % 2 == 1).Cast<T>());
public static IEnumerable <t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t enumerator = func(input).GetEnumerator();
Ok. Here is my solution:
<t RobustEnumerating <t (
<t input, Func <ienumerable<t , IEnumerable <t > func)
return input.SelectMany(x =>
<t();
This time I've escaped the angle brackets to see if that slips past the comment sanitizer.
Looking over Dimid's solution though I realize that I am repeatedly enumerating over the input which could have a performance impact if getting these values was expensive. I love the way that he wraps the input to preserve it's state.
Without having looked at previous posted entries, this should do the trick, while still supporting infinite enumerations:
public static IEnumerable <t RobustEnumerating <t
(
<t input,
<ienumerable<t, IEnumerable <t> func
)
{
<t enumerator = func(singleInputElementArray).GetEnumerator();
}
I cannot believe that 95% of all answers are just wrong. It is not that hard... If your call to movenext is not in a try-catch, your code is wrong. if you use foreach, your code is wrong because an exception will skip the rest of the enumeration. Do not call func multiple times, it is not necessary. It will enumerate the sequence multiple times.
Nice challenge however.
@ayende or if MoveNext just sets some other flag before it throws or if MoveNext disposes of itself before it throws or if MoveNext does something like this (assuming for the moment that I still reset the state correctly):
yield return memberCounterThatINeedToBeUniquePerIncomingItem + transformThatWontThrow();
yield return memberCounterThatINeedToBeUniquePerIncomingItem + transformThatWillThrow();
memberCounterThatINeedToBeUniquePerIncomingItem++;
Basically, like Nick says, if we have the requirement that the func can be called only once (which makes sense since it could be keeping track of its position or could make a million expensive remote calls before returning the first item) you can't solve this in a generic way.
the func transforms a non-throwing sequence into a throwing sequence. you can solve it calling it only once.
return input.Select(i =>
@Ayende "Ajai, You are iterating over the input enum more than once"
Here is what I think works (I had to add a method, don't know if it is allowed)
Very nice problem, do you expect the input enumerable to throw too?
Now I need to go get some real work done :-)
Looking forward to what you have, can't take the suspense any longer!
<t EnumOnce <t(IEnumerator <t e)
<t RobustEnumerating <t(
<t input,Func <ienumerable<t, IEnumerable <t> func)
I think it
s more robust implementation with additional class, but it call function only once so if it contains some initialization logic it doesn
t break.<t : IEnumerable <t {
<t enumerator;
<t enumerable) {
<t GetEnumerator() {
<t RobustEnumerating <t (
<t input,Func <ienumerable<t , IEnumerable <t > func)
<t (input));
<int FaultyFunc(IEnumerable <int source)
Wow I will remember this challenge and use it as an interview question. It seems like everyone gets it wrong the first time so you can discuss the problem with the candidate to see if they are smart or not.
Also I do not understand why everyone seems to restart the sequence when an error has occurred. Just do nothing or else the sequence will restart over and over.
I do not understand the purpose of StatefullEnumerable. It seems to do nothing.
This is probably not in the spirit of the challenge, but ...
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<int().Where(x => x % 2 != 0).Cast <t());
...works just fine. Of course it does break encapsulation. :-)
OK. I felt bad about me previous answer that broke encapsulation. Here's an answer that doesn't.
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<ienumerable<t, IEnumerable <t> safeFunc = source =>
<t)null;
Whoops! Now I feel bad about me, err my, grammar... ;-)
I can't believe so many people are so lazy to compile their code at least.
It's not a crime do not know that try/catch block isn't compatible with yield statement but you can at least try to compile it.
Download SnippetCompiler. It is outdated project but it really really good to test your ideas whenever you're not sure how compiler or .net behaves in a particular situation because. It loaded instantly and I ran Oren's example in two seconds after I saw it.
@Demind, I use LINQPad for that purpose. Copy-paste code and it just runs.
@Demid & Dmitriy: vi & csc works too :-)
How about this?
Ok, and what is the BEST solution?
Hi,
Here is my attempt at this:
<t RobustEnumerating <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
<t result;
<t { item };
It looks like the solution is a variation over iterating the input, because that's the only thing we can control, evaluate each item and only return the results of those evaluations that don't fail.
Here's my attempt:
public static IEnumerable <int RobustEnumerating(IEnumerable <int input, Func <ienumerable<int, IEnumerable <int> func)
{
}
lol
@ tobi, it's not hard but I wouldn't say it's intuitive either. Especially for an interview question unless you've wrote library code where you use MoveNext a lot and have an intimate understanding of how it really work.
Now if you've been exposed to this before somewhere else then it's extremely easy :). Thanks Oren and everyone else that comments. I learned a lot from the exposure... especially from Demid code, very elegant indeed.
Maybe its not the best, but it works
public class Program
<t RobustEnumerating <t(
<t input, Func <ienumerable<t, IEnumerable <t> func)
<t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func, ref IEnumerator <t safe)
<int FaultyFunc(IEnumerable <int source)
Simplest possible solution ...
<int RobustEnumerating <t(IEnumerable <t input, Func <ienumerable<t, IEnumerable <t> func)
Comment preview