<< Git is teh SUCK | Home | Scenario based testing in Rhino DivanDB >>

Challenge: Robust enumeration over external code

Here is an interesting little problem:

public class Program
{
    private static void Main()
    {
        foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc))
        {
            Console.WriteLine(i);
        }
    }

    public static IEnumerable<T> RobustEnumerating<T>(
        IEnumerable<T> input,Func<IEnumerable<T>, IEnumerable<T>> func)
    {
        // how to do this?
        return func(input);

    }

    public static IEnumerable<int> FaultyFunc(IEnumerable<int> source)
    {
        foreach (int i in source)
        {
            yield return i/(i%2);
        }
    }
}

This code should not throw, but print:

1
3
5
7
9

Can you make this happen? You can only change the RobustEnumerating method, nothing else in the code

Print | posted on Thursday, March 04, 2010 3:54 AM

Feedback

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:00 AM Juan Lopes

public static IEnumerable RobustEnumerating(
IEnumerable input,Func, IEnumerable> func)
{
foreach(T t in func(input)) {
try { yield return t; }
catch { }
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:03 AM Ayende Rahien

Juan,
That code doesn't compiles

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:21 AM Michael Stum

Not tested and ugly and possibly cheating, but:

public static IEnumerable RobustEnumerating(
IEnumerable input,Func, IEnumerable> func)
{

foreach(T obj in input) {
try {
var newList = new List();
newList.Add(obj);
yield return func(newList)
} catch {}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:28 AM Juan Lopes

It doesn't work either, because the exception occurs at MoveNext method of the enumerator. After that, the enumerator becomes unusable.

I don't think there is a solution. =/

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:29 AM Ayende Rahien

Michael,
Your code won't compile

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:29 AM David

Stealing Michael's idea:

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
foreach (T item in input)
{
T funcedItem = default(T);
bool worked;
try
{
var funcedList = func(new [] {item});
funcedItem = funcedList.First();
worked = true;
}
catch
{
worked = false;
}
if (worked) yield return funcedItem;
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:29 AM Dennis

The problem is that the whole IEnumerable is evaluated upon the first call to the MoveNext, therefore it is not possible to have a try/catch around each.
I give up :(

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:31 AM vcsjones

This is terrible, but the only way I could get this to work:


public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
List holder = new List();
foreach (var i in input)
{
try
{
holder.AddRange(func(Enumerable.Repeat(i, 1)));
}
catch
{
}
}
return holder;
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:34 AM Ayende Rahien

David,
That is an workable implementation, yes. It has some bugs (there isn't 1:1 mapping between result and reply)
Now try doing this without invoking the enumerator for every single item (initialization cost may be expensive).
There are at least two additional ways of doing it

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:34 AM pb

This works.. And I learned something about enumerables and exceptions today.

private static void Main()
{
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc))
{
Console.WriteLine(i);
}
}

public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
IEnumerator enumerator = input.GetEnumerator();
while (enumerator.MoveNext())
{
T t = enumerator.Current;

T value = default(T);
bool hasValue = false;
try
{
value = func(new[] { t }).First();
hasValue = true;
}
catch (Exception)
{
}

if(hasValue)
yield return value;
}

}

public static IEnumerable FaultyFunc(IEnumerable source)
{
foreach (int i in source)
{
yield return i / (i % 2);
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:34 AM Ayende Rahien

Dennis,
If it isn't a lazy enumerator, you can't do anything, right.
But the code I have shown IS lazy

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:35 AM Ayende Rahien

vsjones,
Your approach won't work with infinite enumerators, can you make it lazy?

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:36 AM Ayende Rahien

pb,
What happens if a function returns two output values for every input value?

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:41 AM Dennis

David, that wont work.
What if the FaultyFuncsaid
public static IEnumerable FaultyFunc(IEnumerable source)
{
Console.WriteLine("Called me");
foreach (int i in source)
{
yield return i/(i%2);
}
}
You cannot presume the function is without sideeffects

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:47 AM Steve Py

It ain't pretty, but it works. :)

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
// how to do this?
foreach(T value in input)
{
T test = default(T);
bool match = false;
try
{
test = func(new T[] { value }).ElementAt(0);
match = true;
}
catch
{ }
if (match)
yield return test;
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:48 AM Justice

var e = func(input).GetEnumerator();
while(true) {
try {
if(!e.MoveNext()) yield break;
yield return e.Current;
}
catch { }
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:49 AM timoconnell

Not pretty also, but...

public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
// how to do this?

IEnumerator enumerator = input.GetEnumerator();
List output = new List();
while (enumerator.MoveNext())
{
if (int.Parse(enumerator.Current.ToString()) % 2 != 0)
{
output.Add((T)enumerator.Current);
}
}

return func((IEnumerable)output);
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:50 AM Steve Py

dam people reply fast. :) Same as pb.

Hehe, your original challenge was to make the example work. ;P

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:55 AM Steve Py

Should satisfy Ayende's adjustment:

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
// how to do this?
foreach(T value in input)
{
List results = new List();
bool match = false;
try
{
results = func(new T[] { value }).ToList();
match = true;
}
catch
{ }
if (match)
foreach (T item in results)
{
yield return item;
}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:57 AM Mike Thomas

Should work - only calls func once so if it has side effects we are fine - works with infinite enumerations :)

public class Program {
private static void Main() {
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc)) {
Console.WriteLine(i);
}
}

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func) {

IEnumerable enumerable = func(input);
IEnumerator enumerator = null;

try {
enumerator = enumerable.GetEnumerator();


while(true) {
bool fault = false;
T t = default(T);

try {
if(!enumerator.MoveNext()) {
break;
}

t = enumerator.Current;

}
catch {
var field = enumerator.GetType().GetField("<>1__state",
BindingFlags.NonPublic | BindingFlags.Instance);

field.SetValue(enumerator, 2);
fault = true;
}

if(!fault) {
yield return t;
}
}
}
finally {
enumerator.Dispose();
}
}

public static IEnumerable FaultyFunc(IEnumerable source) {
foreach (int i in source) {
yield return i / (i % 2);
}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 4:59 AM pb

Since Steve copied me I'll say ditto to his... :)

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:00 AM Nick Aceves

@mike

You stolez my codez! ;)

@oren

This isn't possible if you think about it. If MoveNext throws you have NO idea what the state of the enumerator is. What if it's a custom Ienumerator implementation? The c# codegened ones fail gracefully, but that's not guaranteed if I write one myself.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:01 AM Matt

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
if (!input.Any()) return Enumerable.Empty();

var result = new List();
try
{
var goodResult = func(new[] {input.First()});
result.Add(goodResult.First());
return result.Union(RobustEnumerating(input.Skip(1), func));
}
catch
{
return RobustEnumerating(input.Skip(1), func);
}

}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:01 AM Juan Lopes

Nice Mike. :)

Never thought on reseting the state of the enumerator. This sounds a bit of hack, but it works :)

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:04 AM yh

Steve Py:
you can remove the bool match flag.. as the list will be empty in case of exception...

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:04 AM Jay

What about something like below?

It works, but I feel there has to be a cheaper way to do it.


public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
foreach (var inputItem in input)
{
var tempInput = new[] { inputItem };
var buffer = new List();
var funcEnumerator = func(tempInput).GetEnumerator();
var flag = true;

while (flag)
{
try
{
if (funcEnumerator.MoveNext())
{
buffer.Add(funcEnumerator.Current);
}
else
{
flag = false;
}
}
catch { }
}

foreach (var tempItem in buffer)
{
yield return tempItem;
}
}
}



Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:33 AM Rob

lazy, concise, expressive.
it'd be even better if I had an immutable list

return
input
.Aggregate(new List(),
(acc, i) =>
{
try
{
acc.Add(func(new[] { i }).Single());
} catch { }
return acc;
});

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:35 AM Mike Thomas

@Jual Lopez yeah, just a "little" hack :)

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:37 AM Rob

Oops. Change
.Add(......Single())
to
.AddRange(.....)

to handle multiple returns for a single input

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:40 AM Dmitriy Nagirnyak

public static IEnumerable RobustEnumerating(
IEnumerable input,Func, IEnumerable> func)
{
var res = new List();
var tmp = new T[1];
foreach(var i in input) {
try {
tmp[0] = i;
res.AddRange( func(tmp) );
} catch {}
}
return res;
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 6:31 AM Venu

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
// how to do this?
IList list = new List();
int counter = 0;
foreach (var enumerable in input)
{
if (counter % 2 != 0)
list.Add(enumerable);
counter++;
}
input = list.AsEnumerable();
return func(input);

}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 6:33 AM Steve Py

@nb != copy. :) When I started the current post was from Ayende about Juan's not compiling. I was shocked after posting to see some 10 other responses including virtually the same solution.

@yh Yeah, could just check for the empty list... Hacking that together on my lunch break. :)

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 6:38 AM Rob

Final attempt.
it turns out Aggregate isn't lazy.
The only trick here is the .ToList on the func result to force the error to occur inside the try catch rather than later when selectmany is flattening lists.

return
input
.SelectMany(i =>
{
try
{
return func(new[] { i }).ToList();
} catch {
return new List();
}
});

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 6:58 AM Nick Aceves

@mike,

Your code only works assuming the enumerator has 2 states.

Take the following, for example:

private static void Throw() {
throw new Exception();
}

public static IEnumerable FaultyFunc2(IEnumerable source) {
yield return 1;
yield return 2;
yield return 3;
Throw();
yield return 4;
}

This will 5 different states, and your implementation sends it into an infinite loop.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 7:23 AM Nick Aceves

Also, Mike's code fails when the enumerator disposes of resources in a finally block. The code below sends it into an infinite loop even though there are only 2 states and state 2 is the "get me the next item" state. Attempting to continue iteration at that point leads to an ObjectDisposedException, which promptly gets swallowed and "iteration" continues.

My code-sense is telling me that unless you have an Oracle for the Halting Problem, this can't be done in such a manner that you only initialize the enumerator one time, or in a way that handles every possible enumerator implementation. You will have to impose constrains upon what "types" of enumerators you will consume. Or, better yet, remove this requirement altogether by changing the assumptions/requirements in whatever code needs this.

public static IEnumerable FaultyFunc3(IEnumerable source) {
var file = new StringBuilder().AppendLine("1").AppendLine("2").AppendLine("<>").AppendLine("3").AppendLine("4").ToString();
var bytes = Encoding.Default.GetBytes(file);

using(var stream = new StreamReader(new MemoryStream(bytes))) {
while(!stream.EndOfStream)
yield return int.Parse(stream.ReadLine());
}
}





Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:02 AM Paul Batum

I gave up reading the suggestions, its just too annoying to read code that isn't html encoded and formatted properly. It would be better if people were posting codepaste/github gist/etc links.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:11 AM Leyu Sisay

It ain't pretty.

public static IEnumerable RobustEnumerating( IEnumerable input, Func, IEnumerable> func)
{
var result = new List();

var skippedInput = input;
while(true)
{
var skip = 1;
try
{
foreach (var value in func(skippedInput))
{
skip++;
result.Add(value);
}
break;
}
catch
{
skippedInput = skippedInput.Skip(skip);
}
}

return result;
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:17 AM Bobby Diaz

How about this? (never said we couldn't add methods...)

public class Program
{
public static void Main(string[] args)
{
foreach ( int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc) )
{
Console.WriteLine(i);
}

Console.WriteLine("\n");

foreach ( int i in RobustEnumerating(Enumerable.Range(0, 10).Concat(Enumerable.Range(0, 10)).OrderBy(i => i), FaultyFunc) )
{
Console.WriteLine(i);
}

Console.Read();
}

public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
return input.SelectMany(i => RecursiveEnumerator(new[] { i }, func));
}

private static IEnumerable RecursiveEnumerator(IEnumerable input, Func, IEnumerable> func)
{
var list = new List();

try
{
list.AddRange(func(input));
}
catch ( Exception )
{
list.AddRange(RecursiveEnumerator(input.Skip(list.Count + 1), func));
}

foreach ( var item in list )
{
yield return item;
}
}

public static IEnumerable FaultyFunc(IEnumerable source)
{
foreach ( int i in source )
{
yield return i / ( i % 2 );
}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:38 AM Andrey Titov 


public static IEnumerable<T> RobustEnumerating<T>(
    IEnumerable<T> input, Func<IEnumerable<T>, IEnumerable<T>> func)
{
    foreach (var item in input)
    {
        bool isValid = false;
        T current = default(T);

        try
        {
            current = func(new[] { item }).Single();
            isValid = true;
        }
        catch { }

        if (isValid)
        {
            yield return current;
        }
    }            
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:39 AM Max Uvw

Something something functional ;)

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
return input.
Select(i => TryApply(One(i), o => func(o).First())).
Where(m => m != Maybe.Nothing).
Select(m => m.Value);
}

private static IEnumerable One(T value)
{
yield return value;
}

private static Maybe TryApply(T input, Func func)
{
try {
return Maybe.Just(func(input));
} catch {
return Maybe.Nothing;
}
}

private sealed class Maybe
{
public static readonly Maybe Nothing = new Maybe();
public T Value { get; private set; }
public static Maybe Just(T value)
{
return new Maybe {Value = value};
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:43 AM bob

try the following code

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
// how to do this?
var newInput = from i in input
where Convert.ToInt32(i.ToString()) % 2 != 0
select i;
return func(newInput);

}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:49 AM Demid

It's a little bit ugly but it compiles, returns what you need and doing it in lazy manner:

public static IEnumerable Wrap(IEnumerator input)
{
while(input.MoveNext())
yield return input.Current;
}

public static IEnumerable RobustEnumerating(
IEnumerable input,Func, IEnumerable> func)
{
var wrapped = Wrap(input.GetEnumerator());
var en = func(wrapped).GetEnumerator();
do
{
bool? b = MoveNext(en);
if (b == false) yield break;
else if (b == true) yield return en.Current;
else en = func(wrapped).GetEnumerator();
} while(true);
}

public static bool? MoveNext(IEnumerator en)
{
try
{
return en.MoveNext();
} catch {}
return null;
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:50 AM Franck

What about this?

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{

bool success = true;
foreach(T el in input)
{
try
{
func(new List { el }.AsEnumerable()).FirstOrDefault();
success = true;
}
catch
{
success = false;
}
if (success) yield return func(new List { el }.AsEnumerable()).FirstOrDefault();
}

}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:50 AM Ajai Shankar

Since Ayende kept saying input & output need not be 1:1 here is a solution that keeps track of the input enumerator and tries to minimize initialization of faulty func.

Am sure he has some super beautiful functionally composable elegant solution up his sleeve though :-)

public static IEnumerable RobustEnumerating(
IEnumerable input,Func, IEnumerable> func)
{
int lastIndex = -1;
while(true) {
var result = func(
input.Skip(lastIndex + 1)
.Select(item => { lastIndex += 1; return item; })
).GetEnumerator();

while(true) {
bool done = false;
try {
done = !result.MoveNext();
} catch(Exception) {
break;
}

if(done)
yield break;
else
yield return result.Current;
}
}
}

Also there might be a way to keep track of the current & next in the input sequence (and avoid the Skip) but am algorithmically challenged.

Ajai

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:52 AM Max Uvw

Sorry for formatting :(


        public static IEnumerable RobustEnumerating(
                    IEnumerable input, Func, IEnumerable> func)
        {
            return input.
                    Select(i => TryApply(One(i), o => func(o).First())).
                    Where(m => m != Maybe.Nothing).
                    Select(m => m.Value);
        }

        private static IEnumerable One(T value)
        {
            yield return value;
        }

        private static Maybe TryApply(T input, Func func)
        {
            try {
                return Maybe.Just(func(input));
            } catch {
                return Maybe.Nothing;
            }
        }

        private sealed class Maybe
        {
            public static readonly Maybe Nothing = new Maybe();
            public T Value { get; private set; }
            public static Maybe Just(T value)
            {
                return new Maybe {Value = value};
            }
        }

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:53 AM Demid

argggh, the site engine ate template parameters in my functions so it wont compile straight away but I'm sure it's easy to fix.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:54 AM Dmitry

Here is a version that works with multiple results:

public static IEnumerable<T> RobustEnumerating<T>(IEnumerable<T> input, Func<IEnumerable<T>, IEnumerable<T>> func)
{
foreach (var item in input)
{
bool valid;

IEnumerable<T> results = new T[0];

try
{
results = func(new[] { item }).ToArray();
valid = true;
}
catch
{
valid = false;
}

if (valid)
{
foreach (var result in results)
{
yield return result;
}
}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 9:54 AM Tran Ngoc Van

My attempt.

The code is lazy. In my code, I've duplicated the yield return line and I've got this:

1
1
3
3
5
5
7
7
9
9

http://codepaste.net/gric4t

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 9:57 AM Alex Yakunin

The original problem is easy to solve with interactive extensions from Rx Framework:

Instead of:
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc))

use:
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc).OnErrorResumeNext())

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 9:59 AM Nick Aceves

I have a feeling I know why you need this...

I'm guessing that this relates to building indexes, and that you want to be able to iterate over the LINQ query that generates the index, skipping over items that (for some reason or another) cause an exception to be thrown...

If that's right, then you might try this: http://pastie.org/853146

Basically, I just reimplemented the most basic LINQ operators to swallow exceptions when enumerating. If you ditch System.Linq and just include your own implementation of those methods when you compile the index LINQ queries, you should be able to accomplish this.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:19 AM Nick Aceves

@Alex

I don't think that's what OnErrorResumeNext does. OnErrorResumeNext takes an enumerable of enumerables, and iterates over each one of their members in sequence. If one of enumerables throws, it moves to the next enumerable and starts iterating over its items. That doesn't allow you to iterate over a single enumerable in the way Oren described...

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:40 AM Ayende Rahien

Steve,
What if the func return a different number of elements?
You code with either lose some elements or fail itself when there are no elements

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:40 AM Ayende Rahien

Justice,
It doesn't compile

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:41 AM Ayende Rahien

timoconnell ,
Huh?
What happen if I give you a data source over dates?

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:44 AM Ayende Rahien

Mike,
Nice!
Now what happen if I pass in an enumerable that isn't using yield?
Or a different enumerable where state might be something else?

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:45 AM Ayende Rahien

Nick,
I have three different implementations that work :-)

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:46 AM Ayende Rahien

Mike,
What if you can only traverse the enumerable once?

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:48 AM Ayende Rahien

Jay,
This doesn't work with the function provided.
Those enumerables are stateful, and will stop on the first exception

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:49 AM Ayende Rahien

Venu,
The function cannot have any knowledge of the methods I pass to it.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:52 AM Ayende Rahien

Nick,
Very good observation!
You can still do a lot with it, though.
See my next few posts

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:53 AM Ayende Rahien

Bobby,
You iterate over the source enumerable more than once

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:54 AM Ayende Rahien

Demid,
You hit my second implementation right on the money, but did it much more elegantly!
Nice work

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 10:56 AM Ayende Rahien

Ajai,
You are iterating over the input enum more than once

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 11:01 AM Ayende Rahien

Nick,
You are correct about how I run into this.
Wait for the next post on the subject, though, I'll give it full coverage.
And it looks like Linq actually make this easier

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 11:15 AM Nick Aceves

Oren,

I bet for each one of your implementations, I can devise an enumerator implementation for which they fail ;)

I really don't think this can work in a perfectly general sense (that is, for *any* enumerator implementation); assumptions have to be made as to the behavior of the enumerator under certain conditions.

But, if you're expecting to get compiler-generated enumerators, or enumerators from LINQ chains, then you can, indeed, make assumptions about how they work.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 11:56 AM Kenneth

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
var dd = input.Where(x =>
{
string s = x.ToString();
int i = 0;
if (int.TryParse(s, out i))
{
if (i % 2 == 0) return false;
return true;
}
return false;
});
// how to do this?
return func(dd);
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 11:59 AM Koen

Pretty sure this is not what you want but it works:
return func(input.Cast().Where(i => i % 2 == 1).Cast());

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 11:59 AM Ayende Rahien

Kenneth,
You can't assume the same enumerating function.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 12:03 PM Benny Thomas

Ugly code that needs some tuning.

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{

foreach(T a in input)
{
List results = new List();
try
{
results.AddRange(func(new List {a}));
}
catch
{
// Do some faulthandling here
}
foreach (var result in results)
{
yield return result;
}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 12:14 PM Alex Yakunin

@ Nick Aceves

You're right - I really made a mistake. Btw, such OnErrorResumeNext is fitting very well on their sequence concept (throw terminates a sequence).

So now it looks like a tricky case to implement this fully on default LINQ combinators and Rx.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 12:28 PM Koen

My generic type parameters weren't automatically HTML encoded so here's a second attempt encoding them myself:
return func(input.Cast<int>().Where(i => i % 2 == 1).Cast<T>());

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 1:14 PM Sergey Kachkovsky

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
IEnumerator enumerator = func(input).GetEnumerator();
do
{
bool valid = true;
try
{
if (!enumerator.MoveNext()) yield break;
}
catch
{
valid = false;
}
if (valid)
yield return enumerator.Current;
} while (true);
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 1:26 PM Mike Doherty

Ok. Here is my solution:


public static IEnumerable RobustEnumerating(
    IEnumerable input, Func, IEnumerable> func)
{
    var rowsToSkipOnError = 1;
    var enumerationToWrap = func(input).GetEnumerator();
    while (true)
    {
        T nextValue;
        try
        {
            if (!enumerationToWrap.MoveNext())
                yield break;

            nextValue = enumerationToWrap.Current;
        }
        catch (Exception)
        {
            // This input failed so skip to the next one
            input = input.Skip(rowsToSkipOnError);
            enumerationToWrap = func(input).GetEnumerator();
            rowsToSkipOnError = 1;
            continue;
        }

        rowsToSkipOnError++;
        yield return nextValue;
    }
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 1:30 PM Arielr

return input.SelectMany(x =>
{
try
{
return func(x.AsEnumerableX()).ToArray();
}
catch
{
return Enumerable.Empty();
}
});

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 1:49 PM Mike Doherty

This time I've escaped the angle brackets to see if that slips past the comment sanitizer.

Looking over Dimid's solution though I realize that I am repeatedly enumerating over the input which could have a performance impact if getting these values was expensive. I love the way that he wraps the input to preserve it's state.


public static IEnumerable<T> RobustEnumerating<T>(
   IEnumerable<T> input, Func<IEnumerable<T>, IEnumerable<T>> func)
{
   var rowsToSkipOnError = 1;
   var enumerationToWrap = func(input).GetEnumerator();
   while (true)
   {
       T nextValue;
       try
       {
           if (!enumerationToWrap.MoveNext())
               yield break;

           nextValue = enumerationToWrap.Current;
       }
       catch (Exception)
       {
           // This input failed so skip to the next one
           input = input.Skip(rowsToSkipOnError);
           enumerationToWrap = func(input).GetEnumerator();
           rowsToSkipOnError = 1;
           continue;
       }

       rowsToSkipOnError++;
       yield return nextValue;
   }
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 2:17 PM Frank

Without having looked at previous posted entries, this should do the trick, while still supporting infinite enumerations:

public static IEnumerable RobustEnumerating
(
IEnumerable input,
Func, IEnumerable> func
)
{
var singleInputElementArray = new T[1];
foreach (var inputElement in input)
{
singleInputElementArray[0] = inputElement;

IEnumerator enumerator = func(singleInputElementArray).GetEnumerator();
bool success;
do
{
success = false;

try
{
if (enumerator.MoveNext())
success = true;
}
catch
{
}

if (success)
yield return enumerator.Current;
}
while (success);
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 3:00 PM Sergey Shishkin

return input.SelectMany(x =>
{
try
{
return new[]{func(new[] { x }).FirstOrDefault()};
}
catch
{
return new T[0];
}
});

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 3:16 PM tobi

I cannot believe that 95% of all answers are just wrong. It is not that hard... If your call to movenext is not in a try-catch, your code is wrong. if you use foreach, your code is wrong because an exception will skip the rest of the enumeration. Do not call func multiple times, it is not necessary. It will enumerate the sequence multiple times.
Nice challenge however.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:10 PM Mike Thomas

@ayende or if MoveNext just sets some other flag before it throws or if MoveNext disposes of itself before it throws or if MoveNext does something like this (assuming for the moment that I still reset the state correctly):

yield return memberCounterThatINeedToBeUniquePerIncomingItem + transformThatWontThrow();

yield return memberCounterThatINeedToBeUniquePerIncomingItem + transformThatWillThrow();

memberCounterThatINeedToBeUniquePerIncomingItem++;

Basically, like Nick says, if we have the requirement that the func can be called only once (which makes sense since it could be keeping track of its position or could make a million expensive remote calls before returning the first item) you can't solve this in a generic way.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:33 PM tobi

the func transforms a non-throwing sequence into a throwing sequence. you can solve it calling it only once.

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 5:38 PM Michal

return input.Select(i =>
{
try
{
return func(new[] { i }).First();
}
catch
{
return default(T);
}
}).
Where(i => !i.Equals(default(T)));

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 7:09 PM Ajai Shankar

@Ayende "Ajai, You are iterating over the input enum more than once"

Here is what I think works (I had to add a method, don't know if it is allowed)

Very nice problem, do you expect the input enumerable to throw too?

Now I need to go get some real work done :-)

Looking forward to what you have, can't take the suspense any longer!

static IEnumerable EnumOnce(IEnumerator e)
{
while(e.MoveNext())
yield return e.Current;
}

public static IEnumerable RobustEnumerating(
IEnumerable input,Func, IEnumerable> func)
{
input = EnumOnce(input.GetEnumerator());

var result = func(input).GetEnumerator();
while(true) {
try {
if(!result.MoveNext())
break;
} catch(Exception) {
result = func(input).GetEnumerator();
continue;
}
yield return result.Current;
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 7:44 PM Kvasov Roman

I think it`s more robust implementation with additional class, but it call function only once so if it contains some initialization logic it doesn`t break. 


using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;

public class Program
{
    public class StatefullEnumerable : IEnumerable {

        private IEnumerator enumerator;

        public StatefullEnumerable(IEnumerable enumerable) {
            enumerator = enumerable.GetEnumerator();
        }

        public IEnumerator GetEnumerator() {
            return enumerator;
        }

        IEnumerator IEnumerable.GetEnumerator() {
            return GetEnumerator();
        }
    }

    private static void Main()
    {
        foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc))
        {
            Console.WriteLine(i);
        }
    }

    public static IEnumerable RobustEnumerating(
        IEnumerable input,Func, IEnumerable> func)
    {
        var enumerable = func(new StatefullEnumerable(input));
        var enumerator = enumerable.GetEnumerator();
        while (true) {
            var fail = false;
            var value = default(T);
            try {
                if (!enumerator.MoveNext())
                    yield break;
                value = enumerator.Current;
            }
            catch (Exception) {
                fail = true;
                enumerator = enumerable.GetEnumerator();
            }
            if (!fail)
                yield return value;
        }
        // how to do this?
    }

    public static IEnumerable FaultyFunc(IEnumerable source)
    {
        foreach (int i in source)
        {
            yield return i/(i%2);
        }
    }
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/4/2010 8:52 PM tobi

Wow I will remember this challenge and use it as an interview question. It seems like everyone gets it wrong the first time so you can discuss the problem with the candidate to see if they are smart or not.

Also I do not understand why everyone seems to restart the sequence when an error has occurred. Just do nothing or else the sequence will restart over and over.

I do not understand the purpose of StatefullEnumerable. It seems to do nothing.

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 1:46 AM James C-S

This is probably not in the spirit of the challenge, but ...

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
return func(input.Cast().Where(x => x % 2 != 0).Cast());
}

...works just fine. Of course it does break encapsulation. :-)

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 2:10 AM James C-S

OK. I felt bad about me previous answer that broke encapsulation. Here's an answer that doesn't.

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
Func, IEnumerable> safeFunc = source =>
{
var @return = (IEnumerable)null;
try { @return = func(source).ToArray(); }
catch (System.DivideByZeroException) { }
return @return;
};
return (from i2 in
(from i in input
select new T[] { i })
let o2 = safeFunc(i2)
where o2 != null
select o2).SelectMany(o => o);
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 2:11 AM James C-S

Whoops! Now I feel bad about me, err my, grammar... ;-)

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 2:13 AM Demid

I can't believe so many people are so lazy to compile their code at least.

It's not a crime do not know that try/catch block isn't compatible with yield statement but you can at least try to compile it.

Download SnippetCompiler. It is outdated project but it really really good to test your ideas whenever you're not sure how compiler or .net behaves in a particular situation because. It loaded instantly and I ran Oren's example in two seconds after I saw it.

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 2:57 AM Dmitriy Nagirnyak

@Demind, I use LINQPad for that purpose. Copy-paste code and it just runs.

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 4:01 AM Ajai Shankar

@Demid & Dmitriy: vi & csc works too :-)

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 4:38 AM Ken Tong

How about this?


public static IEnumerable<T> RobustEnumerating<T>(
    IEnumerable<T> input, Func<IEnumerable<T>, IEnumerable<T>> func)
{
    IEnumerable<T> results = new T[0];
 
    foreach (T item in input)
    {
        try
        {
            foreach (T value in func(new[] { item }))
                results = results.Concat(new[] { value });
        }
        catch { }
    }
 
    foreach (T result in results)
        yield return result;
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 12:58 PM Sergey Kachkovsky

Ok, and what is the BEST solution?

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 1:04 PM Allan PArker

Hi,

Here is my attempt at this:


public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
foreach (T item in input)
{
IEnumerable result;
var list = new List { item };

try
{
result = func(list);
var test = result.First();
}
catch
{
continue;
}

yield return result.First();
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 3:59 PM Thomas Eyde

It looks like the solution is a variation over iterating the input, because that's the only thing we can control, evaluate each item and only return the results of those evaluations that don't fail.

Here's my attempt:

foreach (var item in input)
{
T[] evaluatedItems;
try
{
var itemAsEnumerable = new[] { item };
evaluatedItems = func(itemAsEnumerable).ToArray();
}
catch (Exception)
{
continue;
}

foreach (var evaluated in evaluatedItems)
{
yield return evaluated;
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 8:30 PM Chris Martin

public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
return func(input.ToList().ConvertAll(t => int.Parse(t.ToString())).Where(i => i%2 > 0));
}

lol

Gravatar

# re: Challenge: Robust enumeration over external code 3/5/2010 11:44 PM firefly

@ tobi, it's not hard but I wouldn't say it's intuitive either. Especially for an interview question unless you've wrote library code where you use MoveNext a lot and have an intimate understanding of how it really work.

Now if you've been exposed to this before somewhere else then it's extremely easy :). Thanks Oren and everyone else that comments. I learned a lot from the exposure... especially from Demid code, very elegant indeed.

Gravatar

# re: Challenge: Robust enumeration over external code 3/9/2010 10:01 AM sergio

Maybe its not the best, but it works
public class Program
{
private static void Main()
{
foreach (int i in RobustEnumerating(Enumerable.Range(0, 10), FaultyFunc))
{
Console.WriteLine(i);
}
Console.ReadLine();
}

public static IEnumerable RobustEnumerating(
IEnumerable input, Func, IEnumerable> func)
{
// how to do this?
var enumerator = func(input).GetEnumerator();
while (SafeMoveNextEnumerator(input, func, ref enumerator))
{
yield return enumerator.Current;
}



}
static int skip = 0;

public static bool SafeMoveNextEnumerator(IEnumerable input, Func, IEnumerable> func, ref IEnumerator safe)
{
while (true)
{
skip++;
try
{
return safe.MoveNext();
}
catch
{
var e = Enumerable.Skip(input, skip);
safe = func(e).GetEnumerator();
}
}
}

public static IEnumerable FaultyFunc(IEnumerable source)
{
foreach (int i in source)
{
yield return i / (i % 2);
}
}
}

Gravatar

# re: Challenge: Robust enumeration over external code 3/11/2010 12:07 PM James

Simplest possible solution ...

public static IEnumerable RobustEnumerating(IEnumerable input, Func, IEnumerable> func)
{
return new[] { 1, 3, 5, 7, 9 };
}

Comments have been closed on this topic.