Ayende @ Rahien

Hi!
My name is Ayende Rahien
Founder of Hibernating Rhinos LTD and RavenDB.
You can reach me by phone or email:

ayende@ayende.com

+972 52-548-6969

@

Posts: 5,947 | Comments: 44,544

filter by tags archive

Challengewrite a self extracting program


I had to do this yesterday, and it was fun enough to post about it. Not only that, but the solution that I came up with is enough of a twist that I think would make this hard for many people.

Here is the challenge:

Given a zip file, produce an executable that when run on another machine, will unzip the content of the zip file on that machine. (nitpicker corner: yes, there are many ways to cheat this, but don’t try to cheat). You may use any library and process that you want, but the end result must be a standalone file that can be copied to an separate machine and when execute will extract the content of the zip file.

You can assume that on that separate machine you will have the .Net framework installed, but you cannot depend on the presence of external libraries. I’ll note that the default compression streams in the .Net framework are not applicable here, since they can’t handle zip files (which contains a directory structure), they can only handle zip streams.

I took me roughly 30 minutes, most of which were spent dealing with ICSharpCode.SharpZipLib API, which is not something that I dealt with before.

More posts in "Challenge" series:

  1. (28 Apr 2015) What is the meaning of this change?
  2. (26 Sep 2013) Spot the bug
  3. (27 May 2013) The problem of locking down tasks…
  4. (17 Oct 2011) Minimum number of round trips
  5. (23 Aug 2011) Recent Comments with Future Posts
  6. (02 Aug 2011) Modifying execution approaches
  7. (29 Apr 2011) Stop the leaks
  8. (23 Dec 2010) This code should never hit production
  9. (17 Dec 2010) Your own ThreadLocal
  10. (03 Dec 2010) Querying relative information with RavenDB
  11. (29 Jun 2010) Find the bug
  12. (23 Jun 2010) Dynamically dynamic
  13. (28 Apr 2010) What killed the application?
  14. (19 Mar 2010) What does this code do?
  15. (04 Mar 2010) Robust enumeration over external code
  16. (16 Feb 2010) Premature optimization, and all of that…
  17. (12 Feb 2010) Efficient querying
  18. (10 Feb 2010) Find the resource leak
  19. (21 Oct 2009) Can you spot the bug?
  20. (18 Oct 2009) Why is this wrong?
  21. (17 Oct 2009) Write the check in comment
  22. (15 Sep 2009) NH Prof Exporting Reports
  23. (02 Sep 2009) The lazy loaded inheritance many to one association OR/M conundrum
  24. (01 Sep 2009) Why isn’t select broken?
  25. (06 Aug 2009) Find the bug fixes
  26. (26 May 2009) Find the bug
  27. (14 May 2009) multi threaded test failure
  28. (11 May 2009) The regex that doesn’t match
  29. (24 Mar 2009) probability based selection
  30. (13 Mar 2009) C# Rewriting
  31. (18 Feb 2009) write a self extracting program
  32. (04 Sep 2008) Don't stop with the first DSL abstraction
  33. (02 Aug 2008) What is the problem?
  34. (28 Jul 2008) What does this code do?
  35. (26 Jul 2008) Find the bug fix
  36. (05 Jul 2008) Find the deadlock
  37. (03 Jul 2008) Find the bug
  38. (02 Jul 2008) What is wrong with this code
  39. (05 Jun 2008) why did the tests fail?
  40. (27 May 2008) Striving for better syntax
  41. (13 Apr 2008) calling generics without the generic type
  42. (12 Apr 2008) The directory tree
  43. (24 Mar 2008) Find the version
  44. (21 Jan 2008) Strongly typing weakly typed code
  45. (28 Jun 2007) Windsor Null Object Dependency Facility

Comments

configurator

Why not use a WinZip self extracting zip? Isn't it customizable enough?

What about creating the exe with the zip file as a resource and whatever library you want, then using ILMerge?

Ayende Rahien

WinZip doesn't give me the chance to add my own code to the mix.

David Whitney

After using SharpZipLib (and being slightly annoyed by it) in the past, last week I did a quick google excercise to see if anything else existed.

http://www.codeplex.com/DotNetZip - Seemed far easier to use (and worked on the Compact Framework, which was the goal last week).

But yeah, grab the source of a zip component, embed the zip as a resource, go go go, no?

Rafal

Few years ago I did the same thing, but without .Net. The trick was simple: wrote a program, compiled it to an exe and appended zip file to the exe file. Exe size was hardcoded in its source code, so the program knew where the zip contents start. The program read its exe file starting at proper position and extracted the data using some zip library (I don't remember what was the library).

E. Muniz

I'm not sure what qualifies as cheating but...

What about making the ZIP file an embedded resource? In other words, use the standard plumbing to extract the resource and then CSharpCode.SharpZipLib API to perform the actual unzip.

devdimi

I have done this in C++, using the Win32 ressouce APIs last year.

You can save anything in an exe file, even if it is a 40mb MSI or zip file.

On execution the program just extracts the saved ressource and does with it whatever it needs to do - unzip it, install it with msiexec or send it the moon.

I guess with Interop you could use the FindRessouce family of functions and do exactly the same thing in .NET

With C++ I needed a little more than 30 minutes for this though :)

Stuart C

I've used the J# libaries to perform zip compression operations in the past, everything required is in the java.util.zip namespace. As for self extracting, I would create a standard setup exe and include the zip file as an embedded resource which is trivial to do.

Matei

I would also go with the embeded resource, as above. I'd embed both the ziplib and the zip file and than use something like Activator.CreateInstance to load whatever types are necesary to unzip the file. I should give this a try and see if it works.

Cheers,

Matei

configurator

@Matei: How would you CreateInstance for the embedded ziplib? Would you embed a dll, unpack it to a temporary location and then run it? Sounds nasty.

Cheeso

DotNetZip takes the "embedded resource" approach for the self-extractors it creates (created using the ZipFile.SaveSelfExtractor() method). Regarding "CreateInstance", there is no CreateInstance since it is .NET, and not COM, but I get your meaning. If the goal is to have a single exe image, then the zip DLL must be embedded as well as the zip archive itself. So how do you call into the library? Well, .NET allows the app to extend the "Assembly Resolution" logic - the way the loader finds libraries. So, the SFX EXE created by DotNetZip uses a custom assembly resolver, and loads the ZIP assembly (DLL) from a byte array that is read from the appropriate embedded resource. Just a few lines of code and no unpacking the DLL to disk. (Yes, .NET can load libraries from in-memory byte streams). Pretty slick. DotNetZip's SFX has 2 embedded resources: one is the zip file and one is the DLL.

WinZip does it slightly differently. A WinZip SFX is native code, does not use .NET. In the Winzip SFX EXE, there is a "Section" with a well-known name, winzip. That contains the zip entry data. The SFX logic just looks for the zip data in that section; which is essentially an "embedded resource" without all the window dressing. But the cool thing is - Because of the way WinZip structures it, the WinZip self-extractor is an EXE, but is also a legal zip archive. if you rename the WinZip sfx exe to .zip, it will (should) be readable by any compliant zip tool, including Windows Explorer. It is both an EXE and a ZIP. A WinZip-created SFX EXE can be loaded into the WinZip tool (or any zip tool) and examined visually.

Info-Zip takes the "known exe length" trick to build its SFX files.

Matei

@configurator

As Cheeso says we can load an assembly from a byte[]; from there on it becomes and exercise in reflection, invoking the methods/properties on the dynamically loaded types to extract the zip file from the stream we get from the resource. So no hard dependency to the library, the only file that needs to be deployed is the .exe. The ILMerge is also a viable option as far as I'm concerned - bothe you and Simon pointed this out.

Here's something quick and dirty I put toghether. Took me more than 30 mins :) I get no practice with reflection, nor do I know the SharpZipLibApi.

static void Main(string[] args)

    {

        Assembly zipLib =

            GetAssemblyFromStream(

                Assembly.GetExecutingAssembly().GetManifestResourceStream("Engine.lib.ICSharpCode.SharpZipLib.dll"));

        Type inputStreamType = zipLib.GetType("ICSharpCode.SharpZipLib.Zip.ZipInputStream");

        Type entryType = zipLib.GetType("ICSharpCode.SharpZipLib.Zip.ZipEntry");


        Stream inputStream = (Stream)Activator.CreateInstance(inputStreamType,

            new object[] {

                            Assembly.GetExecutingAssembly().

                            GetManifestResourceStream("Engine.assets.Console2.zip") });

        object zipEntry;

        while ((zipEntry = inputStreamType.InvokeMember("GetNextEntry", BindingFlags.InvokeMethod, null, inputStream, null)) != null)

        {

            string path = (string)entryType.InvokeMember("Name", BindingFlags.GetProperty, null, zipEntry, null);


            string directoryName = Path.GetDirectoryName(path);

            if (directoryName.Length > 0)

            {

                Directory.CreateDirectory(directoryName);

            }


            string fileName = Path.GetFileName(path);

            if (fileName != String.Empty)

            {

                using (FileStream streamWriter = File.Create(path))

                {


                    int size = 2048;

                    byte[] data = new byte[2048];

                    while (true)

                    {

                        size = (int)inputStreamType.InvokeMember("Read", BindingFlags.InvokeMethod, null, inputStream, new object[] { data, 0, data.Length }); //zipStream.Read(data, 0, data.Length);

                        if (size > 0)

                        {

                            streamWriter.Write(data, 0, size);

                        }

                        else

                        {

                            break;

                        }

                    }

                }

            }


        }


        Console.ReadKey();

    }
Cheeso

Just one note - You do not need to resort to reflection to invoke the methods and properties on the dynamically-loaded assembly. If you use CreateInstance, I suppose you DO, but if you just load the library "normally" - via a ResolveEventHandler that loads the byte strea - then your code won't have to rely on CreateInstance or reflection.

    static System.Reflection.Assembly Resolver(object sender, ResolveEventArgs args)

    {

        Assembly a1 = Assembly.GetExecutingAssembly();

        Stream s = a1.GetManifestResourceStream("Ionic.Zip.dll");

        byte[] block = new byte[s.Length];

        s.Seek(0, System.IO.SeekOrigin.Begin);

        s.Read(block, 0, block.Length);


        Assembly a2 = Assembly.Load(block);


        return a2;

    }

At compile time you'd reference the dynamically-loaded DLL (DotNetZip or SharpZipLib or whatever) but you would not need to distribute that DLL separately. It'd be embedded as a resource, and it would be loaded via your ResolveEventHandler.

This is the approach DotNetZip takes in its SFX files.

Nair

I had the same problem. Where i need to create a single exe with all resources in it. First I went with .Net approach with Resource files and since my resources are over 640MB my compiler failed. I dropped that I idea and went to ILMerge route and that also had the same limiation. So I ended up going to inno setup which is more like installshield but open source.

My file size is 1.2GB, do you think I would able to use SharpLib and resolve this problem? I thought SharpLib had a limiation of subdirectories??

Comment preview

Comments have been closed on this topic.

FUTURE POSTS

No future posts left, oh my!

RECENT SERIES

  1. RavenDB Sharding (3):
    22 May 2015 - Adding a new shard to an existing cluster, splitting the shard
  2. The RavenDB Comic Strip (2):
    20 May 2015 - Part II – a team in trouble!
  3. Challenge (45):
    28 Apr 2015 - What is the meaning of this change?
  4. Interview question (2):
    30 Mar 2015 - fix the index
  5. Excerpts from the RavenDB Performance team report (20):
    20 Feb 2015 - Optimizing Compare – The circle of life (a post-mortem)
View all series

RECENT COMMENTS

Syndication

Main feed Feed Stats
Comments feed   Comments Feed Stats