http://ayende.comAyende @ RahienCopyright (C) Ayende Rahien 2004 - 2021 (c) 202660yes, sometimes such bottlenecks are used rarely enough to leave them such the way they are. Problems start when you want to get all your flow through that bottleneck;) http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment7http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment7Mon, 08 Nov 2010 12:33:50 GMTI've been working with compression and databases for at least 5 years. I can say that you will get data faster... but if you have to update tables, you will have big bottlenecks. For read-only data warehouse is satisfying. http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment6http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment6Mon, 08 Nov 2010 03:05:30 GMTRafal, I'm not aware of any library. but there are lots of oss ones which wouldn't be that hard to adapt, #ziplib would be candidate. currently i'm looking at lzo compression. Gian, good point. But since I don't use the db as an integration database i have no problem when data can only be read by one application http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment5http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment5Thu, 04 Nov 2010 21:48:04 GMTI do not know if compressing directory where the database file are stored is a good idea. It would be useful if the database engine directly supports compression instead of relying on file system storage. Compressing data at application level is something I do not like very much, because it left you with unreadable data into database, so the data could be read only by the application. Indeed this is a good argument, trading CPU time for size could be a good approach. http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment4http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment4Thu, 04 Nov 2010 15:17:33 GMTChris, that's very interesting. I was thinking about using app level compression for message queuing: I've built a message bus on top of an SQL database and wanted to use compression to improve database performance. An application usually has only few types of messages with similar structure (JSON) and repeating contents so I thought that using a shared compression dictionary would be a better solution than compressing each message individually. However, I didn't find any compression API where I could use an external dictionary so this is still only an idea. Do you know such compression library? http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment3http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment3Thu, 04 Nov 2010 03:05:21 GMTwe are currently in very heated debates, just noticed that you put the comment on your blog. we concluded that we will try a different approach. we will implement compression on application level. reason is that we send a lot over the wire and therefor could also save a lot of bandwith. this is much better because then we can remove some latency when communicating with backend system. but first, we will benchmark it and get the real numbers. cloud computing brings interesting times, suddenly cpu/bandwith/io becomes somehow tradeable i currently don't see compression used extensively, for ex. why not create a dictionary and share it between all servers? you don't need to retransmit the dictionary every time. http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment2http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment2Wed, 03 Nov 2010 23:13:10 GMTwith the CPU mostly sitting there twiddling the same byte until that byte threatens to sue for harassment. LOL. I almost fell of the chair. http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment1http://ayende.com/4683/you-are-only-as-fast-as-your-slowest-bottleneck#comment1Wed, 03 Nov 2010 10:52:31 GMT