reWhy IndexedDB is slow and what to use instead

time to read 2 min | 322 words

I ran into this post and I thought that I would share my thinking on the matter. I don’t actually know anything about IndexedDB, but I have been working with databases for a sufficiently long time to understand what the root problem here is.

The issue the post is talking about is that IndexedDB is slow, to the rate of inserting hundreds of documents takes multiple seconds. When you go a bit deeper into the post, you realize that the trouble is not with the number of documents that are being saved, but the number of transactions.

IndexedDB has the concept of transactions, and I’m going to err on the safe side and assume that it has durability. I dug a bit and found that IndexedDB is based on LevelDB (and I do know that one quite a bit). The underlying issue is that each transaction commit needs to issue an fsync(), and fsync is slow!

Pretty much all database engines have to deal with that, the usual answer is to allow transaction merging of some sort. Many relational databases will write to the log and commit multiple transactions in a single fsync(), RavenDB will do explicit transaction merging in order to batch writes to disk.

However, for pretty much every database out there with durability guarantees, the worst thing you can do is write a lot of data one transaction at a time. Server databases still benefit from being able to amortize multiple operations across different requests, but a client side database is effectively forced to do serial work with no chance for optimization.

From the client perspective, the only viable option here is to batch writes to get better performance. Then again, if you are working on the client side, you are either responding to a user action (in which case you have plenty of time to commit) or running in the background, which means that you have the ability to batch.

More posts in "re" series:

  1. (16 Aug 2022) How Discord supercharges network disks for extreme low latency
  2. (02 Jun 2022) BonsaiDb performance update
  3. (14 Jan 2022) Are You Sure You Want to Use MMAP in Your Database Management System?
  4. (09 Dec 2021) Why IndexedDB is slow and what to use instead
  5. (23 Jun 2021) The performance regression odyssey
  6. (27 Oct 2020) Investigating query performance issue in RavenDB
  7. (27 Dec 2019) Writing a very fast cache service with millions of entries
  8. (26 Dec 2019) Why databases use ordered indexes but programming uses hash tables
  9. (12 Nov 2019) Document-Level Optimistic Concurrency in MongoDB
  10. (25 Oct 2019) RavenDB. Two years of pain and joy
  11. (19 Aug 2019) The Order of the JSON, AKA–irresponsible assumptions and blind spots
  12. (10 Oct 2017) Entity Framework Core performance tuning–Part III
  13. (09 Oct 2017) Different I/O Access Methods for Linux
  14. (06 Oct 2017) Entity Framework Core performance tuning–Part II
  15. (04 Oct 2017) Entity Framework Core performance tuning–part I
  16. (26 Apr 2017) Writing a Time Series Database from Scratch
  17. (28 Jul 2016) Why Uber Engineering Switched from Postgres to MySQL
  18. (15 Jun 2016) Why you can't be a good .NET developer
  19. (12 Nov 2013) Why You Should Never Use MongoDB
  20. (21 Aug 2013) How memory mapped files, filesystems and cloud storage works
  21. (15 Apr 2012) Kiip’s MongoDB’s experience
  22. (18 Oct 2010) Diverse.NET
  23. (10 Apr 2010) NoSQL, meh
  24. (30 Sep 2009) Are you smart enough to do without TDD
  25. (17 Aug 2008) MVC Storefront Part 19
  26. (24 Mar 2008) How to create fully encapsulated Domain Models
  27. (21 Feb 2008) Versioning Issues With Abstract Base Classes and Interfaces
  28. (18 Aug 2007) Saving to Blob
  29. (27 Jul 2007) SSIS - 15 Faults Rebuttal
  30. (29 May 2007) The OR/M Smackdown
  31. (06 Mar 2007) IoC and Average Programmers
  32. (19 Sep 2005) DLinq Mapping