IndexedDB is one of HTML5 web API, IndexedDB allows us to create a large persistent data in browser (client-side).
I want to share what I have learned about how to use IndexedDB without using any third party library/wrapper because I want to learn the 'raw' IndexedDB using its native APIs. All the code in this page is a working demo tested in these browsers Google Chrome v73.0.3683.86, Mozilla Firefox v66.0.2 and Safari v12.1 (under macOS Mojave v10.14.4), my experience is short and some part of the code may need more optimization.
Be warned this article is intended only for beginner to IndexedDB who want to know how to use IndexedDB but got confused after reading many pages from IndexedDB documentations, for experienced developers who are facing some issue with IndexedDB may not find the answer here, this is me trying to share my experience with IndexedDB, so it is from-a-dummy-for-dummies kind of things, LOL.
Glossaries the meaning of some keywords often used in this articles.
- Transaction (IDBTransaction): every database operations (create/read/update/delete/etc.) require a transaction,
to get the transaction we call
mIDBTransaction = IDBDatabase.transaction([storeNames], transactionMode);
because each transaction require at least one store to operate, so store is like a working 'scope' for the transaction. - Store (IDBObjectStore): store object allows us to make operations such as create/read/update/delete/etc.,
to get a store object we need to get a transaction object such as:
// get single store fruitStore = IDBDatabase.transaction('fruit').objectStore('fruit'); // get multiple stores using 1 transaction, example: mIDBTransaction = IDBDatabase.transaction(['person', 'fruit']); personStore = mIDBTransaction.objectStore('person'); fruitStore = mIDBTransaction.objectStore('fruit');
* a store is like a 'table' in RDBMS (Relational Database Management System) - Primary key (KeyPath): it is a unique key value in a store, it is defined during store creation but it is optional (may not exist), if we didn't define keyPath during store creation then everytime we call IDBObjectStore.add() or IDBObjectStore.put() we must provide an index field as the Primary Key key-value, also remember in any database that 'Primary Key' field if defined then it is always unique (no duplicate nor null allowed).
- Secondary Key (IDBIndex): this is the 'Index' in a store, index is also optional, an index is like a subset of a store, it provides a way to get/search for very specific records based on this index, we may create index as many as we want in a store. In IndexedDB if Primary Key (keyPath) is not defined then at least a Secondary Key is required, otherwise can not add/update record.
- Record: data that we store into a store or read from a store, consider this as a 'row' in a 'table' in a RDBMS.
Reminder notes before see the code, to avoid lost-in-code.
- IndexedDB is an object/document oriented database like NoSQL (Not Only SQL), it is not a relational database (RDBMS) such as MySQL, Oracle DB or others.
- IndexedDB is schema-less, we dont have to pre create/define a table structure like in RDBMS (ie: column name, column datatype), there is no 'relation' between stores.
- IndexedDB does not use SQL (Structured Query Language) to add/search data,
it uses some methods (APIs) from IDBObjectStore, ie:
- add(): add new record
- clear(): delete all records
- count(): get total count of 'key/index matched' records
- createIndex(): create a 'Secondary Key'
- delete(): delete a single record
- deleteIndex(): delete a 'Secondary Key'
- get(): get one or more key/index matched records
- getKey(): get only key value from a matched record
- getAll(): get all records
- getAllKey(): get only key values from all matched records
- index(): open a created index, to get/search by this index
- openCursor(): get IDBCursorWithValue to iterate through the matched records 1 by 1
- openKeyCursor(): get IDBCursor to iterate through an object store with a key
- put(): update a record if key is existed or else insert a new record
- We can save data into IndexedDB as key-value pairs (object oriented) or as JSON document (document oriented)
- Each store need at least 1 key, either 'Primary Key' (KeyPath) or 'Secondary Key' (IDBIndex),
Primary Key may be created only at the time we create a store:
IDBDatabase.createObjectStore(storeName, [optional] {keyPath: object, autoIncrement: boolean})
KeyPath value could be any Javascript object (ie: boolean, number, date, object, array, regexp, undefined and null), including file or Blob. - IndexedDB can only stores and retrieves record(s) using a key, so we need to use either Primary Key or Secondary Key.
- IndexedDB APIs are mostly asynchronous, each API doesn't return value immediately but will send the result in the callback with 'DOM Event' object, in either 'onerror', 'onsuccess', 'oncomplete', etc. This makes IndexedDB operations as event-driven.
- Each transaction will have a callback with a DOM Event value and it has 'bubble' effect, if there was an error happend at 'request' (IDBRequest) then it will bubble to 'transaction' (IDBTransaction) then bubble to 'database' (IDBDatabase), so this error can caused the transaction closed or at worse the database connection may be closed, to prevent this undesire bubble effect then we can catch the error in IDBRequest.onerror then call event.preventDefault() to stop bubble.
Demo code only use callback function (no Promise) to handle event, the code is written with many console.log() to immediately see the output in browser console and also has many comments for learning purpose.
- Setup IndexedDB variables for cross browsers
// get proper IndexedDB object for many browsers window.indexedDB = window.indexedDB || window.webkitIndexedDB || window.mozIndexedDB || window.OIndexedDB || window.msIndexedDB; window.IDBKeyRange = window.IDBKeyRange || window.webkitIDBKeyRange || window.msIDBKeyRange;
- To avoid many code repetitions, I have made some common functions:
- Store (IDBObjectStore) handling, such as to createStore(), to getStore() and to deleteStore()
- Wrap IndexedDB CRUD (create/read/update/delete) operations : dbCommand()
- Common transaction callback to display the output for this demo : commonResultCallback()
function createStore(db, storeName, keyPath, keyOptions, deleteIfStoreExisted) { if(! db || ! (db instanceof IDBDatabase)) { console.log('db instance is not exist or invalid object, please open it'); return null; } if(! storeName) { console.log('store name is not defined.'); return null; } let mIDBObjectStore; // Create an objectStore for this database try { if(db.objectStoreNames.contains(storeName)) { if(deleteIfStoreExisted) { // delete it db.deleteObjectStore(storeName); } else { return null; // store already exist } } if(keyPath) { // has key if(keyOptions) { // has option mIDBObjectStore = db.createObjectStore(storeName, { keyPath: keyPath, keyOptions }); } else { // no key options, default autoIncrement = false mIDBObjectStore = db.createObjectStore(storeName, { keyPath: keyPath }); } } else { // no keyPath is provided mIDBObjectStore = db.createObjectStore(storeName); // IF we want to define a default key-generator then we can create a key named 'id' with auto increment, // so that if IDBObjectStore.add(data) or put(data) and data does not have 'id' then 'id' will be added automatically //mIDBObjectStore = db.createObjectStore(storeName, { keyPath: 'id', autoIncrement: true}); } } catch(e) { console.log('caught exception: ' + e.toString()); } return mIDBObjectStore; } function getTransaction(db, storeNames, transactionMode) { if(! db || ! (db instanceof IDBDatabase)) { console.log('db instance is not exist or invalid object, please open it'); return null; } if(! storeNames) { console.log('store name is not defined.'); return null; } // get a transaction to multiple 'stores', default operation is readonly var mIDBTransaction = db.transaction(storeNames, transactionMode ? transactionMode : 'readonly'); return mIDBTransaction; } function getStore(db, storeName, transactionMode, onTransactionCompleteCallback, onTransactionErrorCallback) { if(! db || ! (db instanceof IDBDatabase)) { console.log('db instance is not exist or invalid object, please open it'); return null; } if(! storeName) { console.log('store name is not defined.'); return null; } if(Array.isArray(storeName)) { console.log('getStore(..), parameter "storeName" should not be an array, use getTransaction() instead.'); return null; } // get a transaction to single 'store', default operation is readonly var mIDBTransaction = db.transaction(storeName, transactionMode ? transactionMode : 'readonly'); if(onTransactionCompleteCallback) { mIDBTransaction.oncomplete = onTransactionCompleteCallback; } if(onTransactionErrorCallback) { mIDBTransaction.onerror = onTransactionErrorCallback; } // get 'store' object var mIDBObjectStore = mIDBTransaction.objectStore(storeName); return mIDBObjectStore; } function deleteStore(db, storeName) { if(! db || ! (db instanceof IDBDatabase)) { console.log('db instance is not exist or invalid object, please open it'); return; } if(! storeName) { console.log('store name is not defined.'); return; } if(db.objectStoreNames.contains(storeName)) { // found it, so delete it db.deleteObjectStore(storeName); } } function dbCommand(objectStore, command, data, param) { if(!objectStore) { console.log('dbCommand(...), objectStore parameter is not exist or invalid'); return null; } var objectStoreRequest; if(command == 'add') { // WARNING: if keyPath is unique and same key value already existed then transaction.onerror will be called !! objectStoreRequest = objectStore.add(data, param); } else if(command == 'get') { objectStoreRequest = objectStore.get(data); } else if(command == 'getAll') { objectStoreRequest = objectStore.getAll(); } else if(command == 'put') { // NOTE: if key already existed then the record will be UPDATED, // else will INSERT NEW record objectStoreRequest = objectStore.put(data); } else if(command == 'delete') { objectStoreRequest = objectStore.delete(data); } else { // TODO: for simple DEMO purpose, we do not handle other commands console.log('dbCommand(...), command parameter is not handled'); return null; } // NOTE: for DEMO purpose, set default callback to display result in console objectStoreRequest.onsuccess = commonResultCallback; return objectStoreRequest; } // for this simple demo, this is the common callback just to view the 'result' function commonResultCallback(result) { if(typeof(result) == 'string') { console.log(result); } else if(result instanceof Event) { if(result.target instanceof IDBRequest) { let requestResult = result.target.result; if(requestResult) { if(Array.isArray(requestResult)) { let totalResult = requestResult.length; console.log('Request return with ' + totalResult + ' records'); requestResult.forEach(function(item, index) { console.log('record[' + index + '], key: ' + item.personId + ', content: ' + JSON.stringify(item)); }); } else { console.log('Request return a single record: ' + JSON.stringify(requestResult)); } } else { console.log('Request return without data or data not found.'); } } } }
- Open IndexedDB wrapper
function openDB(dbName, dbVersion, callback) { if (!window.indexedDB) { return 'Your browser does not support a stable version of IndexedDB.'; } // must have callback, because indexedDB use async operation if(! callback) { return 'can not openDB because there is no callback'; } // create/open database let mIDBOpenDBRequest = indexedDB.open(dbName, dbVersion); mIDBOpenDBRequest.onerror = function(event) { console.log('open database error, code: ' + event.target.error.code + ', message: ' + event.target.error.message); } mIDBOpenDBRequest.onupgradeneeded = function(event) { // Save the IDBDatabase interface mIDBDatabase = event.target.result; if(event instanceof IDBVersionChangeEvent) { console.log('need upgrade from old: ' + event.oldVersion + ' to new: ' + event.newVersion); // tell callback to handle upgrade callback(event); return; } } mIDBOpenDBRequest.onsuccess = function (event) { console.log('Success creating/accessing IndexedDB database'); mIDBOpenDBRequest.result.onerror = function (event) { console.log('Error creating/accessing IndexedDB database, code: ' + event.target.error.code + ', message: ' + event.target.error.message); }; callback(mIDBOpenDBRequest.result); // send back the 'IDBDatabase' object for next operation } }
We call the method testOpenDB() as below:// for simple demo, we created a global DB instance var mIDBDatabase; // open database function testOpenDB() { openDB('quickwork', 1, function(result) { if(result instanceof IDBDatabase) { console.log('openDB(), finished successfully.'); mIDBDatabase = result; } else if(result instanceof IDBVersionChangeEvent) { // upgrade needed, ONLY in this point we can create/delete store !! testCreateStores(); } }); } function testCreateStores() { // for this SIMPLE DEMO we will only create a store let mPersonObjectStore = createStore(mIDBDatabase, 'person', 'personId', true); // OPTIONAL, create another index for DEMO search by index (see below code) if(mPersonObjectStore) { // (optional) we may create 'secondary index' to search by mPersonObjectStore.createIndex('search_by_firstName', 'firstName', {unique: false}); } // create another store, without keyPath let mFruitStore = createStore(mIDBDatabase, 'fruit'); if(mFruitStore) { // because we do not provide 'Primary Key' (keyPath) for 'fruit', // so we must create at least an index as 'Primary Key' during add() and put() // an index also provides a way to search record by mFruitStore.createIndex('search_by_fruit_price', 'price', {unique:false}); } // at this point, if we want then we can delete any store //deleteStore(mIDBDatabase, 'fruit'); }
* If we called testOpenDB() then a database called 'quickwork' will be created along with 2 stores 'person' and 'fruit', please open browser developer browser and see the created IndexedDB, if database is not created then please refresh it but if you still can not see the database then maybe your browser does not support IndexedDB. - Create data
// dummy array data to insert var persons = [ {personId: 111, 'firstName': 'Hariyanto', 'lastName': 'Lim', 'dob': '19760810'} , { personId: 222, 'firstName': 'Mickey', 'lastName': 'Mouse', 'dob': '19281001'} , { personId: 333, 'firstName': 'Bing', 'lastName': 'Microsoft', 'dob': '20090601'} , { personId: 444, 'firstName': 'Jon', 'lastName': 'Doe', 'dob': '20190410'} , { personId: 555, 'firstName': 'Stan', 'lastName': 'Lee', 'dob': '19221228'} ]; var fruits = [ , { name: 'Apple', price: 19} , { name: 'Banana', price: 29} , { name: 'Cherry', price: 39} , { name: 'Dates', price: 49} , { name: 'Eggfruit', price: 59} , { name: 'Fig', price: 69} , { name: 'Grape', price: 79} ]; function testCreateData() { let personsLength = persons ? persons.length: 0; if(! personsLength) { return; } let fruitsLength = fruits ? fruits.length : 0; if(! fruitsLength) { return; } // get a transaction for multiple stores let mIDBTransaction = getTransaction(mIDBDatabase, ['person', 'fruit'], 'readwrite'); // define transaction callback for 'oncomplete' and 'onerror' mIDBTransaction.onsuccess = function(event) { console.log('transaction to add, complete: ' + event); }; mIDBTransaction.onerror = function(event) { // transaction error to add this data, maybe key must be unique and already existed if(event.target && event.target.error instanceof DOMException) { console.log('transaction error, code: ' + event.target.error.code + ', message: ' + event.target.error.message); } else { console.log('transaction error: ' + event.toString()); } }; let mPersonStore = mIDBTransaction.objectStore('person'); persons.forEach(function(item, index) { let mIDBRequest = dbCommand(mPersonStore, 'add', item); if(mIDBRequest) { mIDBRequest.onsuccess = function(event) { console.log('add succeed for key: ' + event.target.result); } mIDBRequest.onerror = function(event) { console.log('add failed, code: ' + event.target.error.code + ', message: ' + event.target.error.message); // prevent the transaction from aborting, so allow transaction to continue for other command event.preventDefault(); } } }); //let mFruitStore = getStore(mIDBDatabase, 'fruit', 'readwrite'); let mFruitStore = mIDBTransaction.objectStore('fruit'); fruits.forEach(function(item, index) { // NOTE: remember previously we did not define Primary Key ('keyPath') during 'fruit' store creation, // so we need to handle onerror to catch error if Primary Key is value is duplicated. // in this demo, we created an index call 'price', so we will use it as Primary Key and apply the value let mIDBRequest = dbCommand(mFruitStore, 'add', item, price = item.price); if(mIDBRequest) { mIDBRequest.onsuccess = function(event) { console.log('add succeed for key: ' + event.target.result); } mIDBRequest.onerror = function(event) { console.log('add failed, code: ' + event.target.error.code + ', message: ' + event.target.error.message); // prevent the transaction from aborting, so allow transaction to continue for other command event.preventDefault(); } } }); }
* To insert test data please open browser console then run testCreateData(), please see the created records in browser and don't forget to refresh each store to see new records. - Read data
function testReadData() { // get a store with default 'readonly' transaction mode let mIDBObjectStore = getStore(mIDBDatabase, 'person'); // read a single key dbCommand(mIDBObjectStore, 'get', 111); dbCommand(mIDBObjectStore, 'get', 222); // test get a non-existing key value '9090' --> no result dbCommand(mIDBObjectStore, 'get', 9090); // get all records dbCommand(mIDBObjectStore, 'getAll'); }
- Update data
function testUpdateData() { let mIDBObjectStore = getStore(mIDBDatabase, 'person', 'readwrite'); // using 'put' will UPDATE record if key has existed else will INSERT NEW record if key value is not existed. // update a single key let newUpdatedData = {personId: 111, 'firstName': 'Harry', 'lastName': 'MaxF', 'dob': '20010101'}; dbCommand(mIDBObjectStore, 'put', newUpdatedData); // insert a new data (because key 'personId:888' is not exist yet) newUpdatedData = {personId: 888, 'fullName': 'Barack Obama', 'dob': '29610804'}; dbCommand(mIDBObjectStore, 'put', newUpdatedData); // read all dbCommand(mIDBObjectStore, 'getAll'); }
* After run then please view the store and refresh it to see newer update. - Delete data
function testDeleteData() { let mIDBObjectStore = getStore(mIDBDatabase, 'person', 'readwrite'); // delete a single key let personIdToDelete = 111; dbCommand(mIDBObjectStore, 'delete', personIdToDelete); // read all dbCommand(mIDBObjectStore, 'getAll'); }
- Search data is limited to the 'key' (either 'Primary Key' or 'Secondary Key')
function testSearchPersonByPrimaryKey() { // get object store for default readonly let mIDBObjectStore = getStore(mIDBDatabase, 'person'); // define key range between 300 to 500 var keyRangeValue = IDBKeyRange.bound(300, 500); let mIDBRequest = mIDBObjectStore.openCursor(keyRangeValue); // get total records with this query mIDBObjectStore.count(keyRangeValue).onsuccess = function(event) { console.log('total record search by key: ' + event.target.result); } mIDBRequest.onsuccess = function(event) { let mIDBCursorWithValue = event.target.result; if(mIDBCursorWithValue) { let person = mIDBCursorWithValue.value; console.log('cursor value: ' + JSON.stringify(person)); // get next mIDBCursorWithValue.continue(); } } } function testSearchPersonByFirstNameIndex() { // get object store for default readonly let mIDBObjectStore = getStore(mIDBDatabase, 'person'); // search by index 'firstName' (previously created) let mIDBIndex = mIDBObjectStore.index('search_by_firstName'); // define key range 'firstName' first letter start from 'B' to 'L' var keyRangeValue = IDBKeyRange.bound('B', 'L', true, true); let mIDBRequest = mIDBIndex.openCursor(keyRangeValue); // get total records with this query mIDBIndex.count(keyRangeValue).onsuccess = function(event) { console.log('total record search by index: ' + event.target.result); } mIDBRequest.onsuccess = function(event) { let mIDBCursorWithValue = event.target.result; if(mIDBCursorWithValue) { let person = mIDBCursorWithValue.value; console.log('cursor value: ' + JSON.stringify(person)); // get next mIDBCursorWithValue.continue(); } } } function testSearchPersonWithCustomCriteriaAndDelete() { // get object store for read and write because we want to delete let mIDBObjectStore = getStore(mIDBDatabase, 'person', 'readwrite'); let mIDBRequest = mIDBObjectStore.openCursor(); let firstNameContain = 'an'; mIDBRequest.onsuccess = function(event) { let mIDBCursorWithValue = event.target.result; if(mIDBCursorWithValue) { let person = mIDBCursorWithValue.value; // search and avoid exception if(person && person.firstName && person.firstName.indexOf(firstNameContain) >= 0) { // found matching record // delete it mIDBCursorWithValue.delete(); console.log('found record and delete: ' + JSON.stringify(person)); } else { console.log('found record and keep: ' + JSON.stringify(person)); } // get next mIDBCursorWithValue.continue(); } } } function testSearchFruitByPriceIndex() { // get object store for default readonly let mIDBObjectStore = getStore(mIDBDatabase, 'fruit'); // search by index 'price' (previously created) let mIDBIndex = mIDBObjectStore.index('search_by_fruit_price'); // define key range var keyRangeValue = IDBKeyRange.bound(30,60); let mIDBRequest = mIDBIndex.openCursor(keyRangeValue); // get total records with this query mIDBIndex.count(keyRangeValue).onsuccess = function(event) { console.log('total record search by index: ' + event.target.result); } mIDBRequest.onsuccess = function(event) { let mIDBCursorWithValue = event.target.result; if(mIDBCursorWithValue) { let person = mIDBCursorWithValue.value; console.log('cursor value: ' + JSON.stringify(person)); // get next mIDBCursorWithValue.continue(); } } }
Facts finding
- All three browsers Google Chrome v73.0.3683.86, Firefox v66.0.2 and Safari v12.1 for macOS Mojave (10.14.4) do not have automatic refreshing the content of database/store, I found myself clicking the refresh button very often, I wish they provide a toggle to automatically refresh screen every time a transaction is finished successfully, this will definitely make developer life easier ;-)
- Safari v12.1 has some quirks in the beginning, need to remove all data after database creation and I experienced some data 'hangs' and refresh does not solve it, only with Safari close and re-open can display the content properly.
- Using indexedDB's 'raw' APIs directly is very inconvenience and a-callback-management hazard, beginner to IndexedDB maybe frustated :(
- Too many async APIs are troublesome, creating wrapper to provide callback or use Promise seem like a must for real project.
Fortunately there are some small IndexedDB wrappers (libraries) to provide better usability using Promise,
increase compatibility with many browsers, can automatically fallback to use LocalStorage
and also an easier way for a beginner to use IndexedDB, such as:
- localForage :: simple get/set
- JsStore :: query using JSON object schema
- Dexie :: simple wrapper using Promise
- There are some other wrappers for indexedDB ..
- Browsers compatibility, hopefully all browsers can implement the latest version or patch the bugs quickly especially MS Edge because many window users are using MS Edge, for more info see CanIUse IndexedDB
- IndexedDB does not work in 'private' (incognito) browsing mode, need additional error handling.
- IDBObjectStore's add() and put() maybe little confusing for beginner, add() is strictly for insert new record and will generate error if key existed, put() is to update if key existed but will insert new record if key not found.
- If we only want to use IndexedDB for simple storing a few data then maybe we can use LocalStorage instead of IndexedDB, but LocalStorage has limited capacity and if we need more capacity and do not want to use any library then this simple demo Javascript code is 'good enough' to provide CRUD operations.
- Is there a way to export/import (backup/restore) IndexedDB data? unfortunately, there is no function in any browser to accomplish these operations, so at this moment if you need to do it then the only thing you can do is to create simple Javascript just read all records and dump into CSV file or display to <textarea> (for copy-and-paste).
Summary
After testing with 'raw' IndexedDB APIs, I have no doubt to believe that IndexedDB is very important and much needed feature for browser's large persistent storage engine with data 'searching' ability, larger storage limit than LocalStorage is very welcome, the max size is depend on the browser, please see your browser specification.
Seeing my own code above, I feel it is not pretty, it may look like I put too many comments but I hope it may help other beginners like me. Hopefully IndexedDB 3.0 will have 'beginner friendly' APIs.
Use cases
- eCommerce companies (or other projects which require heavy data transmission between server and client) will definitely can take advantage to combine these APIs:
- IndexedDB: to store large data, including images, API JSON text and files.
- Service Worker: to run in the background to pre-fetch data for cache.
- Web Push: to silent sync/update data.
- Local (personal / SOHO) web app, because the initial data maybe big and need to retrieve/sync from local server.
- Offline application (server-less) personal todo-list, schedule, notepads, etc. which maybe only 1 HTML file (include all Javascript code) to be use with personal tablet.
Are you ready to create a project now?