It will call a custom import method described below. Then we start a new node process using the module fork. As soon as the upload is done, we send a http answer to the client so that he can continue using the app while the data is processed. In order to separate the file upload and the data processing, we are going to store the file in the filesystem and save in the database the state of the upload (a PENDING status). Start by creating an upload remote method in the Invoice model that will be called by your client app with a POST request. Let’s say we want to import invoices in our system and we already have an Invoice loopback model with two properties invoiceId and amount. I will also assume that you know the basics of the Loopback framework and coffeescript syntax. We will demonstrate the process with the database PostgreSQL, but transactions can used with different connectors in Loopback. The process allows the user to know which cells of his excel file have failed and rollbacks if an error raises.

In this article, we will explain how to build your own transactional csv import using the node framework loopback and a relational database. Lastly, in order to import large datasets without impacting the user experience, you need to separate the upload process (where the user is waiting for a few seconds for the file to be uploaded) from the import process which happens in the backend, maybe for a few minutes. A solution is to use SQL transactions to control when your changes to the database are eventually applied. You don’t want to write anything in the database if an error has risen in the process. The second challenge is to keep the database in a consistent state: if the user imports a file that creates an error at the line 50, the first 49 lines should not be written in the database.

If a user uploads a file and the import fails, he needs to be able to know why and how to fix his file in order to try to import it again.

The first challenge is to make a user friendly process. This article explains how to tackle the main challenges of csv import and provides a fully functional repo on github to get started quickly on a project! Often referred as ETL (Extract, Transform and Load), these functionalities are tricky to implement and often hard to understand for the end user. When you build an app where users need to regularly import data from an external source, you are bound to import csv files to populate your database.

Loopback download csv with header