Data Indexer

Work Project

One of my favorite work projects so far. This application is used to get all the data from the database into a flatten json format and send it to elasticsearch. Once it is there it has to be updated when it changes in the database so we used database triggers to make an event and use sidekiq to reindex the objects in the background. So far in production sidekiq has reindexed over 225 million records! Running almost 1 million jobs a day. We have 25 million docs stored in elasticsearch over 70 indices.

Application Photos

Sample screen shots from the application. Click the photo title to expand or collapse the photos.

  • Jobs per day
    Data Indexer Jobs per day
  • Job count
    Data Indexer Job count