Data Indexer

Work Project

One of my favorite work projects. This application is used to get all the data from a legacy database and convert it to a flatten json format and send it to elasticsearch. Once it is there it has to be updated when it changes in the database so we used database triggers to make an event and use sidekiq to reindex the objects in the background. So far in production sidekiq has reindexed over 225 million records! Running almost 1 million jobs a day. We have 25 million docs stored in elasticsearch over 70 indices.
ElasticSearch
Sidekiq
PG Triggers
API
Clockwork
SaltStack
Ruby