Code Name: Searchy the Search Engine

The finance industry is heavily regulated and so firms have to stay on top of their employees to make sure they're not violating any rules, regulations or laws.

This has particularly become problematic in the digital age because employees have the ability to break these rules more easily than ever and often times, without even knowing it.

Locking down and monitoring digital infrastructure has long been the domain of the IT department.

However, these are often kludgy and required archaic searches for IP addresses and indicators in log files to see if improper sites were accessed or certain communications were made.

We proposed a front-end that would allow compliance directories to monitor employees and retroactively search for violations without having to contact the IT department and send them on a search for a needle in a haystack.

Additionally, we added the ability for administers to note whether suspicious activity was or was not a compliance issue so Searchy would get better at flagging potential issues.

The main challenge was identifying the right technology that could ingest data from hundreds of thousands of sources, parse them and store them in a searchable format for an indefinite amount of time in a scalable fashion.

Logstash is an Open Source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to a “stash.”

In our case, the stash was Elasticsearch on AWS.

The raw data was sent through AWS Kinesis where it was processed, parsed and indexed on Elasticsearch and also fed into a Machine Learning notebook within Amazon SageMaker.

We built the front-end in React which allowed the compliance directors to search through the massive database in a very Google-like fashion.

We utilized Agile Scrum with one-week iterations and weekly deployments prior to each IPM.

We only needed one Research Sprint at the beginning of the project and this was basically to figure out what all the data sources would be and how we would integrate with them.

From there, we proceeded with development sprints until the launch of the project.

We learned just how much data flows through a larger financial institution and how to create a scalable pipeline to tag that data and make it searchable.

After the completion of the project, compliance directors were able to do a large portion of the monitoring without involving company IT.

This included setting up proactive monitoring for words, names and destinations that would escalate attention to particular areas of the company.

Directors would also have a historical record as well. So if it came out that a certain site was problematic, directors could see who accessed that site going back to the inception of the project.

Gunner Technology heard about the compliance problem and proposed this solution.

We built a risk-free prototype to demonstrate how it would work, which was immediately hailed.