Post Mortem Discussion
Check out architectural diagrams of this project
What was the problem to be solved?
JIRE purchased a subscription to PropertyBase, which is Salesforce's real estate plugin.
Unfortunately, the model of integration Saleforce had in mind (where the website would be generated from their end) turned out be unfeasible, so JIRE needed a way to integrate with its newly purchased technology.
What was the proposed solution?
Gunner proposed using the Salesforce API to push data from the Open Source CRM to Salesforce and maintain the existing platform.
This would give JIRE the flexibility it needed from a product standpoint while still feeding the data into PropertyBase, which would keep the CRM functionality JIRE desired in the first place.
What challenges arose during the project?
PropertyBase turned out to be more of a bolted on afterthought than a first-class citizen in the Salesforce world.
That meant, once JIRE was under contract, the Gunner team didn't get much support help and was left figuring out how to integrate with the Salesforce API.
"It was kind of like 'Well, it's your problem now,'" Gunner founder and CEO, Cody Swann, said. "The minute they heard we weren't doing things the exact way they wanted it done, was the minute we were on our own."
Testing was also a challenge because JIRE didn't have a test environment for PropertyBase and couldn't always tell if the data was correct.
"Unfortunately, it was kind of 'Fire and Fix,'" Swann said. "We just had to make sure that, if discrepancies were found, we could easily fix them."
What was the technical approach to the project?
Gunner needed a way to keep the data in the two platforms synced without help from Salesforce and without a testing environment.
From a technical perspective, this meant the team needed a way to be able to repeatably run large data syncs but also be able to do them in a way that they could be backed out since it was dealing with live data.
"Fortunately, Amazon Web Services has a number of technologies to make that a bit easier," Swann said. "We needed to create an initial process that we could run over and over and over again until the data on both sides was perfect."
To achieve this initial sync, Gunner had to get all the data in the Open Source PostgreSQL database that was hosted on Amazon RDS into Salesforce using the Salesforce Sales Cloud REST API and format the data so it would conform to the PropertyBase specifications.
"The real tough part was that Salesforce imposes a hard rate limit on their API, so we couldn't just move all the data at once," Swann said. "That's where Amazon SQS came in handy."
Instead of making the calls in real-time, each sync operation (one per listing) was placed into an SQS queue and retrieved via a Node.js powered AWS Lambda function in an interval that was spaced out enough to not hit the rate limit.
Once the data was in sync, it needed to stay in sync.
This meant that the brokers who used PropertyBase needed to see the latest data from the website database without having custom data they entered get deleted.
"In theory, this should be really easy," Swann said. "During the initial sync, we created an id schema that would tie the two systems together so when data was entered or edited on the Refinery side, it would update the corresponding data on the PropertyBase side. But, of course, it wasn't that easy because the Salesforce API fails quite often and you can't keep retrying or you'll hit their rate limit."
In practice, what the team did was created an API of their own which was fronted by Amazon API Gateway.
If a failure occurred, the Node.js function would place a marker in another SQS queue which would retry the operation at a later time that would not jeopardize the rate limit.
"We've worked with Salesforce before," Swann said. "But this was, by far, the biggest pain in the butt."
What was the project management approach to the project?
Once the team had a good grasp of the API, it began the usual approach:
1) One week function sprint where we added new functionality
2) One week QA sprint
3) One week bug fix sprint
Repeat until finished.
What platform was built for this project?
As always, Gunner created a separate AWS account for the client and manages it from the parent organization.
What did you learn from working on this project?
The team learned a lot about Salesforce and PropertyBase in particular, but there were other lessons that were reinforced along the way.
"First, when doing any time of data migration, you need to make sure you can run it over and over and over again without a lot of setup work - script it," Swann said. "Secondly, and any good developer knows this, don't assume anything is going to work and with Salesforce, plan on it not working."
How did this project benefit the client?
After the project's launch, JIRE was able to use a single source of data while still leveraging the benefits of PropetyBase.
This allowed the marketing team to enjoy all the benefits and familiarities of Refinery while giving the brokers the power of PropertyBase.
Most importantly, work was not duplicated as data was kept continuously in sync.
Why was Gunner selected for this project?
Gunner has worked extensively with the Salesforce API, so we were immediately familiar with how to get up and running, so we put together a demo for JIRE that illustrated our proficiency with Salesforce.
What tools, techniques and methodologies were used on this project?