Hey Zcash community, Anthony from Jaxx here.
I wanted to take this time to apologize to the community for the recent downtown of Zcash services in Jaxx, provide an explanation of what happened, and let you know what we are doing to minimize service interruptions in the future.
This past Sunday afternoon (EST) we began experiencing connection issues with our Zcash node. Typically when any issues arise our automated processes are quickly able to fix the problem and correct itself. Unfortunately, this wasn’t the case this time and we had to manually investigate and work to get things back up and running. This was achieved Monday morning as soon as our team arrived at the office.
Although normal Zcash operations commenced, we were unable to determine the cause of the connection loss. Since then, we’ve been attempting to see why connections were lost and what we could possible do to ensure it wouldn’t happen again.
Fast forward to Tuesday evening. Our automated notification system alerted us again that connection between our node and the blockchain was lost. Unfortunately, this was after office hours for both our Toronto and Argentinean teams and I was unable to communicate with our back-end team leader in Argentina until this morning (Wednesday) to rectify the situation.
We hate being down, and strive for and are proud to say we achieve 99.9% up-time. That being said, I must apologize for the length of downtime of the two recent Zcash outages. I take full responsibility for not having a better plan in place to ensure that after the first issue we would have someone available 24/7 that could initiate the processes necessary to fix the connection issues if our automated service were unable to correct itself.
Since we are a small team, offering a free service, it isn’t possible for us to have systems monitored 24/7 when everything is working well. However, when there are known issues with unknown cause, we must do better to get things back online faster than we were able to do this week.
Next steps. This morning I have a team meeting with both our Argentina and Toronto teams to work out an emergency plan that will remain in place until we are confident that the issue behind the two recent outages has been resolved. We also further prioritized the work on determining why our automated systems aren’t responding to this particular connection loss issue.
I can’t promise that services won’t be interrupted moving forward, however, I can promise that should another issue arise, communication with the Zcash community will be more responsive, as will our speed at getting services up an running again.
Again, I’d like to apologize for our recent performance. We will be better.
Anthony Diiorio
CEO & Founder Jaxx