MJ Freeway Stuck In Traffic, Yet Again, In Washington

Here are some observations from the Tuesday July 16th Washington State Liquor and Cannabis Board (WSLCB) Board Caucus.

My top 3 takeaways:

• Board Chair Jane Rushford briefly noted “there were some considerations” which were keeping Deputy Director Megan Duffy from providing an update on the traceability system re-launch; the agency and its vendor MJ Freeway took Leaf Data Systems offline later that afternoon and worked through the night to patch the system in uncertain and unsettling circumstances.
  • Following a planned two-day outage spanning Sunday and Monday, the agency announced the launch of release 1.37.5 of Washington State’s version of MJ Freeway’s Leaf Data Systems on Tuesday morning at 7am. Reports of unanticipated problems began surfacing shortly thereafter.
  • After the 10am PT Board Caucus, the agency distributed an announcement at 2pm PTconfirming “an error in Leaf Data Systems software release 1.37.5 that is causing a breakdown in the ability for licensees to do wholesale business transactions.” The system was taken offline as “MJF is currently isolating the code error that is causing the disassociation and preparing a fix.” To facilitate business-to-business (B2B) transactions, licensees were encouraged to use an online manifest form – the same form prepared for licensee usage during the contingency reporting period necessitated by MJ Freeway’s failure to hit their first deadline on November 1, 2017.
  • A planned 3:30pm PT “Post Go-Live Check-In Call” with the software integrators and interested parties was repurposed into a forum for deliberation about six primary issues encounteredwhose causes were not yet identified. The most problematic issue was a mysterious recurring disassociation of lab testing results from licensee inventory items. WSLCB Chief Information Officer Mary Mueller said roughly 4 million lab results were orphaned, although the underlying data was not lost. The agency was preparing for two options: successful identification and repair of the root cause of the issues, or rollback to the previous software release including erasure of traceability data collected by the state on Monday.
  • At 4:15pm PT, the agency convened with their vendor MJ Freeway to learn about progress towards identification of root causes. A hastily arranged call with the software integrators at 8pm PT revealed the agency was still at sea and the vendor still had no solutions to offer.
  • On Wednesday morning at 12am PT, the agency released a carefully worded description of two scenarios for bringing the traceability system back online. 3am PT was declared as the decision point for the agency to choose a path forward (or backward).
  • At 5:30am PT, the third-party software providers were informed that a mitigation had been achieved and tested to the agency’s satisfaction. At 6am PT, an announcement was broadcast by the WSLCB that the industry would be moving forward with a re-launch of a patched version of release 1.37.5 inclusive of Monday’s traceability data.
  • Leaf Data Systems version 1.37.5 has been particularly challenging for the vendor and the state to achieve. During the course of this latest incident, Cannabis Observer has witnessed calls for a class-action lawsuit against the state; concerns about reversion to a spreadsheet-based contingency reporting paradigm; and indications that legislators and the Office of the Governor will be asked to force the WSLCB to cancel the contract with MJ Freeway. At a minimum, the WSLCB will be negotiating and collecting liquidated damages from MJ Freeway, a stipulation established by former Deputy Director Pat Kohler when the vendor first failed to deliver release 1.37.5 in November 2018. Unfortunately, it’s worth noting that while the agency’s decision to move forward was likely a better path than attempting to shift an entire industry backwards in time – it remains to be seen how well the patch was executed and what other unanticipated issues may lurk in MJ Freeway’s closed, proprietary source code.