–Dan Eichhorn, President and COO, PSEG Long Island
PSEG Long Island has been proudly serving Long Island for seven years, carving out a strong track record of improving what matters for our 1.1 million customers. During Tropical Storm Isaias, we did not provide the experience our customers have come to expect. We own that mistake, and have been working nonstop since Isaias to resolve all of the issues our customers experienced.
I want to be crystal clear about something: We remain steadfast in our commitment to serving Long Island as evidenced by the significant improvements that we have made in our systems. I want to share some of this progress with you.
Since Tropical Storm Isaias in August we have made several improvements, including removing bottlenecks and increasing capacity of parts of the telephony solutions that caused the problems customers experienced during that storm.
We’ve made multiple design changes in concert with our vendors and our third party subject matter advisors. And we have conducted a series of tests along the way to validate the fixes.
We have now completed seven major tests on the telephony system. Tests included injecting 5,000 concurrent calls into the system. We held those calls for approximately 2 minutes each for a period of time. This represents a possible total call volume of approximately 150,000 calls per hour which exceeded the calls attempted during Isaias’ peak calling hour by 38 calls.
Second, we tested the connectivity to our on-premise call center. This is important because in a storm, if a customer needs to speak to a representative to inform us of what is known as a “wire down scenario,” we need to be sure that their call can get to our call center. We tested 575 concurrent calls under that scenario. Again, the system functioned properly in this test and without any delays.
We agree with LIPA that an end-to-end test is appropriate to confirm the results of these tests, and we expect to complete that test as expeditiously as possible in the next few weeks.
Outage Management System
The Outage Management System is being addressed in two parallel paths.
First, after Isaias we reverted the OMS software to the pre-storm version 5.5. In late August and early September a 90% Customer Outage test over a 24-hour period modeled after Super Storm Sandy was run. The results yielded valuable information on the limitation of OMS v5.5.
Since September the following actions have been taken to improve the current production OMS system:
- On Jan. 24, 2021, we reran the 90% Customer Outage test. The results of this test were very positive. The core outage management system was stable throughout the test. The system remained usable throughout the entire test with transaction times slowing only during the peak periods of outages. The function of updating the outage map was able to scale to reflect 70% of the customer outages. This compared to only 12% in September’s test.
- As noted already, we also plan to conduct an “end-to-end” test of the system, including not only telephony and digital channels, but also OMS in order to confirm these results and that the systems will operate as expected.
- We have continuously improved our testing practices which will be incorporated into our annual testing plans.
Second, in addition to improving the current production version of OMS, we are following the recommendation of our OMS vendor to obtain new infrastructure for our OMS system so that we can re-implement the upgraded v6.7. Since the December board meeting the following activities have been completed or planned:
- Ordered and started to receive hardware to re-platform the system.
- We plan to follow lessons learned and only implement changes in production that have been thoroughly tested in the test environment.
- Pursuant to our project plan, we expect to be able to begin testing the upgraded software in February, and to place that new version into production in April.
- We intend to complete the testing and to make sure version 6.7 is re-implemented successfully and as expeditiously as possible.
We believe this “belt-and-suspenders” approach to the OMS – testing and improving version 5.5 while we upgrade to 6.7 on new hardware at the same time – is the best way to ensure that our OMS works properly during a major storm.
I cannot praise PSEG Long Island’s 2,500 employees enough for the tireless work they did during Isaias to continue restoration work despite the technological challenges. We have put significant effort into integrating the lessons we learned from their efforts, as well as input from other stakeholders, into an updated business continuity plan that addresses these scenarios.
We submitted this plan to LIPA and convened a team of more than 20 employees across the company to incorporate LIPA’s feedback. We functionally tested the revised plan this week and are making final edits and improvements and plan to resubmit it for LIPA’s review.
The progress I just shared does not encompass everything PSEG Long Island has been doing to improve things for our customers. There were a total of 85 recommendations in the LIPA 30 and 90 day reports. LIPA requested these plans to be worked in a priority based on three tiers. All Tier 1 and Tier 2 plans have been submitted to LIPA. Tier 3 plans are currently being worked and are due to LIPA on Feb. 5, 2021. We will review LIPA’s feedback on these plans and work with our consultants and staff to address it as expeditiously and robustly as possible.
PSEG Long Island will never stop working to put the customer at the heart of everything we do. It’s part of our culture to find new ways to improve the experience for the people we serve. It’s what we have done for seven years, and our stumbles during one storm have not shaken us.
We deeply value our relationship with LIPA and with the people of Long Island. We remain firmly committed to providing the best service for Long Island that can be provided, for many years into the future.