I am forwarding this, not to embarrass EBMUD or anyone else, but so that everyone can understand how a project got delayed for days because no one involved in the repair knew or was concerned about the scale of inconvenience it was causing to thousands of people.

I applaud Lesa McIntosh for steping up and doing the right thing, the hallmark of a responsible elected official.

From: Coate, Alexander [mailto:alexander.coate@ebmud.com]
Sent: Wednesday, June 28, 2017 4:37 PM
To: McIntosh, Lesa <lesa801@aol.com>; Butt, Tom <tom.butt@intres.com>
Cc: Kastama, Alison <alison.kastama@ebmud.com>; Sullivan, Sharla <sharla.sullivan@ebmud.com>; Zell, Eric <eric@zellandassociates.com>
Subject: RE: TOM BUTT E-FORUM: Our Tenuous Connection to Technology

Lesa and Tom – Thank you for respectively apologizing on behalf of EBMUD and expressing an understanding of how these circumstances can unfold.  Below is a summary of events related to the outage. Comcast contacted us and EBMUD responded, but in the communications Comcast did not share and EBMUD did not ask, the extent or severity of the outage. EBMUD proceeded to make the repair and was not aware that internet was out for parts of Richmond until I heard it from Lesa on Tuesday. Had we known, we could have worked through Monday night. Our practice is to ask PG&E and ATT if the outage is severe, though they usually let us know. Going forward we will ask Comcast. Again, I too apologize and thank you for your understanding. – Alex
-----------------
The break was on a 1962 12-inch steel main and was a blow-hole on the bottom of the pipe that damaged a Comcast cable located directly below our pipe in the vicinity of the blow hole. Here is the timeline of events.

• Monday, early morning
• Comcast notifies the District of water running in a storm drain at West Cutting and Wine. They observed the water during their investigation of a cable outage in the area and believed that it might be related to their outage. The District did not inquire about the outage or its magnitude. Comcast did not express a sense of urgency about the situation.
• District staff investigates the report of water. There was no water surfacing but their investigation identified a leak on our pipe.
• Monday, morning  
• District notifies USA and BNSF Railroad. It took several hours for both to respond before the excavation could begin.
• Excavation was mostly completed at 7 pm in part because there was 3 feet of asphalt to remove. Since no customers were out of water, the crew barricaded the excavation and left for the day.
• Pipeline was throttled to reduce the leakage
• Tuesday, morning
• Excavation, approximately 11 feet deep, continues and the pipe and the Comcast cable were located at 11 am. Damage to the cable was found. The crew foreman contacted Comcast following the exposure of their cable. Comcast employees were onsite during our crew activities and were in contact with the EBMUD crew.
• Tuesday, afternoon
• Excavation was shored and Comcast begins their repair. Cable repair was completed late in the evening.

Alexander R. Coate
General Manager
East Bay Municipal Utility District
(510) 287-0101 alexander.coate@ebmud.com

From: Butt, Tom [mailto:tom.butt@intres.com]
Sent: Wednesday, June 28, 2017 9:38 AM
To: McIntosh, Lesa
Cc: Coate, Alexander; Kastama, Alison; Sullivan, Sharla; Zell, Eric
Subject: RE: TOM BUTT E-FORUM: Our Tenuous Connection to Technology

Accepted, but I can tell you as mayor that I am also often the last to find out about anything critical in the City. I share your frustration.

From: lesa801@aol.com [mailto:lesa801@aol.com]
Sent: Wednesday, June 28, 2017 9:17 AM
To: Butt, Tom <tom.butt@intres.com>
Cc: alexander.coate@ebmud.com; alison.kastama@ebmud.com; sharla.sullivan@ebmud.com; Zell, Eric <eric@zellandassociates.com>
Subject: Re: TOM BUTT E-FORUM: Our Tenuous Connection to Technology

Tom,

I apologize for EBMUD.  This was a huge failure in communication and we are trying to get to the bottom of it.  I am a tenant in your building here in the Point and I knew we had no internet all day Monday but had no knowledge that it was an EBMUD issue.  In fact, I had no knowledge on Tuesday until Eric Zell contacted me while I was at my EBMUD board meeting.  To go even further, our GM had no knowledge of this matter until I told him on Tuesday.

Again my apologies to you and all those effected by this problem.  We will endeavor to do better.

-----Original Message-----
From: Butt, Tom <tom.butt@intres.com>
Sent: Wed, Jun 28, 2017 8:13 am
Subject: TOM BUTT E-FORUM: Our Tenuous Connection to Technology
Some time on Sunday, a large EBMUD water main under West Cutting Boulevard at Wine Street in Point Richmond burst, severing an adjacent conduit carrying the only  Comcast fiber optic cable that provides some 3,000 people and hundreds of businesses Internet, telephone and cable television service.

Initial estimates projected a repair by mid-day Monday. The protections changed more than a dozen times until Tuesday night when service was finally restored about 9:30 PM. Because I use a server at Interactive Resources, my architecture-engineering firm for email, I was unable to send or receive email for most of three days. The phones at Interactive Resources were also out during that period.

The facts, as I understand them, are that the water main break damaged the Comcast cable. EBMUD had to excavate the street, repair the water main, and shore the trench before Comcast could do anything. That process lasted until about 4:30 PM on Tuesday. Apparently BNSF had to also have a crew present because of proximity to a rail spur, and their slow response also resulted in some delay.

During this entire period, neither EBMUD nor Comcast made any effort to contact me, the mayor, to explain what was going on and update progress. As far as I know, they did not contact any other City officials or make any public announcements. Information about the cause of the problem and its repair was sketchy at best. Nextdoor and Facebook were the predominant sources, with fake news and alternative facts predominating.

The disruption, including costs for lost time and revenue, this caused to thousands of people and many businesses was considerable, and the lack of communication is unacceptable.

An incident like this is helpful for us to understand how tenuous and fragile is our connection to the technology we depend on. Communication companies have merged to the point that all of our services come from the same place. And a single cable serving 3,000 people with no redundancy? That seems like a recipe for disaster.

And the technology sector wants us all to go to the cloud, which puts us entirely at the mercy of Comcast or some other company for much of the work we do.

The City Council had a presentation from EBMUD last night that was mostly good news, like precipitation was 171% of average and reservoirs are nearly full. But we also heard that EBMUD maintains 4,200 miles of pipeline, that 35% of the system is cast iron pipes with an average age of 78 years (meaning much is a lot older) that result in75% of leaks. Another 30% is cement-asbestos that results in 16% of leaks with an average age of 47 years.

The next big thing we are told, is autonomous vehicles, tethered to the Internet via wireless transmission of GPS and other critical information sources. Can you imagine what would happen if you along with thousands of others including a bunch of semi-trailer trucks are riding own the freeway at the speed limit, and the system controlling everyone instantly goes down? Or someone seizes control of your car? Can’t happen? Well yesterday, the headlines blared, “A major cyber attack hit companies in Europe, the Middle East and the US on Tuesday, wreaking havoc for employees and customers alike.”

Welcome to the future.