Network Technical Log

4/5/17: 15:30 Power restored and all links and repeaters are back to normal.

4/5/17:  Devilshead power still out.  IREA has found 3 trees across the power lines and is working on fixing.  They are working on it.

4/4/17:  Commercial power failure at Devilshead has shut down Devilshead and terminated microwave connectivity from Westcreek, Badger and Cripple Creek.  The backbone is routed on commercial internet restoring north/south service awaiting restoral.

3/26/17:  We are upgrading and moving the cBridge within our network. There will be an outage on Sunday Mar 26, from approximately Noon to 4pm while we make changes.  During this time all talk-groups will continue to operate within their individual IP Site Connect clusters, but not cross IP Site connect communication is possible.   An example, Rocky Mountain Wide will continue to operate within the region you are using (for example, South, Central and North) but will not be cross patched to the other regions.    Also repeaters with individual patching (like Devils Head, Genoa and West Creek) will operate as standalone repeaters until the cBridge is back on-line.

3/3/17:  New repeater in testing mode at Westcreek.  It’s not up at full strength, but it is usable in the Denver Metro area as well as Douglas County.

2/10/17:  Almagre back on-line.

2/10/17:  Almagre Mountain site is off-air due to power outage.  No Estimated Time of Repair at this time.

2/10/17:  Power restored at 1:15pm.  Damage due to wind on the power lines.

2/9/17:  Commercial power is off at Squaw Mountain.  We’ve disabled the repeaters at the site due to this condition.  Microwave is up and operating on battery.  We should be good for quite some time.  We hope that IREA gets the power restored.  It’s a snowcat trip.

1/15/17:  Canon City is back at the Fremont Peak site and operational.

12/10/16:  Canon City DMR is offline due to hardware issues.  The rest of the south network is functioning as normal.

9/26/16:  Almagre VHF Return to normal service.

9/23/16: Almagre VHF DMR has been moved temporarily to a new configuration to support the Tour De Cure this weekend. We will restore normal service after the event.

7/30/16:  Link restored from Cheyenne Mountain to Almagre.  Lightning and wind took a toll on the hardware.  We’ve replaced it with some more heavy duty mounting and beefed up the cabling.  We’ve also restored our redundant route from North to south and now have triple paths from Denver to Canon City.

6/16/16: Fremont restored to service.  Replaced DMR Repeater, all microwave power supplies and the router.  All surge suppressors on the ethernet were replaced too.

6/12/16: Fremont peak sustained a lightning strike and the DMR in Cañon City and Salida are off air til further notice.  It damaged 5 link radios and destroyed the DMR repeater which is the master for the South network.  This will take the South talkgroup off the air.

5/27/16: Akron restored.

5/26/16:  Akron repeater off the network.  We have not been able to confirm what’s up with the repeater.  We believe it might have been a lighting strike.

4/20/16:  Devilshead is restored to normal.

4/17/16:  Devilshead power is off as of 00:24.  Network is rerouted over VPN until restoral.  Location is inaccessible at this time of year.

4/10/16:  Cheyenne Mountain DMR-MARC is offline until summer when we move sites.

4/2/16:  Lookout Local restored to normal operation by Paul Deeth.  Repeater replaced.

3/27/16:  Lookout UHF DMR (Lookout Local) power supply died.  Waiting for snow to melt to get there to repair.

3/17/16:  Cheyenne Mountain temp internet link is back up.   Mike Palmer, K0HYT made a trip to the site and fixed the issue.  DMR-MARC machine has been restored to normal operation.

3/9/16:  Cheyenne Mountain Internet link is down.  This affects the UHF DMR-MARC machine’s connectivity.  We’re working on a solution.

1/27/16: Fort Morgan off air due to damaged DSL router. Unknown ETR

11/20/15:  Cheyenne Mountain is restored on VPN thanks to Cheyenne Mountain Repeater Group.  Permanent repair scheduled for next Spring (link replacement at Almagre.)

11/18/15: Cheyenne Mountain is Offline due to link failure.  Planning a restoral trip Friday morning to get the connection back up and restore service.  Believe it’s weather/wind related.

11/18/15:  Genoa showing down during very high winds.

11/16/15:  Fort Morgan IP is down.  Believe it to be a bad DSL Router.  Rob, W0FT is going to diagnose.

11/13/15:  Fort Morgan returned to the air after a long outage due to lightning damage last summer.  ALL gear had to be replaced due to vaporization of certain things like the receiver and power supply!  Thanks to Doug, K2AD and Rob W0FT.

11/4/15: Installation completed on the new Akron UHF repeater covering large portions of NE Colorado.  It’s up and online!

9/16/15:  Almagre site will be without power for the day for some power pole upgrades by the power company.  It will be restored by the end of the day.

8/12/15: Intermittent IP connectivity to Lookout has been restored due to bad radio hardware at Lookout.

7/20/15:  Jeff Carrier, K0JSC traveled to the Badger site and power cycled the repeater.  This solved the problem.

7/18/15:  Badger Mountain Fairplay is down.  Likely went to sleep and won’t wake up without a power cycle.  We’re going to try to get up there in the next week to cure the problem.

6/26/15:  Lookout Mountain microwave link was restored today.  All should be back including the D-STAR system.

6/17/15: Lookout Mountain microwave link is down and we’ve lost the TRBO link on VHF and UHF as well as the link to the D-STAR system. We will get it into the queue to figure out what’s wrong and resolve it.

5/19/15: Methodist Mountain MotoTRBO repeater is down due to flaky power systems.  It is under about 5′ of snow at the moment and will be restored some time this summer.  Power restored itself.

5/17/15: K0JSC made the trip to Fremont and repaired the 3 dead microwave radios and myriad of blown up surge suppressors.  The site is restored to service and the microwave system is back in working order.

5/13/15:  Lightning strike at Fremont Peak (Royal Gorge) has damaged the network and caused us to fail over to commerial internet and re-route traffic.  We’ve lost to microwave radios and two power injectors.  Spare parts are delivered and will be installed on Sunday 5/17.  All TRBO and Analog radios functioning as normal.

5/11/15:  Lightning stike at the Fort Morgan EOC has taken out the DSL and the repeater.  It will need to be completely replaced.  This equipment is in the process of being ordered.

4/30/15: Salida Methodist Mountain is back online after a multi-day power failure.  The site is flaky and should be considered unreliable.  Our plan is to replace at the first access.  This is a July 1 expected date for access.

4/18/15: Salida Methodist Mountain repeater is offline due to bad hardware.  Site is expected to be back online in July when driving access is available.

3/14/15: NV0N and N0HAP with the help of WA2YZT made a service call to Lookout Mountain to replace the W0CDS D-STAR gateway machine. It have previously been taken down by NV0N and K2AD with the help of WA2YZT after it’s disks failed. W0CDS is now back online. In addition, a superfluous router and power strip were removed.

3/7/15:  Squaw Mountain lost an APC Masterswitch and therefore the site was down until Willem AC0KQ and Zeke K0XK hiked into the site to replace the IP Power switch with a standard outlet strip until we can drive to the site.  It took them 1.5 hours to hike up to the site in 2′ of snow.  There was chest deep snow on the north side of the building by the site door.  The system was down from Friday morning to Saturday Midday.  It is now online.

2/11/15: After a power failure at Salida/Methodist Mountain, the TRBO repeater did not come back up.  The IP is working at the site, but the TRBO is down.  ETR is July 15, 2015.

12/27/14: Found a link in the system that was not performing up to par and causing some issues for the whole network.  Rerouted traffic to repair this and fixing packet loss issues and increasing the throughput on the network.

12/23/14:  Sandia Crest/NM has had a problem with power causing some damage to the power system on our rack.  The ABQ team is working to deploy a replacement power system that can stand up to the power hits.  Unknown ETR.

12/21/14:  New repeater was added in Cheyenne, Wyoming to augment Colorado Wide coverage into Wyoming.  Wyoming EOC is working to install their own network across the state of Wyoming and they have plans to remove the Cheyenne EOC machine from the Colorado network.

10/25/14: On Saturday 10/25/14 three teams were dispatched to Squaw Mtn, Devilshead Mtn and Badger Mountain to button up some microwave items for the winter and facilitate the installation of a new site in Fairplay (Badger Mountain.)

10/18/14: W0VG went to Almagre to investigate router outage.  Found a problem with the backup battery system which resulted in low voltage to the router and associated microwave radios.   K2AD went to Cheyenne Mountain (Colorado Springs)  to bring the new DMR-MARC repeater on-line.   This repeater now carries the DMR-MARC talk groups for Worldwide TG1, North America TG3, Mountain Regional TG3177 and Local TG2.

10/7/14: K0JSC and KB0RFC went to Almagre to move the 5Ghz dish to a point that was not impeded by the tower next door.  Dish was relocated and throughput went from 2mbps to 30mbps bi-directional and the packet loss dropped to nearly nothing.

9/22/14: K0JSC and W0VG went to Fremont and Almagre (respectively) to install a 5gig link between those sites and abandon the 3gig link for improved througput on 5gig.  It was determined that there is an old abandoned 12′ dish in the path attenuating the signal level.

9/15/14: K0JSC went back to Methodist to change the router, fix the PTZ Camera and check the VHF repeater.  The VHF repeater had a bad preamp, the camera needed to be reset to defaults after a firmware upgrade and the router was swapped as the 1st one was acting strange.  Firmware on the router was brought to ver 6.19 while on site.

9/14/14:  W1ATE and W1NDX at Fremont Peak to peak the 5 gig link on the fremont peak side.

9/14/14:  K7PFJ, W0VG, K0JSC and K0GUR went to Methodist Mountain in Salida to extend microwave from Fremont Peak at the Royal Gorge to the new MotoTRBO repeater.  At the same time that this was completed, we completed an upgrade/update of the Colorado Connection repeater at that location to provide digital back-haul back to the hub site at Thorodin Mountain.

8/27/14: Several RMHAM Members went to Cheyenne Mountain to deploy a new DMR-MARC Repeater.  Microwave link from Almagre Mountain to tie into the backbone.  As of the writing of this email there was still some technical glitches causing the system not to work. We also installed a new repeater and digital link for the Colorado Connection on 145.13 tied into the network via microwave.

7/27/14: W0VG & K0JSC established a wireless backbone link from Devil’s Head to Almagre. Previously traffic had to pass along a terrestrial internet connection.  K7PFJ & NV0N replaced a broken router at Horsetooth and while they were there they upgraded the DMR repeater to a newer model and installed a new 24volt battery backup system.

7/26/14: NV0N performed a system-wide router upgrade to bring all the remaining routers up to the latest supported firmware and software levels.

7/20/14:  Some programming issues with the Almagre repeater.  The repeater is functional in a local mode but is not linked at this time.

7/19/14: Major trip to upgrade and cleanup our Almagre Mountain site.   Upgraded the repeater equipment on UHF, and installed a new Mixed Mode repeater on VHF.   Upgrades to the microwave backbone between our Devils Head and Fremont Peak sites.   Also installed the first half of a “short hop” to Cheyenne Mountain so we can carry IP linking traffic for the Colorado Connection and Colorado Repeater Association.   Removed a lot of scrap metal and garbage from the site.  Many thanks to the crew that attended the two-day work party;  K0GUR, K0JSC, K0PWO, K0RM, W0KVA, W0VG, K2AD, WA2YZT, K3COT, K7PFJ and Aspen our canine mascot.

7/12/14: Trip to Squaw Mountain to repair the defective microwave radio pointed to Devils Head.

7/9/14:  Our Internet Service Provider at Thorodin Mountain has dropped our circuit.  Our backup circuit is not hooked in yet.  This affects any VPN Connectivity and Cheyenne and Fort Morgan.  We will be making a site trip within a couple of days to restore these.

5/22/14:  Devilshead went offline at about 22:00 last night 5/21 due to a lightning strike at Squaw Mountain.  It has been rerouted via VPN and is back online as of 9:00AM on 5/22.  All sites working as normal.  This will continue until drive access is available at Squaw….or July-ish.

12/29/2013: Trip by K7PFJ and K2AD to the Boulder site to upgrade the DMR repeater.  Removed existing XPR8300 repeater, and installed new Motorola MTR3000 repeater with backup batteries and charger.  Also converted all microwave IP link equipment to operate on battery power rather than 110 VAC.  

12/28/2013: Trip to Lookout Mountain by K2AD and WA2YZT and NV0N.  Fixed VHF repeater issue.  Added filters and pre-amp to UHF system.  Corrected IP microwave and router issue.  Also installed Colorado ARES W0CDS D-STAR repeater at site on a temporary basis.   System has temporary antennas and is “hitchhiking” on the RMHAM IP backbone for connectivity.  System will relocate to Devils Head at a date to be determined by ARES Leadership. 

12/7/13:  Router issue at Lookout Mountain.  Backhaul off-line.   Hope to have repaired shortly.

11/29/13:  Reports indicate the Denver VHF Lookout Mountain repeater on 145.3625 MHz has a transmitter issue.  We will investigate and repair as soon as possible.

11/7/13: W0VG tracked down the problem with the Almagre system and got it to sync back up with the help of Jeff, K0JSC adding an additional (temporary) repeater in Pueblo.  The systems are fully linked again and reporting working.

10/26/13: Trip to Thorodin by W0VG and K2AD.   Replaced old XPR8400 repeater with new MTR3000.   Re-configured receive antenna system to eliminate desense from Colorado Connection repeater.   Thorodin should be working at 100% now.  Have a hammy day.

10/25/13: IP backhaul failure to Almagre.  Repeater operational but in local mode only.  

10/18/13:  Trip to Almagre successful.  Backhaul restored and system on-line.  Thanks to W0VG and K0PWO.

10/15/13: Fixed packet loss issues on Pueblo causing severely choppy audio in and out.

10/14/13: Found wet cable at Fremont/Pueblo radio site which solved some of the packet loss and latency issues.

10/13/13: Fixed routing loop on Thorodin Mountain router causing Fort Morgan to be non-functional.  It’s now on the network and functioning.

10/11/13: Trip attempted to Almagre Mountain in Colo Spgs to repair internet issues.  Road access is impassible and system remains on Color Code 1 with no internet connectivity.  Attempts will be made before winter to access the site.

10/10/13: Experiencing choppy audio from packet loss between Denver and Canon City repeater.  Troubleshooting this internet connectivity issue.

10/7/13: Trip to Devils Head by W0VG.   Changed router settings, IP backhaul back on-line.

10/7/13:  Trip by W0VG to Thorodin.   Replaced router due to periodic lockups.

10/6/2013:  Repaired microwave link between Boulder site and “Firestone Hub” located at K2AD.   Restored backup internet based connectivity.  Installed MikroTIK router.   Corrected programming issue with repeater code plug.   Site is now the master repeater for the Northern IP Site Connect (consisting of the Boulder, Fort Collins and Fort Morgan repeaters).    Timeslot 1 on this network is a regional network (temporarily using ALLCALL), and Timeslot 2 is connected to the other repeaters on the network (temporarily using ALLCALL).    Look for the transition from Color Code 1 to Color Code 7 when we transition this network to talk groups late-Friday / early-Saturday as announced.   Thanks to W0VG for visiting the site, and to K2AD for supporting from the hub site.

10/5/2013: Devils Head router having issues creating a backhaul connectivity failure.  Repeater continues to operate in local mode on both TImeslot 1 and Timeslot 2.   We will try to have this fixed as soon as possible.

10/5/2013: Trip to Thorodin Mountain by W0VG and WA2YZT.   Installed 3GHz link between Squaw and Thorodin sites.

10/5/2013: Trip to Squaw Mountain by K2AD.   Installed battery backup on microwave radios.  Installed 3GHz link between Squaw and Thorodin sites.

9/28/13: Trip to Horsetooth Mountain with K7PFJ and K2AD.   Installed 3 GHz microwave equipment.  Repeater is now connected to the cBridge and other repeaters in the network.    Temporary connected Horsetooth Time Slot 2 to other repeaters using ALLCALL.

9/23/13:  Temporary link to support ARES activities (noted 9/18/13) has been removed.

9/21/13: Trip to Squaw Mountain with W0VG, WA2YZT, WN0EHE and K2AD.   Repaired microwave link from K2AD-Squaw (blown surge suppressor at Squaw site), installed MTR3000 repeater, and aligned all equipment.   Squaw is now at 100% capability and connected to “The Cluster” on both slot 1 and slot 2.   We should be able to re-configure the Denver regional repeaters soon.  We are working out the final details of the transition plan and hope to announce this information very soon.

9/18/13: Configured Canon City repeater to link on TS1 & TS2.

9/18/13:  NOTE!   In order to support ARES activities we have created a new mapping of the ALLCALL Talk Group.  The change from the 9/15 configuration is to add the Devils Head UHF system on TS2 to the new mapping.     The Lee Hill DMR-MARC Repeater, Time Slot 2 (normally “LOCAL” Talk group 2) is  mapped for ALLCALL to link this repeater to the Thorodin TS1 / Lookout UHF TS1 / Lookout VHF TS1 / Canon City UHF TS1 / Devils Head TS2 systems / Squaw UHF TS1.   Please give priority use of these repeaters and time slots to ARES Aux Comm communications.

9/18/13:  Devils Head repeater added to the “temporary cluster of repeaters.”    Sites now active on “the cluster” are:  Thorodin UHF (TS1 & TS2), Lookout UHF (TS1 only), Lookout VHF (TS1 & TS2), Devils Head UHF (TS1 & TS2), RMHR Trailer (TS1 & TS2), Canon City UHF (TS2 only).       The Boulder UHF, Almagre UHF, Fort Collins UHF sites continue to operate in a standalone mode (i.e. no links).

9/15/13:  NOTE!   In order to support ARES activities … The Lee Hill DMR-MARC Repeater, Time Slot 2 (normally “LOCAL” Talk group 2) has been mapped for ALLCALL and linked to Thorodin TS1 / Lookout UHF TS1 / Lookout VHF TS1.   Please limit usage of Lee Hill Time Slot 2 (DMR MARC LOCAL) to disaster response use only.

9/13/13:  Lee Hill DMR-MARC Repeater off-line due to site power failure.

9/8/13: Trip to Squaw Mountain by K2AD and WA1JHK.  We were working on repairing the 3GHz microwave radio … but then we heard the loudest BANG! we have ever heard.  Direct lightning strike to the building or tower.  Site AC power failure, as well as a building generator failure.   After power was restored, all repeaters are back on-air but the 3GHz microwave is down.  The Squaw TRBO repeater remains on-air but is in a “local only” mode.

9/8/13: Trip to Devils Head Mountain by W0VG, K0PWO and W0UAW.   Added router, installed microwave dish on tower to build link to Almagre.  Other general work on microwave system.

9/4/13:   The Canon City repeater is now linked to the Denver repeater.

9/4/13:  The Squaw Mountain IP data connection is experiencing difficulty and providing intermittent service.   Until it is repaired the repeater will operate as a “local repeater” with possible network operation when the link is working.   We hope to have it repaired shortly.