Wireless Telecom Timelines 2023

Introduction:

Now, a simple timeline for your reference.

So, as I was putting together some regulation notes, I realized that there is no timeline to cover wireless telecom as a whole, or at least I could not find it. So, after going down this rabbit hole, I thought I would share with you what I have so far. This was work. Not everything is here and I skipped a few things.

Feel free to tell me if you have a better list. Most lists are just about one company or the history of a specific merger. Not all lists match, the dates are somewhat varied so I did the best I could to put it all here. Who can you trust?

I am not saying this is the best or most correct list, but based on what I could find, it’s the best I could do right now. That and this was driving me crazy for a week or so.

I hope you enjoy this but remember, it’s only a list with some comments in it. Really, not my final but something I wanted to share.

Yes, I did miss one or two things. I will continue to update. This was exhausting, but if you see a mistake or something I’ve missed, let me know, please.

1800s

  • 1844
    First telegraph message sent from Baltimore to Washington DC, thank you Samual Morse.
  • 1876
    Alexander Graham Bell makes the first telephone call. The first wireline call was from Boston to Cambridge, 2 miles. Then later that year a 6 mile call was made between Brantford and Paris, Ontario, Canada. 
  • 1885
    AT&T became a company, technically a monopoly but hey, we started to have nationwide communications. This was the Bell System.
  • 1899
    Brown Telephone Company was formed by Cleyson L Brown, in Abilene, Ks

1900s

  • 1900
    Brown installed the first long distance line to compete with Bell, being an independent telco.
  • 1911
    Brown merged with 3 other telcos to form the United Telephone Company.
  • 1925
    UTC became United Telephone & Electric, (UT&E).
  • 1034
    The Communications Act of 1934 was passed, This was supposed to “combine and organize federal regulation of telephone, telegraph, and radio communications. The Act created the Federal Communications Commission (FCC) to oversee and regulate these industries.”
  • 1939
    UT&E (and Brown) renamed itself to United Utilities. Brown had declared bankruptcy in the early 1930s and restructured by creating United Utilities.
  • 1946
    Clarke McRae Williams receives Oak Ridge Telephone Company as a wedding gift and then purchases Marion Telephone Company.
  • 1963
    Microwave Communications Inc, (MCI) was formed when John Goeken filed with the FCC to create a microwave private line system between Chicago and St Louis. However, the company MCI wasn’t actually founded until 1968. Goeken’s dream was big, to construct microwave towers between the two cities and all points in between to have a solid microwave path connecting voice calls between the two cities. He was trying to compete with AT&T. Goeken said he could do it cheaper and better. Which he actually did keep that promise. At that time, AT&T saw him as small, but still a viable threat to their business. Suddenly, they had a competitor which, in those days, was a bad thing for the incumbent. This is why it took the FCC until 1968 to allow MCI to build the network. William McGowen became the CEA and put money into MCI, and then things got moving.
  • 1968
    McGowen setup Microwave Communications of America, announced that the 11,000 mile system would go through 40 states, and had affiliates to support MCI.
  • 1969
    The FCC authorizes the microwave system that MCI proposed. AT&T accused MCI of only picking the profitable routes. I can imagine how angry AT&T was at the FCC when MCI got the approvals. The FCC’s decision was a 4 to 3 decision, not a sweep. Think about that for a minute. AT&T and Western Union fought the approval.
  • 1971
    MCI hired Kenneth Cox. Cox was a former FCC Commissioner and his job was to lobby the FCC to break apart AT&T’s monopoly. They were very brave to take on a behemoth like AT&T. This is when the FCC actually allowed MCI to move forward.
    Oak Ridge Telephone renamed to Century Telephone Enterprises, Inc.
  • 1972
    United Utilities became United Telecommunications.
    MCI went public to raise capital to fund the project.
    Southern Pacific needed to name its long distance company, so it picked Sprint which stood for “Southern Pacific Railroad Intelligent Network of Telecommunications”.
    Century Telephone acquired La Crosse Telephone Corporation in Wisconsin.
  • 1973
    MCI renamed and restructured itself, they were now MCI Communications Corporation, (it’s funny because technically it’s name is microwave communication incorporates communications company). The company was in serious financial trouble. The FCC really did want competition so MCI asked for and got permission to allow FX lines
    FYI – FX lines are foreign exchange service (FX) is really just voice line access outside the local telco’s area. Hence the name is foreign. But to be honest, it’s a voice line with numbers. Basically, a voice line connecting the FXO, foreign exchange office to the FXS, foreign exchange subscriber.
    McCaw Communications, a paging and broadcast company run by Craaig McCaw after it was nearly shut down, and started to build a large cable empire called McCaw Communications Companies, Inc.
  • 1974
    Antitrust suits against AT&T begin. MCI filed one against AT&T asking for damages and then the Department of Justice (DoJ) filed theirs against AT&T. Let the Bell break up begin!
    MCI was expanding, but not making money. If you run a business, making money helps keep you running the business.
    MCI was bleeding money from 1973 to 1975.
    McCaw gets into the common carrier business, offering paging to its customers.
  • 1975
    A division of South Pacific Company (SPC), South Pacific Communications Company, filed with the FCC to provide long distance to compete with AT&T using the lines they ran alongside South Pacific Railroad, another division of SPC.
  • 1977
    American Radio Telephone Service and Motorola start a Joint venture called Cellular One.
  • 1980
    United Telecommunications started to build a 23,000-mile fiber optic network for long distance communications.
  • 1981
    Craig McCaw, now CEO of McCaw Communications, read an AT&T document that cellular would have over 900,000 subscribers by the turn of the century.
  • 1982
    The US DoJ commands AT&T to break apart, set to be complete in 1982,
    General Telephone and Electronics, GTE, agrees to buy SPC’s long-distance service, Sprint. This deal was finalized in 1983.
  • 1983
    AT&T Corp breaks up the Regional Bell Operating Companies (RBOCs) expected to be 1) The Bell Telephone Company of Pennsylvania, 2) New Jersey Bell, 3) Diamond State Telephone, 4) C&P Telephone.
    Cellular One offers services in Baltimore and Washington DC.
    The FCC could not handle all the cellular license applications, so they went to a lottery system.
    McCaw files for cellular licenses and starts to build out its network. Craig McCaw had been buying licenses for the past several years.
  • 1984
    AT&T broken up and becomes what we called, at that time, the baby Bells officially split: 1) Ameritech – someone I don’t think I dealt with often, 2) Bell Atlantic – A very good Bell company, 3) BellSouth – OK to deal with, 4) NYNEX – a company I hated to deal with and in my opinion was terrible to work with, 5) Pacific Telesis – someone hard to deal with but fair, 6) Southwest Bell – I didn’t do much business with them, 7) US West – I think they were OK to work with,
    McCaw purchases Dominion Cellular.
  • 1985
    McCaw purchases MCI Airsignal, closing in 1986
    McCaw buys MaxCell Telecom Plus and Charisma Communications.
  • 1986
    GTE Sprint merged with United Telecom Company’s US Telecom. The new name? Sprint. Sprint stood for “Southern Pacific Railroad Intelligent Network of Telecommunications”. Sprint was in Carlisle, Pa, and they were wonderful to deal with.
    GTE spins off its Sprint division, I dealt with GTE in York, Pa, and they were very hard to work with,
    McCaw begins to sell off its paging to Mobile Communications and cable assets to Jack Kent Cooke.
    Cincinnati Bell buys 45% of Ameritech Cellular
  • 1987
    Fleet Call formed and later became Nextel.
    McCaw Cellular Communications purchased MCI Communications.
    Southwestern Bell Company acquired Cellular One which was the cellular division of Metromedia. At that time Metromedia was a paging and cellular company.
    United Telecom officially becomes Sprint
  • 1988
    Pacific Northwest Cellular was founded. They later renamed themselves as VoiceStream. At that time, voice was the future.
  • 1989
    United Telecommunications long distance service was profitable for the first time.
    Southwestern Bell Mobile Systems and McCaw Communications came together to form Cellular One Group.
  • 1990
    United Telecom completes its acquisition of Sprint.
    McCaw partners with AT&T as a technology partner.
    Thanks to McCaw, Cellular One started to use SS7 across the network for their cellular connections to the tower.
  • 1992
    Vanguard Cellular Systems joined Cellular One Group.
    AT&T purchases 33% of McCaw Cellular.
  • 1993
    Sprint entered the wireless market by purchasing Centel, then based in Chicago and the 10th largest cellular carrier operating across 22 states.
    Fleet Call becomes Nextel.
  • 1994
    AT&T Wireless acquired the rest of McCaw Cellular. They started using AT&T Wireless Group as their name.
    Pacific Telesis spins off its cellular company as Airtouch.
    John Stanton founded VoiceStream Wireless as a spin off of Western Wireless Corporation.
  • 1995
    Southwest Bell Cellular changes its official name to SBC.
    Cellular One opens up membership to affiliates. They served over 5 million customers and covered 69% of the US population.
    SNET, Southern New England Telephone, an AT&T affiliate, joined the Cellular One Partnership.
  • 1996
    Sprint partners with Radio Shack to sell phones
    Bill Clinton’s Telecom Reform Act was passed in 1996. This was supposed to be “An act to promote competition and reduce regulation in order to secure lower prices and higher quality services for American telecommunications consumers and encourage the rapid deployment of new telecommunications technologies.” PDF found here.
    Qwest is formed.
    Radio Shack declares bankruptcy closing 1,440 stores, later  sues Sprint claiming Sprint used the Radio Shack data to locate its own stores.
  • 1997
    SBC acquired Pacific Telesis.
    WorldCom, a long distance company, acquired MCI, another long distance company forming WorldCom MCI. Yes, long distance companies were huge back then and very profitable. I remember paying over 25 cents a minute was considered a good deal and that the long distance plans were so complicated.
    Bell Atlantic acquired NYNEX.
    Century Telephone purchases Pacific Telesis,
  • 1998
    Sprint builds its headquarters in Overland Park, Ks. A huge campus.
    Cincinnati Bell Wireless (CBW) is formed to serve southeast Indiana and Western Ohio. Cincinnati Bell buys 80% of AT&T Wireless’ Cincinnati Dayton PCS network.
  • 1999
    SBC acquired Ameritech.
    Airtouch and Vodafone UK merge to form Vodafone Airtouch. Creative name, right?
    VoiceStream spun off from Western Wireless Corp.

2000s

  • 2000
    Qwest takes over US West.
    Vodafone Airtouch combined with Bell Atlantic to create Verizon Wireless, Vodafone owned 45% and Bell Atlantic owned 55%.
    On the landline side, Bell Atlantic acquired GTE to create Verizon.  FYI Verizon is a combination of veritas (Latin for certainty or reliability) and horizon (visionary and forward-looking), taking the Ver and adding izon to form Verizon.
    VoiceStream Wireless acquired Omnipoint Communications and Aerial Communications Inc.
    Century sells wireless business to AllTel. 
  • 2001
    Cingular is formed when SBC combines with BellSouth.
    German Deutsche Telekom AG took over American companies VoiceStream and Powertel Inc.
    VoiceStream becomes T-Mobile US, an entity of T-Mobile International. A subsidiary of Deutsche Telekom AG.
  • 2002
    AT&T spins off AT&T Wireless Services.
    VoiceStream officially became T-Mobile (still owned by Deutsche Telekom in Germany).
  • 2003
    ClearWire is formed and offers WiMAX service
    WorldCom changes its name to MCI after accounting scandal
  • 2004
    Cingular takes over AT&T Wireless. Cingular was 40% owned by Bellsouth and 60% by SBC Communications, both Baby Bells.
    Qwest sells off its wireless division, keeping landlines, and becomes a reseller for Sprint.
    Sprint merges with Nextel forming Sprint Nextel. The goal was to increase Sprint’s subscriber base, but the networks had literally nothing in common and could not be easily merged. What really happened was that drove away customers and on top of the loss of subs, they took a $37B loss.
    Verizon launched Fios Internet in Keller, Tx, using fiber to the home, FTTH.
  • 2005
    Verizon takes over MCI.
    SBC acquired AT&T, keeping the AT&T name.
    Verizon acquired MCI.
    Verizon launched Fios TV in Keller, Tx.
  • 2006
    AT&T takes over BellSouth, renames Cingular Wireless to AT&T.
    Verizon loses its Telephone Directory Service.
    Cincinnati Bell bought out all of Cingular’s share of Cincinnati Bell Wireless.
  • 2007
    Verizon sells off landlines in Maine, New Hampshire, and Vermont to FairPoint Communications.
    Verizon sued and won against Vonage for patent infringement, Vonage appealed and lost, $120 million settlement.
    Verizon acquired CyberTrust, they provide information security services.
  • 2008
    Qwest drops Sprint and becomes reseller for Verizon Wireless
    Sprint Nextel forms Xohm, (a 4G WiMAX provider) and merges it with Clearwire taking a 54% ownership claim.
    Verizon takes over Rural Cellular Corp.
    T-Mobile acquired SunCom Wireless Holdings Inc.
    CenturyTel acquired Embarq and becomes CenturyLink. Embarq was the landline side of Sprint.
  • 2009
    Sprint’s Clearwire is renamed Clear. Xohm name is phased out.
    Verizon spun off more wirelines, Arizona, Idaho, Illinois, Indiana, Michigan, Nevada, North Carolina, Ohio, Oregon, South Carolina, Washington, West Virginia, and Wisconsin to Frontier Communications.
    Verizon completes Alltel purchaser gaining 13 million subscribers to pass AT&T subscriber count.
  • 2010
    Verizon opens up 4G service in several markets.
    While this is not USA, it is T-Mobile related, T-Home and T-Mobile German merged to form a new DT subsidiary to become Telekom Deutschland  Gmbh and was no longer T-Mobile. Then T-Mobile UK became part of France Telecom’s UK and Orange (UK) joint venture to name it EE and it became the UK’s largest mobile network operator.
  • 2011
    AT&T attempted to take over T-Mobile, but it failed, which made Sprint very happy since they were considering a merger with T-Mobile.
    CenturyLink buys Qwest formerly US West.
    Verizon bought Telemark Worldwide, an IT services company.
    Verizon takes over CloudSwitch.
  • 2012
    T-Mobile mergers with MetroPCS Communications which closed in 2013 and the new leadership, John Legere, will lead for the next decade or so.
    Verizon buys Hughes Telematics, entering the automobile market with wireless products and services.
    Verizon completes purchase of AWS spectrum from Comcast, Time Warner Cable and Bright House Networks.
  • 2013
    SoftBank invests in Sprint and takes 70% ownership.
    T-Mobile buys spectrum from US Cellular for $308 Million.
    Verizon buys upLink.
    Verizon buys EdgeCast.
    T-Mobile is now the “un-carrier”.
    TMUS goes public on the NYSE.
    Sprint and Softbank attempt to acquire a stake in T-Mobile US.
  • 2014
    T-Mobile buys 700MHz spectrum from Verizon Wireless for 2.365B.
    Verizon completes its buyback of Vodaphone’s 45% stake in Verizon.
    Verizon buys Intel Media.
  • 2015
    Sprint’s Clear is to be shuttered.
    Verizon acquired AOL, (what a disaster!).
    Verizon spun management of its tower division to American Tower Corp, ATC.
    AT&T purchases DirectTV and Time Warner.
    T-Mobile passes Sprint as the new #3 carrier in the US.
  • 2016
    SBC Communications buys AT&T and adopts the AT&T name.
    The United States Court of Appeals for the District of Columbia Circuit upheld the FCC’s net neutrality rules which made broadband access a public utility, no longer a luxury. The telecom industry appealed the decision.
    Verizon Communications buys XO Communications’ fiber business.
    Verizon takes over Social Radar.
    Verizon takes over Fleetmatics.
    Verizon takes over Telogis.
    Verizon takes over Sensity.
  • 2017
    Verizon takes over Yahoo.
    Verizon buys Straight Path Communications.
    Sprint and T-Mobile end merger talks.
  • 2018
    T-Mobile starts the takeover of Sprint, with the blessing of SoftBank and an all-stock deal.
    Verizon takes over Niddel.
  • 2019
    Verizon rebranded the media division as Verizon Media, which included AOL and Yahoo.
    DISH Wireless enters the mobile market as part of an agreement to allow T-Mobile and Sprint to merge.
  • 2020
    T-Mobile and Sprint merger completed, Sprint name removed and terminated.
    T-Mobile acquired Assurance Wireless as part of the Sprint merger.
    Qwest became Lumen Technologies.
    Verizon launched a search engine, OneSearch.
    Verizon acquired BlueJeans Video Conferencing Service.
    DISH Wireless officially purchases Boost Mobile.
    CenturyLink, Inc becomes Lumen Technologies, Inc.
  • 2021
    Verizon completes acquisition of TracFone from American Movil.
    Verizon wins C Band spectrum from FCC Auction.
    DISH Wireless signs an agreement with AT&T to become MVNO.
  • 2022
    DISH goes live on their network in Las Vegas.
  • 2023
    T-Mobile acquired Ka’ena Corporation who owns Mint Mobile and Ultra Mobile.
    DISH Wireless officially achieves 70% per FCC requirements for 5G coverage as required in the merger.

Links:

A Short History of RAN

I think a lot of people see new technology and wonder why the carriers don’t adopt it right away. I get asked that all the time. The reality is it takes time to bring something to market. It has to be tested so that the bugs can be worked out and then gain system acceptance before it can be deployed on a mass scale. This takes time, sometimes a year. 

That is why carriers rely heavily on larger OEMs. It’s because they will work on something until it works as close to what the carrier wants as possible. They charge more because they stick with it until release. Do they always succeed? Of course not, but that’s why the win rate is high! 

It’s not like the RAN hasn’t evolved in the past 20 years. Think about what the systems were like 20 years ago. Also, think about which systems out there still operate the same, or very close, to how it was 20 years ago. When you think back, it’s fun to see how much evolution came about.

This all started when someone asked me when Open RAN would be ready for carriers. I responded by saying, more or less, that it depends on what the expectation is. Rakuten says it works but for mostly data. For the carriers in the US, they want to be sure it’s going to work. That takes time to get through the process. I’ll talk more about that below. 

The other thing to think about is how public safety radio networks work today. It’s not a far cry from what we had 20 years ago. They rely on maybe P25 or TETRA networks to communicate. The reason for this is because they work and they’re reliable. Granted you have FirstNet, but that is for data and it rides on the AT&T network and dedicated priority spectrum.  I would bet municipalities that use FirstNet have a priority voice system, like TETRA or P25, that they rely on in emergencies.

First, some clarifications: 

So let’s go down the RAN evolution rabbit hole shall we? If you don’t want to, then jump down to the Open RAN section. 

First, we have to deal with the acronyms. Here is a quick synopsis:

  • BBU is a BaseBand unit, the interface between the core/backhaul and the radios in the RAN.
  • CRAN means Concentrated RAN, which is not C-RAN (Cloud RAN).
  • C-RAN is Cloud RAN and that means that the RAN will be controlled by BBU functions in the Cloud. That means the Cloud will be running the BBU application.
  • DRAN is a Distributed RAN, see below.
  • MIMO is Multiple In, Multiple Out, referring to an integrated radio & antenna that can transmit and receive simultaneously for network and throughput improvements.
  • Open RAN is a model where the RAN can be any radio controlled by a remote source.
  • PIM means Passive InterModulation.
  • RAN is the Radio Access Network which means the radios and antennas out at the sites.
  • UE is User Equipment.
  • vRAN is Virtualized RAN.

As you read on, remember I am talking about the RAN, not devices (UEs) or cores or backhauls. I am trying to make that clear because the first thing everyone will say is, “Do you remember the first iPhone?”, and I do. It was on AT&T’s system exclusively and it helped speed up the 3G data (1xRTT & 1xEV-DO) revolution in their RAN. I want to focus on the RAN which Most people tend to overlook unless there is a mass deployment happening.

Early RAN Evolution

Most people forget at one time there were just radios. I mean, think of public safety and paging. We just attached radios to controllers. In the early days, we had 2-way full duplex radios that were a stepping stone. A bit limiting, but they worked. Push-to-Talk (PTT) was a thing and we all knew how to use it.

Then we had patches, devices that would “patch” you from the radio to a single telephone line. The early ones I worked on required a Dual Tone Multi-Frequency (DTMF) code to get to the phone line. I had a keypad on my mic that I would have to punch in to gain access. 

So on those systems, no privacy unless you put encoders and decoders on your radios. Most companies didn’t pay for that feature back then. 

Then we could pass data, but using modems. It wasn’t magic, we had a receiver or modem, an interface, a controller, and a radio. Evolution baby.

Then, we had trunked radio, which took privacy and connections to the next level. You would talk over a wide area radio network, across the state and eventually the country. Think Nextel. When Motorola built the Nextel network, it combined the idea of PTT on trunked-radios and cellular networks at the time, that was pretty cool! Push the button, and talk to another Nextel subscriber like you’re on a walkie-talkie even though they’re in the next state. Very nice. Multiple users in one band on multiple carriers.

Cellular took us past trunked radios by offering a phone in your car, then your bag, and eventually in your pocket. 

At the site, it connected to a T1 line which carried 24 voice lines. At that time, voice was all that mattered. 

The radios had to be able to do certain things, key up and key down. (Key up means it would start transmitting usually at full power, key down means it stops transmitting.)

This was used back when we had the older technologies which most people forgot about. 

Pre-Cellular RAN

  • MTS – Mobile Telephone Service – 2-way radio, yes, this was the early mobile phones. We would key up a radio and enter a code using DTMF to access a phone line to make calls. Anyone could listen to your calls if they had a radio. If you wanted to make a call, you waited until the person on the line was done or you would listen to another channel to see if it was open. Each radio had a site, there was no handoff. 
  • IMTS – Improved Mobile Telephone Service – This was MTS, only better because the device would transmit and receive the entire time, no more PTT (Push to Talk). Crazy advanced, right? OK, there was a little more to it, but you get the idea. 

Then, as man advanced, we left the Stone Age and entered the Cellular Age!

  • 2G – second generation – This was when we entered a new age of communication. Unfortunately, we had so many technologies back then it’s hard to list all of them, (sorry if I missed yours). This was the entry of smart connections and using GSM, CDMA, and several other technologies that have been lost to time. Please note that the radios were on the ground so we ran Coax up the tower to connect to the antennas. Slowly, the idea of sectors was starting to concentrate the radio signal in one specific direction. Building the site, you would have 1 crew that did almost everything, from the tower top down to the commissioning and integration of the base station.
  • 3G – third generation – This is when phones and networks got more secure and smarter. CDMA and GSM technologies dominated but there were a few others. Most of the other technologies faded away. Spectrum was still very limited and almost all systems were FDD. TDD really wasn’t a thing back then in cellular. Slowly the FCC released higher spectrum, in the Gigahertz (GHz). This meant that Coax loss was an issue. As a reminder, the higher the spectrum, the higher the loss through coaxial cable. We used larger coax cables, and lower loss, which caused a lot of issues from cost to space to tower loading. Then, against conventional wisdom at the time, OEMs started putting radios on the tower close to the antennas to reduce RF loss through cables. Those were transitional times that rocked the industry, but few outside of the industry cared.

OK, let’s get back to the RAN and how it’s evolved past being a radio at the base of the tower. 

We’ve evolved past 3G to 3.5G and 3.75G. Sound familiar? Thinking of 4.5G and 5.5G and 5G advanced here.) Now we’re entering 4G which leads to 5G.

For those of you that remember a lot of other technologies, I get it. I purposely didn’t cover them all because I could drone on forever about past technologies. This is a lot to take in. 

Enter the Age of DRAN 

OK, let’s talk about distributed RAN, DRAN. We pronounce it Dee ran. 

Why is it distributed? Well, in the 3G days, we started to split the radio from the controller. The controller evolved to become the BBU, (which back then was a controller for either GSM or CDMA radios). That means that the radio had to be smarter and needed some type of interface to run up the tower. In the beginning, some OEMs used an intermediate frequency (IF) so you could use cheaper and smaller coax up the tower. This was soon replaced with data lines using copper or fiber. Copper had issues because of all the RF at the tower. So fiber was run up the tower for data and copper for power. Noise was still an issue, but the OEMs worked through that. 

We have the BBU at the base of the tower and we run fiber and copper up the tower to the radios. Then we have shorter coax cables, normally 6 to 20 feet, from the radio to the antennas unless the radios are integrated. Integrating the radio and antenna eliminates the need for coax. That is how massive MIMO came about.

This wasn’t just for cellular. We also did this for microwave, which was a major rethink of how to design microwave backhaul. 

Why is it such a big deal in the history of RAN? Because in the old days, we had so much loss from very long coax runs. We needed huge radios to pump out the power we wanted. Suddenly, we were able to lower the power of the radios because coax runs went from 100 to 300 feet to under 20 feet. The radios would be outside, so sir conditioning savings were seen in the beginning. We thought we would save money by running less coax. We thought the coax was expensive and the labor was tedious. Little did we know that hybrid fiber/power cables would be almost as much. We did save space with the hybrid cable but adding the radios to the tower caused us to do loading studies everywhere and we had to beef up the older towers to support radios at the top. We solved one problem just to create another. Not to mention anytime a radio failed we needed a tower climber to replace it.  

Pros:

  • Low RF loss through the coax.
  • Higher spectrum is now an option. Now we can go into mmWave without issue. New radios can house multiple RF bands and technologies without running new COAX anywhere. 
  • Less RF power means less utility power usage at the site and hopefully lower utility bills.
  • Shorter coax meant, at the time, way fewer dollars spent on very expensive COAX cable. However, hybrid cables, (copper and fiber), became the thing and they are expensive.  
  • Less copper at the sites meant less theft. Coax is full of copper.
  • Lighter radios, cheaper shipping, more efficient radios. 
  • Heat generation was moved to outside of the cabinet or shelter.
  • Better data quality with shorter coax runs.
  • More space in the shelter without the radios.
  • Fewer and smaller openings between indoor racks through the wall/cabinet panel up to the antennas. We also used less space in the cable ladders running up the towers. At the time this looked great, but we found a way to mess that up with huge hybrid cables. 
  • This model led to the MIMO technology and active antennas we commonly use today!

Cons:

  • We moved the weight of the radios up to the tower top. That meant we had to beef up all the existing towers and re-engineer the towers to support the weight equivalent of hanging a VW BUG on top of the tower. This added delays and huge costs to the projects and spurred a lot of new towers to be built. Some towers were too expensive to upgrade. Today, we need to do a structural analysis on every tower and understand exactly what will be added to the tower, so planning ahead is critical, and over estimating the weight by predicting what you plan to add over the next 3 to 5 years helps you in the long run.
  • Tower leases became more complicated with the tower companies counting each antenna, radio, jumper, and cable run to the lease limiting carriers, and charging for each item, wherever they can.
  • We had to run power from the base of the tower to the top, so now we had to calculate the required copper to carry such power over 100 feet in many cases.
  • When we started this, PIM was a huge factor in copper. PIM was a major thing, and running copper up to power a radio created new problems. So we had to overcome that with filtering and creative power supplies. PIM is still an issue, but not nearly as much. Today we need tower climbers to troubleshoot PIM issues, but we always needed climbers to troubleshoot line issues. Pre-tested PIM coax cables became a requirement on non-integrated antennas.
  • Fiber skills were not common among tower workers when 4G deployment started. Early on this meant we needed tower workers to learn to splice fiber. (No longer an issue because now all tower fiber jumpers are connectorized). 
  • Any radio troubleshooting now requires a climbing crew to replace faulty radios. Today, radios are very reliable but early on they failed often. Lightning strikes were a nightmare. We needed an entire team to repair lightning damage at the time. Today, not so much, the OEMs have done an outstanding job overcoming all of these issues. There may still be issues, but the percentages are very low compared to 10 years ago.
  • New cabling solutions had to be implemented.

OK, I think you get the idea. To get here we went through a lot of pain, but it was worth it.

These breakthroughs in design led to Massive MIMO in integrated antennas. That is leading to less equipment on the tower and improved coverage everywhere.

Not only this but as the spectrum got higher and data became the standard (instead of voice), we made each cell smaller.

At the time, we looked at smaller cell sites and more money, because you had to build more towers and put radios on more building tops, Bad, right?

No, again, it was counterintuitive. Why? Because loading was becoming an issue at that time sites would get overloaded a lot. We had to think of how to improve this without waiting on the OEMs to get their heads out of their asses.

So, the way I remember it was that Verizon came up with the idea of making cells smaller, more sites in a given area. At the time, this was counterintuitive because sites were expensive. Today it makes sense. They were building new sites everywhere in every market. They wanted a small cell, which Alcatel-Lucent developed. On paper, it looked great.

The reality is that what Verizon really wanted was a macro site without the BBU.

So this paved the way for CRAN and the BBU Hotel.

The Rise of CRAN and the BBU Hotels

OK, CRAN in this context is Concentrated RAN. What is this? Well, this is another evolution story that we owe to DAS systems. (DAS = Distributed Antenna Systems.) 

Since we started putting radios with antennas, whether with a radio and antenna or an integrated antenna, it changed DAS systems going forward, at least for carrier bands. The BBUs could be concentrated in one room, hence the name BBU Hotel, and run fiber all over a venue instead of coax. Remember this was a savings in the beginning and for DAS systems it saved a ton of money. Remember that inside most buildings, Coax had to be indoor fire-rated which was very expensive. 

So when the radios were collocated with the antennas, it made sense and installation of the cabling was way cheaper. Remember in venues and buildings you run thousands of feet of cable in one venue. Fiber is way cheaper than coax for material and labor. You can also run extra fiber strands for radio expansion.

However, it wasn’t that easy. DRAN runs were originally very short, mostly under 300 feet. Why does this matter? Because timing between the BBU and radio was a major issue in the beginning. We had to come up with a way to account for delays of milliseconds between the BBU and the radios. 

Most of you don’t remember but there was a lot of drama improving fronthaul. There were no standards. Then came CPRI, Common Public Radio Interface. This was created as a standard so that DAS vendors could use their own radios, this was back in the early 4G days. At that time they were trying to open up the RAN to new radio vendors but the BBUs were still matching the local carriers’ core, just assuming ALU/Nokia or Ericsson. At this time new cores, like Cisco, were just coming out to open up the core.

Don’t forget how the carriers have opened up the networks, core, backhaul, and fronthaul to avoid being married to one vendor. This allowed Cisco, Juniper, and other vendors to embed themselves into the carrier’s networks. If you see all the big names, remember it takes a long time, like over a year, to get tested and accepted for adoption in any major carrier. 

See my old fronthaul posts here and here. Morse on CPRI here and here and here

So, CPRI was opening up the fronthaul with standards that others could use. Then Nokia developed and adopted eCPRI, which evolved CPRI. This allowed a longer delay between the BBU and the radio. It also assumed the brains in the radio would complete some of the BBU functions. In other words, the radio had to be smarter.

Think about the ongoing radio evolution. It was no longer just a radio or upconverter, it was now a radio, a fiber interface, a data processor, and a CPU deciding on delays and data. It also became a remote live test set for the NOC. It got way smarter!

Oh, keep in mind, it is still point-to-point. It can only talk to specific radios. Each card in the BBU has a specific interface for each radio. Today they can do multiple technologies, 3G & 4G & 5G, but this is a recent development. They also can handle many carriers on any given radio.

You get the idea, the BBU and radio can do a lot.

But when connecting from the BBU to the radio, it normally fibers from the BBU to the radios. One BBU can feed many radios and some radios will allow you to daisy chain to the next radio depending on the configurations. Each site is its own system. Each site has its own RF design. Each site has to be designed for its spectrum, radio density, and about a hundred other factors. Again, it’s complicated.

So, back to fronthaul, it wasn’t a quick journey to make this work. There had to be standards. There had to be engineering updates and improvements. There had to be evolution to get us to where we are today. Remember, Rome wasn’t built in a day.

CRAN to C-RAN and Open RAN

C-RAN in this article means Cloud RAN. Open RAN means Open RAN, I wanted to use ORAN but there is so much confusion because the O-RAN Alliance took the name before industry adoption. 

So we figured out how to separate the BBU and the radios by a few miles. That is a step in the right direction, right? 

If any of you remember your 5G drawings you will remember that we had the CU (Centralized Unit) and the DU (Distribution Unit) and the RU (Radio Unit). 

  • CU is basically the Core, the way I see it today.
  • DU is the BBU or a version thereof.
  • RU is the radio portion of the RAN.

Remember the goal of 5G was to make the network an IP network end-to-end as well as settling on one RF standard worldwide. The reality of 5G was that it became a network transformation for the carriers. 

No more CDMA or GSM, it’s all one standard now. It is more like a network and less like a proprietary system. 

Believe it or not, it has opened up to more vendors than ever. In the days of old, it was so proprietary that one vendor had dominance. Those days are gone and more vendors will have a shot if they can endure the marathon of testing the carriers will put them through. However, running true Open RAN isn’t quite there on a large scale. It’s still more closed than Wi-Fi, which is your true open network. 

The plug-and-play (PnP) models are not quite there yet. I see them coming, but to be honest I thought they would be here by now. They’re not. We can do a lot remotely, but we can’t plug it in and wait. We have to upload a base config, add the local parameters, and then call the NOC to complete. It’s still a process that is less automated than expected. Again, it’s not as easy as Wi-Fi.

So to go from CRAN to C-RAN you have to move BBU functions into the cloud. The larger OEMs are working on this, but it’s been a struggle. The BBU has a lot of processing power and does way more than converge the data in and data out. It has a brain that is constantly processing all the data for that particular site. It also has to be aware of any neighbors so it doesn’t create problems with them or itself. It also has to process and route data the best it can. It’s running uphill all the time. Now it is being asked to be more energy efficient by figuring out when to turn down unused carriers. 

I’m exhausted thinking about it.

The next step is to have it connect to radios like the cloud would connect to your laptops. It’s a tall task and one that will require high-level security along with so many proprietary functions that 5G signal processing requires. 

To me, Cloud RAN has to happen before a true Open RAN system can happen. The fact that the Open RAN vendors made it work, you have to think about what apps they’re running. You also have to think beyond the data.

For instance:

  • Voice, like VoNR, will require that e911 works at every site, tested at every site,
  • Timing needs to be considered, probably using GPS timing at every site,
  • Broadband for mobile and FWA users will be demanding, can it handle the traffic?
  • Interference with neighbors will be an issue if not set up properly,
  • Carrier aggregation needs to happen, can the radio and BBU communication work using the Cloud?

While these seem complex, it’s because they are. 

I do think that Open RAN would work for a data-only system today using a smaller vendor’s radios. This would make sense. 

I think evolution requires C-RAN to work before we go to Open RAN on a carrier’s network.

Open RAN sounds wonderful, but in execution, there are so many details that need to be worked out. I think we had to get to 5G to make it work better. I think it works better in a digital use case-driven network, not in a network that is built to do many different things. 

Evolving to Open RAN, someday:

We have been talking about Open Ran a lot lately. I wanted to explain why it was a bit premature to the market for carriers. Mainly that version needed time to mature. One thing about a carrier’s network is they have to make sure the equipment works end to end without incident. 

It’s not that Open RAN won’t work, it’s that it was too early and hasn’t evolved yet. 

I’ve been in this business for some time and I realize that sometimes you’re ahead of technology. While the vision may be there, the dots can’t be connected. Approvals and testing take time, it’s not all about the vision. 

I talked to Open RAN vendors early on and their main argument was that they were going to make RAN super cheap. Here is a list of the promises as I understood them.

  • Use any OEM radio anywhere, it will be an open system,
  • Just connect to a backhaul and it will come up,
  • Use our software in the RAN and the Cloud and it will work,
  • Imagine putting any radio anywhere without any real commissioning or integration.

However, it didn’t play out like that for several reasons, which I will explain later in this article. First, I want to explain why things take so long in this industry to evolve. It’s not just about money, it’s about moving technology ahead with proven and trusted technologies. You have to make sure something works before you put millions of users on it. 

Like it or not, there is a reason why carriers trust larger OEMs. A large OEM does endless testing, updates, and upgrades to ensure a reliable and working system that is secure. They spend a long time, sometimes up to a year, to get approvals and acceptance. You must play the long game in this industry. The lifespan of the radio could go well beyond 10 years. 

As a side note, security and trust are huge factors. Huawei and ZTE lost trust by allowing employees (assuming the Chinese government people) to log into end users’ networks and access information. I wrote about it here. 

What is Open RAN, really?

The network is evolving as well. Open RAN is part of the evolution that some vendors have bet on. Open RAN is really opening up the backhaul and, (in theory), allowing any radio to be used on the network. Think about how you deploy Wi-Fi, it’s easy, cheap, and you can mix and match radios, sort of. It could be a game changer or it could lead to security issues. We have to do research and testing to be sure, carriers can’t just dive in without justification, they have to be sure.

Back in 2021, Jon Gold outlined it in his article, “What is Open RAN?” The way I read this was that Open RAN would set standards so that components of the RAN could be easily interconnected. What does this really mean? 

Juniper wrote a statement that says, “An Open RAN, or open radio access network, is made possible by a set of industry-wide standards that telecom suppliers can follow when producing related equipment. Open RAN enables programmable, intelligent, disaggregated, virtualized, and interoperable functions”.

Back in 2020, Chris Pearson wrote for 5G Americas and said, “ Open RAN involves the interoperability of open hardware, software and interfaces and cellular wireless networks. The Open RAN concept introduces advanced features and capabilities beyond the current standard centralized approaches to the RAN through leveraging a programmable open-software development approach.” 

To me, it means that the radios would just be radios and any BBU could connect to them and control them. It would mean that the system would open up and be easily connected like Wi-Fi hotspots. It would mean lower security and increased troubleshooting time. 

If you’ve ever done any carrier work you know nothing is that easy. These systems do a lot more than Wi-Fi. These systems are highly complicated telecommunications systems that people rely on for not only data and broadband, but also for voice, 911 calls, and safety. 

5G networks also have tons of KPIs that are collected every second. They have to worry about security, reliability, performance, and quality of service. 

It’s great that the standards would be set, but as you know, it never works out that well. If you worked on networks, you know Juniper and Cisco may or may not work together well without working out the bugs because each manufacturer has its own proprietary way of doing things.

But Open RAN is a stepping stone to Virtualized RAN. Think about it like a DAS system where you have a controller connected to several OEM’s systems. 

I also think back to when CPRI and eCPRI were setting standards. That did not go smoothly because not all OEMs bought in. Nokia was one of the first, but others were dragging their feet.

Open RAN also opens up a lot of security risks. 

Open RAN to vRAN

For this article, vRAN is Virtualized RAN. That is when the RAN can be configured for a specific use for a specific customer. I think this is cool, but I know very little about the vRAN models at this point. It seems like it would be ideal for private networks. If I can virtualize the RAN and make each area a use case-driven model which makes sense for private networks. 

One of the dreams of Open RAN was to run BBU functions on COTS (Common Off The Shelf) hardware. Why, because it’s cheaper than buying a BBU. The only issue is through lots of testing, so far, COTS just can do it. Not reliably anyway. 

So, vRAN was planning to virtualize even more functions of the RAN. There is a good comparison here, in this article written by Mohamed Abdel Monem where he explains the difference between Open RAN and vRAN. He explains that the big difference is that Open RAN would literally open things up whereas vRAN would still have proprietary interfaces.

I think vRAN is well on its way to becoming an industry standard,  but not until 6G is close. I think vRAN missed the 5G bus and now it has to wait until 6G takes off. 

Why was Open RAN too early for Large Carriers?

In my opinion, and I am spitballing here, most people think Open RAN will allow them to choose cheap radios. They thought Open RAN would be way cheaper and that they could use any radio on their network. I did too. 

Let me tell you what I learned through testing feedback I’ve gotten. Open RAN was more expensive than expected and lacked end-to-end ownership to resolve issues. When you deploy, finger-pointing is an issue, especially among smaller vendors that don’t have the manpower to resolve complicated issues. Large carriers are demanding. 

It turns out that wise sage, Nevile Ray from T-Mobile, said at a UBS conference according to Fierce Wireless, “We just have to be patient. The ecosystem in O-RAN is going to take some time to develop. You look at the radio business today and you have three major incumbents, right? One of them is Chinese. It’s a massive scale game and it’s a complex business… There’s massive R&D in this space to future-proof your networks. There’s a whole debate about who built and developed this stuff and who owns the IP. We see that almost every day… That’s not just the radio vendors,” but Qualcomm and many others that are involved.”

That guy wasn’t the T-Mobile President of Technology for nothing!! By the way, he said that back in 2021. 

However, if the Open RAN model makes sense going forward, we’ll start moving in that direction. If it doesn’t then we will find something better. Moving into 6G I believe that Open RAN and vRAN could be part of it unless something better happens. 

How does Massive MIMO fit in?

One thing about the evolution of RAN, it goes beyond the full network. We have to look at how the radios and antennas are evolving. By putting the radio and antennas together in the past, they would have less room used on the tower and eliminate RF jumpers. Seems like a win, right? 

Well, the drawback was that when new spectrum was added or they would go from 2×2 MIMO to 4×2 MIMO then a complete change had to happen.

Radio upgrades happen all the time. That was also an issue.

Today we have smarter radios that are somewhat future-proofed. They have more hardware in them that can be activated in the software. That makes moving ahead better.

So along comes massive MIMO where they crammed antennas into the antenna, roughly 32×32 or 64×64, where each small antenna is controlled by the radio to concentrate the signals, (receive and/or transmit), where the users are. This is an amazing breakthrough allowing better coverage and more throughput using science. It’s pretty cool. I wrote about it back in 2018.

This is an amazing improvement over standard radio antenna models because it allows concentrated coverage using beamforming, better throughput by simultaneously transmitting and receiving data at the same time, and allowing connections that are reflected or not line-of-site to each device. 

Evolution in the radio and antenna is still happening today. Long live R&D!!

That brings us to the network.

What about Private Wireless Networks?

OK, here is where I could see Open RAN really taking off. P5G networks would get a boost from Open RAN and all the smaller OEMs could get in on the act.

When building a smaller network, you want fast, easy, and cheap.

If the vendors are correct, then Open RAN in P5G would make deployments so much easier. Assuming radios would get closer to plug-and-play. In the US, SAS is already in most of the radios. 

It makes sense to start here and get it rolling out on a smaller scale. Most private networks are a controlled and limited environment.  

I have seen this so many times in the industry, smaller companies and OEMs chase the carriers thinking they can break in and impress their investors. I’ve seen companies live and die by this. It takes deep pockets to make it happen. It also helps to have some friends in high places within the carrier to support your efforts. 

It’s not so easy and when someone tries to disrupt the industry and when it does happen it’s impressive. However, that is only part of it because then they are under the gun to deliver, perform, and maintain. 

I’ve seen a lot of companies die this way. It helps to know someone high up.

I believe that private networks will be the next RAN system to be revolutionized. I think that these networks will open up new ways the networks can be utilized and deployed. This will be an evolution to put a dent in the industry. 

I believe when we talk of RAN evolution, private networks will spark the industry as they catch on. 

Tough Times Are Here for Wireless Workers

I was reading an article by Iain Morris in Light Reading where he talks about the harsh business conditions ahead for new RAN participants. I see it hitting way more than RAN vendors, it’s already hitting the front lines, climbers, engineers, project managers, and so on.

Mr. Morris mentions that Mavenir is going to get hit hard. I can see that. It makes sense, right? They will be hurt when all of their customers slow spending. 

I think the real question is, how will they move forward? Will they be bought? Would a larger OEM buy them just to make them go away? Do they have another trick in their bag to spark new growth?

I guess we’ll see. Personally, I would like to see them survive to provide competition, but they have to turn a profit. In business, money matters. 

Oh, when I say hard times are coming, they’re already here for most of the industry. I look at 2023 as a huge course correction for tech. Look at all the downsizing we’ve seen from Facebook, Twitter, Google, and more. In telecom, we’ve seen the downsizing recently. It’s very sad. There is a wave of downsizing across all of tech. 

I predict people will start leaving the telecom industry. I get it. We work long hours, get little recognition, and the pay could be better. My only question is, where will they go? It’s not like it’s  better in other industries, is it? Fiber? Construction? Aerospace? Electrical? I just don’t know because I am an old dog. At some point, they may put me down. 

Evolving RAN Business Cases have to make sense.

When making a change in the RAN, it has to make sense. I don’t think people realize the expenses that carriers put into their RAN. It is something they have to change or modify across an entire market. In the US alone, carriers spend billions of dollars to make that happen. That is why in some countries they put off the 5G RAN upgrade as long as they could. To make sure it was mature to eliminate as much site work as possible. 

Look at the expenses around RAN work. All the site work goes well beyond tower work. It all adds up.

Services:

  • Lease amendments.
  • Surveys.
  • Construction drawings, updates, redlines.
  • The actual tower work.
  • Structure upgrades where required.
  • Additional steel and mounts on the tower.
  • Utility power upgrades where required.
  • Backhaul upgrades where required.
  • Commissioning of equipment.
  • Integration of equipment.
  • On-air validation and testing.
  • E911 testing and validation.
  • Coverage testing.
  • Alarm clearing and monitoring.
  • Upgrades, bug fixes, etc.
  • Bring carriers live.

Then the actual hardware at each site.

  • New radios.
  • New antennas if required.
  • Additional steel mounts, standoffs, clamps, etc.
  • New fiber at site.
  • New copper for power.
  • New RF jumpers if required.
  • New BBUs, cards, and radio support products.
  • New networking equipment as required.
  • Batteries.
  • Updated power plants.
  • New cabinets if adding equipment.
  • Maybe a platform expansion if required.

Then reliability.

  • Product test and acceptance.
  • Downtimes.
  • Network risks.
  • Potential failures.
  • Network outages.
  • Utility upgrades.
  • Backhaul modifications.
  • Core updates.
  • Alternative solutions.
  • Temporary solutions.
  • Software test and acceptance.
  • Feature testing and acceptance. 
  • Large customer complaints.
  • Repairs and replacements.
  • Product upgrades during rollout.
  • Market feedback.

Then the use of the end device needs to be considered. That’s the difference between a private network and a carrier’s network, the size and use. A carrier has to approve every device that may run on it and it has to appeal to a huge audience whereas the private network may be built for one purpose or limited purpose. That’s why it can run into so much money.

Each carrier needs to think through the use of the network and what they want to add for a feature. How they utilize the network is how they will make money moving forward. A lot of thought has to be taken into consideration. 

Moving Ahead.

I think we have a lot of changes coming to move ahead. I know that the industry is already buzzing about 6G. They’re already trying to determine standards and a vision of what that will look like. 

I would think the innovation will be amazing, but only if they have a futuristic vision to make it happen. I want them to think outside the box.

I would also think that software will be the future. They should be able to make changes in days instead of months. 

I would think that moving to SA and P5G will make networks personalized.

I think the Open idea may close due to security concerns. Security will be a huge part of moving forward. You see how many hacks, lockdowns, and ransoms are happening today. This has to stop and can’t be allowed on wireless networks. 

  • Secure networks will be a priority. 
  • Expanded reach matters.
  • New use cases will be discovered.
  • Localized networks will take off.
  • Individualized features will be the rage.
  • Throughput will be critical.
  • Low latency will push the Cloud connections to the Edge and then to the Fog.

I still get excited, do you?

Links to Learn:

Reducing RAN Energy Costs

Macro site cooling is becoming an issue moving ahead. As it turns out, energy is the real issue. Utility bills have become a huge expense. Why? Did you look at your own bill lately? Mine shot up over 20% over the past 2 years, and I don’t see it going down. Think about that.  I am just paying for my home, no data room, no radios, and normal heating and cooling. 

At a site, there is so much equipment that sucks down huge amounts of electricity. A macro site has more to do than ever before. I think people outside the industry imagine we just update the radio or add a new radio, it’s way more than that. There is more equipment at a site than ever before, using more channels, spectrum, and bandwidth at all the carrier sites. More than ever in telecom history. 

While 5G equipment is more efficient, the additional equipment uses more power. The equipment uses less energy per bit but the bandwidth went way up, from 20MHz/site to over 100MHz/site at major carriers’ sites. More bits mean more energy, and that means higher electric bills. 

The OEMs have done a good job at getting the radios more efficient, but they need electricity to do all the functions they’re asked to do.

The two critical things at every site are the utilities – power and backhaul.  

Also, think about what equipment is at a macro site. It has radios, BBUs, rectifiers, routers, switches, possibly servers, muxes, fans, air conditioners, cameras, lights, telco interfaces, and security systems. Even more which I am missing.

In other words, there is a lot of crap to power in a macro site. 

Let’s break down why power matters;

  • Carriers usually have a lot of Macro sites, according to Steel in the Air:
  • Now, let’s look at what the assumed average annual macro site electric bill was, according to Lucas Mearian from Computer World, in 2022,
    • Urban sites used about 57.5 MWH costing $7,877.50 per site,
    • Suburban sites used about 46.25 MWH costing about $6,336.25 per site,
    • Rural sites used about 12.5 MWH costing about $1,712.50 per site,
  • Now, let’s make up a guesstimate of the ratio and believe me this is all hypothetical,
    • 45% Urban,
    • 35% Suburban,
    • 20% Rural,
  • Now, let’s look at the averages I put together down below, and remember, I am making most of this up based on the articles referenced above. Actually, I would bet it’s much higher now. I remember in early 2022 my utility bill shot up. The actual costs shot up so a comparable bill from the prior year was 10% less. So imagine that attached to the numbers below. 

Let that sink in for a few moments. Imagine the electric bills on all those sites, Imagine your collective utility bill is over $400M every year! Now, imagine that goes up 10%, which is $40M for every year after that. Imagine paying hundreds of companies across the US and not being able to get any type of bulk discount because the sites are so spread out, not in one large facility like a manufacturer or warehouse could do. 

Now, imagine you could save 20%, that would be over $80M/year. Let that sink in. This is an annual savings, so over 5 years that becomes a $100M savings. 

FYI – utility bills do not go down. Have yours gone down?

What am I missing? Well, utility bills shot up in 2022 and 2023, adding 5G and networking equipment to sites added to the utility cost. I didn’t take any data centers or small cells into consideration. The real number that carriers pay for utility is probably way higher! Probably 40% to 80% higher if I had to guess. 

In today’s world, all we need is another thing to maintain. However, we’re going to take a look at liquid cooling in networks and at tower sites. I want to go over how this may help shape the networking industry moving forward.

Can we make the radios more efficient?

To be honest, the major OEMs have done a good job making their equipment energy-efficient, although some carriers may disagree. The equipment is always getting more and more efficient.

The OEMs also have another trick up their sleeves, they realized that by using intelligence, specifically ML and AI, they could lower power or even shut down channels or sectors during slow periods saving power at multiple sites across the country. 

Each OEM seems to have a strategy. 

They have more control over software than they ever did. That means that the radios and their controllers are smarter than they ever were before. Everything is monitored more or less in real time. Software and automation have gotten smart. 

Let me give you a high level about how this helps. Using ML and AI we track usage at a site and build models around the busy hours and the slow hours. When a site is very slow or not used at all, it can do a few things.

  • Reduce power, if it doesn’t need high power, why use it?
  • Shut down carriers, if they’re not needed, why use them?
  • Shut down sectors, or put them into a “sleep” mode.

In other words, the radios are learning to be more efficient with their power. They are becoming smarter. This means that if they can accurately track usage, they can use less power in the downtime. 

Think of your home, I would bet many of you have smart thermostats. When you go to bed at night you may set the thermostat to use less cooling or heating. You may have smart lighting control the lights on/off times. You’re saving power, good for you! 

Smart, right?

OEMs mention that the radios are designed to be more efficient. That’s nice to hear, but it’s an expectation that equipment has evolved from the previous generation. They can pack more features into one unit and make it more energy-efficient. In today’s world, that is an expectation with any high-tech equipment. 

The one thing I will talk about further down is how Nokia is offering liquid-cooled equipment for the cell site. It’s pretty cool. 

What can the carriers do?

Looking at it from a carrier’s perspective is different from most OEMs. They have to look at each site holistically like they do when they pay rent. That’s why shelters were replaced by cabinets. Smaller ground footprints meant less rent money. More bands in one radio and antenna means less tower rent.

Let’s look at the site energy holistically and then look for reductions. This means to break out each item that uses serious energy. Let’s look at the list below.

  • Radios,
  • BBUs,
  • Networking gear,
  • Telco and fiber equipment,
  • Cooling and heating systems,
  • Rectifiers,
  • Fans,
  • Battery charging systems (which might be in the rectifiers, sorry for being redundant),
  • Lighting,
  • Security systems like cameras, sensors, alarms, and so on,
  • Any other item using power, 

All of this adds up. 

For utility and maintenance, there was one thing that had to go, radio fans. Fans are an inductive load using energy and would fail often. When the radio is on the tower, you need a crew to replace the radio unit just because a fan failed. In the old days, when radio fans failed then they would soon overheat. Radio fans are minimized in radios today. The OEMs did a good job minimizing the number of fans.

Cooling is a site issue. Why? AC units use a lot of power, tend to run all summer long, and fail  from time to time. At least that was my experience. AC units also need to be maintained which adds cost because a different skill set was required to maintain and repair. Normal site techs didn’t normally do this. 

Cooling units had to be reduced. Air conditioning (AC) units add a lot of cost for many reasons, including maintenance and utility bills. If these units can be reduced or removed, then the carriers save in multiple ways. Many cabinets are working to use airflow for cooling. They use sun shields to reduce solar heating. Cabinet fans are used to reduce added AC costs,  (even after what I said above). Fans are cheaper than AC units. Anyone can replace a fan. 

As for the radios, this is where we leverage the OEM’s smart savings features. This is already happening, but as I mentioned before, implementation takes time. These features need to be tested and go through acceptance. That is never quick in the carriers’ world.  

What if we increase revenue per site?

One thing to look at is how each site claims revenue, to make the energy per bit more efficient. We want to make each site a profit center. Thank you FWA.

When the carriers started offering FWA, the suburban (and maybe rural) sites became profit centers and not fill sites. Suddenly, more sites became profitable.

Even in urban areas where sites may be slammed during the day can now offer home internet at night making each site more profitable by providing connections all day and night.  

By adding more subs and new services at each site, the sites may use more energy, but they are making more money. This means each site can provide more value to the carrier. 

This is an upside to FWA. All those rural and suburban sites that used to just handle a few subs when they traveled into that particular coverage can now provide internet access to the homes surrounding it. 

Can we use alternative power solutions?

People think that renewable power is the end-all solution when in reality it creates a new series of problems. 

People think you can just add solar and wind then you’ll be fine. That is so far from the truth.

I have looked into this and it’s not as easy as you think. Remember that a macro site requires a lot of power. Especially in the US. The power needed to make the equipment run is usually too much for a standard renewable power system. 

Let’s look at what’s needed.

If you go solar, you need a lot of real estate to get the power you need. That’s why in an urban setting, solar is next to impossible unless you have a parking lot or an entire roof to spread out solar panels. You need a lot of room to put panels out.

If you add wind, the same thing only now you’re going up and out. Wind turbines need space and they do fail from time to time. They also require maintenance.

Now, with all the other issues listed above, you need to engineer for those services. Sunlight in some places is not as abundant as in others. The wind isn’t always blowing. Planning and engineering matter.

Finally, this is the elephant in the room. You will need batteries, lots and lots of batteries to sustain the power once you capture it. Even if you have power, you need batteries. If you rely on sustainable power, you need many times the batteries a normal site needs. That means more real estate is required inside the shelter or on the ground. You also need all the equipment to support it. 

Batteries are an issue for so many reasons, and not just because of what they can do to the environment. There is the ethical sourcing of batteries. Some supplies come from the Congo in Africa and the work conditions there are just horrible. I would recommend you read the book “Cobalt Red” by Siddharth Kara, it was enlightening to me. We are all moving to lithium-ion batteries, and the electric vehicle situation made them even more of a priority. Also, with those batteries, the energy to make them is ridiculously high causing more of a carbon footprint than most gas vehicles cause for a long time.

It’s not a solution for most stand-alone sites, at least not in the States. It could be a backup or something to lower the electric bill, but at what cost? The more ground cover you need, the more rent you pay. All the other stuff is required but it’s just not a great model in the US.

Maybe for small cells, but even then you need batteries to keep it on air. Batteries are heavy and require room, which if it’s a small cell you may not have it.

So as you can see, sustainable power sounds like a great idea, but in reality, you have to take it for what it is, a somewhat viable backup solution unless, or use it if it’s the only thing you have available. Then you have to make it work.

What is Net Zero?

I hear bigger companies talking about Net Zero often like it will save the world. Maybe it will. So, what is Net Zero?

I have looked into this to see what it is and an article that I could follow was found here. It’s by Vaishali Deshmukh, called “Net Zero: A Short History”. She says that Net Zero is, “Being at net zero emissions refers to a point where the GHG emissions released by humans into the air are balanced by the emissions removed from the air.” 

What does that mean? Well, GHG means Greenhouse Gases. It’s complicated, but the link explains it. For this article let’s just say it’s Carbon Dioxide and other gasses. It has more than that in it, but read the links I provided for more information.

GHG gasses are one of the reasons, so they say, that we have global warming because it traps heat

To go Net Zero means that we would reduce our GHG emissions so that it’s reduced to match what is removed from the air. That means that climate change would be reduced back to where it theoretically should be. Again, this is speculative. 

The United Nations had a conference in Paris to reduce GHG being released, but they did not call the initiative Net Zero. It’s called the Paris Agreement by the United Nations Climate Change Conference (COP21) to have countries limit the GHG emissions. 

Don’t forget the backup power!

A carrier’s requirement in the US is to have backup power. I don’t know what the requirements are but most carriers have batteries at macro sites. Depending upon where it is in the US they could have more than others. They also rely heavily on generators to keep the sites on the air.

In areas where tornadoes or hurricanes happen often, you need to have a better contingency plan than in other areas. 

This does impact design and costs. Some carriers rely on a fleet of generators to be delivered immediately after the disaster. In this situation, you have to have emergency personnel pick up, deliver, and connect the generators to each site. This takes a lot of drivers.

Also, you need to have fuel at each site. If power is out for days, then fuel delivery becomes critical. If you know a storm is coming, make sure they top off all the generator fuel tanks.

Fuel could be diesel or propane. Maybe gas, but gas is explosive and dangerous at a site. It’s hard to believe we put it in our cars.

Some generators run on natural gas. These are great because neutral gas usually is the most reliable thing you can have if you can get it connected to the site. Most sites are too remote, but if you are near public gas connections, this is a good way to go.

No matter what your backup plan is, you need to do maintenance and test it regularly. If it doesn’t work in a test, it will fail during an emergency. Then you have a bigger problem! Now you have to fix the problem immediately after a crisis. 

Batteries need to be monitored as well, just because they’re sitting there won’t mean they will work. They need to be looked at and tested if possible. All batteries are capable of failing just like anything else at the site.

Plan properly!

Is Open RAN more energy efficient?

To be honest, I am not sure, even though you can find articles that will argue either side. Some say it is and others say that the goal is to make it more efficient someday.

To be honest, I think we need to see OpenRAN deployed in mass by major carriers and then we can see what they say. For now, let’s just assume it’s on the roadmap.

Liquid Cooling, a brief history:

This is where you use a liquid to cool down equipment. For those of you older like me, think of your fossil fuel car and the radiator. This is one of the earliest examples of liquid cooling. In the early days of automobiles, they had air-cooled engines. Air-cooled means that you rely on flowing air to cool your device. Smaller motorcycles are generally air-cooled. They rely on the moving air to cool the engine. However, they will overheat, causing damage to the motor. Unfortunately, when they do overheat, it’s generally too late, damage is done!

So, the idea of liquid-cooled engines helps the engine stay cooler and prevent damage. If you’ve ever had a car overheat for an extended period of time, then you know that it causes damage to the engine, not to mention expensive repairs. 

Hence, we have idiot lights and temperature gauges to let us know when the engine gets too hot. 

Liquid Cooling in Data Centers

Now, imagine that data centers have the same idea. They connect the equipment to heatsinks and, in some cases, run the fluids through the equipment to a radiator outside in the shade.

The radiator has fans on it forcing the air through to cool the liquid as needed. You have pumps that keep the fluid circulating. Now you have an alternative cooling method for data centers. 

Yes, utility power is a critical thing at data centers. One of the key reasons for the high electric bills is due to the air-conditioning systems. I don’t know if you’ve ever designed or built a data center, but I helped a few times. The cooling system is critical for many reasons. 

Here are a few reasons why:

  • You have to have multiple cooling units, for redundancy. Some people install multiple smaller units. Either way, it has to be redundant. 
  • You need to understand what the power requirements are. Larger units are usually 220VAC in the US and the smaller units are generally 120VAC on 20A breakers.
  • You need to know where the bulk of the equipment will be and route the cool air toward those cabinets. Back then we would normally pump the cool air under the floor and have holes under each cabinet so the cool air would be sucked up into the cabinet where the equipment was.
  • Over the cabinets, we would have the returns so the hot air would immediately get routed back to the cooling units. 
  • All of this planning had to be done in advance and have the option to make changes as plans change. Not an easy thing to plan.

So, you can see it takes a lot of planning and power to make the AC units work. Then, you have to think about the generator. Keep in mind that air-conditioners are inductive loads. That means there is a heavy load on the generator. Add in all the equipment in the data room and you have a large generator to sustain your data center.

If you want to read a good data center article, Vlad-Gabriel Anghel wrote a good article recently found here. Good and long article if you have the time. 

Now, what if we used that technology at tower sites? Remember we already  moved the radios outside, but we have the networking equipment and the BBUs inside the cabinet/shelter.

A short history of Liquid Cooling for Transmitters

I know, most cell sites have their radios outside, but let’s look at transmitter cooling.

Many years ago, broadcast transmitters used tubes to transmit thousands of watts to hit the neighboring towns and people. To keep these tubes and equipment cool they would run liquid through the heatsinks of the equipment. When I say tubes, I mean Vacuum Tubes which use high voltage to power them.

Remember that air conditioning did not work as well back then as it does now. All the same, they put off a lot of power and they needed to be cooled so they wouldn’t burn up and fail. With or without AC liquid cooling was a critical part of the system.

As time went by, most radio station transmitters ran cooler and cooler but TV stations still needed to be water-cooled. 

Think about TV and radio stations. They were very powerful transmitters that used AM for video and FM for audio. Pretty amazing, right? They needed high power back then. They needed to be cooled. Today the broadcast transmitters are all digital and run much cooler. They also just transmit a digital signal, no more analog AM and FM. All that is behind us now, at least in the USA. Broadcast is way more efficient than it was 30 years ago. 

However, TV broadcast transmitters transmit high power, so some still use liquid cooling. They are solid state but require serious cooling that air conditioning just can’t handle. It’s efficient to cool using liquid. When I worked for FLOtv for Qualcomm they used Rohde and Schwarz transmitters that required liquid cooling. They had to be maintained. We still needed air conditioners though, crazy!

If you want to see some, here’s the Rohde and Schwarz product link.

To maintain them you need to be sure the system is good. You have to make sure of the following:

  • No air in the system,
  • Antifreeze had to be a certain mix, usually 35% to 50% antifreeze to water,
  • Bleed out all the air,
  • Verify no leaks, all connections have to be treated,
  • Make sure the pipes are away from traffic and protected,
  • Make sure your radiator is safe and if at a site it needs to have an ice bridge over it, as do all the pipes that connect to it,
  • The pump needs to be tested as do the fans on the radiator.

Maybe you didn’t realize this, but this liquid was not just water, at least not the ones I worked on. They had antifreeze which not only prevented freezing in cold temperatures but would allow the water to heat up over 212 degrees Fahrenheit. Remember you didn’t want it to freeze or overheat. This was all taken care of with the antifreeze you put in the system. 

You also had to check for leaks under pressure which means all your connections had to be treated. That was critical. Sometimes they would start leaking months later, I have no idea why but when that happened you had to repair it in the maintenance window. 

It was always something. 

What about Carrier’s Macro Sites?

OK, here is where I was headed. In today’s carrier sites, the radios are generally located outside the shelter or cabinet. Normally up the tower near the transmitter. Their frequencies are much higher than broadcast and line loss became a problem as carriers went higher on the spectrum chart. 

So that eliminates one cooling problem, doesn’t it?

However, there is networking equipment and a BBU inside the cabinet. The BBU can overheat and needs to be cooled. Fans can do a good job but fans fail. That’s why so much of today’s equipment relies on natural airflow.

So Nokia came out with a liquid-cooled BBU! How cool is that? (I mean that figuratively and literally.)

The idea is that you can cool the BBU by moving liquid through the BBU to keep it cool using a radiator to make it happen. The idea here is that you won’t need AC units, or at least they’re used a lot less. AC units use more energy and need cooling agents. Maintenance requires an HVAC tech to take care of problems. 

With the liquid-cooled model, you could have your tech trained to maintain and repair the units as required.

Acronyms

  • AC = Air Conditioning
  • AI = Artificial Intelligence
  • BBU = Baseband Unit
  • KWH = KiloWatt Hour
  • HVAC = Heating Ventilation Air Conditioning
  • ML = Machine Learning
  • MWH = Megawatt Hours

Links:

Why is Wireless Talent Leaving the Industry?

Why would they? Maybe the sudden slowdown in 5G has something to do with it.

If people are leaving, how will this hurt the ecosystem?

It’s not just climbers, but the ecosystem of talent will be affected.

That’s what I look at, each job type, each type of job being downsized. I may not cover everything, but I cover a lot of it.

If you want the short version, here it is. The 5G build is slowing down quickly. That means contractors and companies lose work. That means they have to lay people off. That means people are out of work. Workers have to pay bills and support their families, so they are moving to other industries to make money.

I would bet even the carriers will downsize. They will lose people along with the rest of the ecosystem.

There, I just saved you 15 minutes of drudging through this detailed report. Now you can skip to your section or job and just read about it. What’s another 2 minutes?

Read on wireless fan! You know you want to.

The Mystery of Losing Wireless Talent

I was talking to my friend, Brad Larson (shameless plug) from CommScope, who mentioned how the wireless telecom industry is losing talent. He knows people who either got laid off or left to go to other industries. I see it too. The talent pool is definitely dwindling. Not only through attrition and retirement but, (as the 5G builds dwindle), people get laid off. 

The current talent pool may be too heavy for the industry, or is it? Is the talent pool for wireless shrinking mainly because we were shorthanded until we were bloated? Not through any industry mistakes, it’s just the wireless deployment cycle. 

No wonder people are leaving for greener pastures.

We know the number of wireless and telecom jobs are declining due to automation and less deployment. It will continue to do so for the next 3 to 7 years? Another reason to leave if you have another opportunity. Who can wait that long if they’re not working today?

As always, the cycle repeats itself. The work peaks, then declines. Usually over a 5 to 10-year period. This is how the cycle works. It happened in 3G, 4G, and now 5G. It will happen again. Unfortunately, this time it was around 5 years. We also have more automation than ever. 

Since 3G, a lot has changed in how we deploy. We had people at the site that could do anything, not just tower work, and commissioning, but integration and troubleshooting as well. Thanks to AT&T the turf system back in the 4G build helped destroy that model. All the carriers adopted it to save money. Shareholders love that. Unfortunately, after they started it back around 2008 a lot of young climbers died.  

Why didn’t I like it? Let me give you a list.

  • We lost work because suddenly crews that didn’t train or have good equipment or take safety seriously, could come in and underbid us. 
  • The model spread out from carriers’ work to state and local work, broadcast, and other industries. It eroded the entire ecosystem for work.
  • It took experienced climbers and made them move to other industries.
  • It took away any security that these jobs could have.
  • It made the climber a common construction worker which became a commodity. 
  • It devalued a skilled segment of the industry. 

Hey, that’s what I saw.

However, I want to cover more jobs and skills in this article. I often focus too much on the climbers. So, let’s look at them along with other skill sets. 

But, do we need them?

Well, if you remember just 5 years ago, everyone was desperately trying to get more climbers, PMs, technicians, and so on. We were worried because there were so few wireless workers in the industry.

Today with the slowdown, they are being let go. Sorry, but that’s the cold truth, the industry no longer needs them. So, while we need some people, the ramp-up has changed into a ramp-down. Just like we did in 3G and then 4G. The cycle repeats itself.

This time, many people get to retire. Not me. I think I’ll stick around to see 6G, why not? I like this stuff and as long as I can get paid, I will drudge through this and get ready for the next G. In fact, I am looking forward to it. Oh, all the drama around 6G, all the what-ifs. Sorry, I digress, let’s talk about today.

Tower Climbers

Guess what, carriers are not giving up the GC turf system. Why? It’s profitable. But what are the long-lasting effects of the turf model? I have some ideas.

For one, tower climbing was once a skill that took time to acquire and you more or less had apprenticeships to train people. You also taught your climbers additional skills, like connectorizing cables, using test equipment on the tower, grounding, civil work, and sometimes electrical work. In other words, when you had a good climber, you trained them with new skills. 

Today, that model went out the window. Unlike broadcast tower climbers, who still take their time, telecom climbers are about being quick. Then, they had to lower the training standards from months to a few days. At least we have apprenticeships now.

Don’t get me wrong, there are great climbing schools out there, but so many people took shortcuts in the past. I think we’re doing much better today, but we all know there are loopholes out there.

the new training must be working since the number of deaths this year is much lower, but even one is too many. 

So now, when a large industry build comes, like 5G, we just hire younger people to climb, like we expect to see in construction jobs. The climber has slowly become a commodity. 

However, that has changed as the US’s economic downturn has changed. Assuming these Zippia statistics are correct. Stats below from this Zippia Article.

  • 12,181 tower climbers currently employed in the United States, (2022)
  • 4.3% of all tower climbers are women, while 95.7% are men. (Step it up, ladies!)
  • The average age is 45, (seems old to me unless you go beyond telecom).
  • Ethnicity breakdown: White (57.7%), followed by Hispanic or Latino (18.1%), Black or African American (12.2%), and Asian (6.6%).
  • Sacramento, CA pays an annual average wage of $53,283, the highest in the US.
  • Tower climbers are 82% more likely to work at private companies in comparison to public companies.
  • Over 49% are high school graduates or GED, 25% to 30% have some college.
  • They also say 18% are in telecom, 12% in construction, 12% in professional, 11% in manufacturing, 11% in technology, and the rest are scattered. I have no idea what all those industry breakdowns mean. Maybe the tower is used in a different context for other industries.

Now, back in 2015, Wireless Estimator reported over 29,000 climbers in the US. So, from then until now, we had a huge decline. I know that after the 4G climber boom, many left the industry and they had to build back up again. Maybe it never recovered from that loss. 

The above numbers fluctuate greatly. Who knows what the real numbers are. On a personal note, I am trying to find an annual tower climber count. I can’t seem to find anyone tracking that statistic. I am open to ideas on how to track this. It doesn’t have to be accurate, but ballpark numbers would be helpful.

You may not remember but when the 5G deployments started happening, we had a shortage of climbers. Unfortunately, it didn’t mean a big pay increase for them, instead, they just trained more people. I would think they make more money now than back in 2015, but I am guessing. 

Maybe some came back thinking it’s better this time. I would bet that most of those people were older, not the 20-somethings we had years ago. Maybe I think that because I am older, not sure.

Here is what I’ve seen, and please, keep in mind my world view of climbers is very limited.

  • More and more Hispanics are becoming climbers. They can break into this industry with little training and some construction experience. It helps to be healthy and strong. After a few years, they are experienced. Maybe this can be part of their path to citizenship. I don’t know about legal or illegal but I do know they can do the job. They also understand that they have to work when it’s available.
  • While I see climbers that are of all ages, they are getting older and older. I don’t know what this means in general, but I don’t see as many early 20-somethings as we had 10 years ago. 
  • Many climbers go to the highest bidder and when times are good, they have more companies that will take them.
  • Loyalty is split. Some climbing companies will work for companies that may pay less if they are guaranteed work. We all know that work will dry up, like I said before, this work is cyclical.
  • I have seen a rise of climbers from Western Europe, Ukrainians, and others who have come over to live in the States. As long as they have their green cards, they can experience life in America. 

What can climbers do as the market turns down? They could try to work in other areas, like broadcasting or water tanks. They could also transition to a new job with new skills, like estimating or construction management. They could work in construction. They have options. They can transition.

Unfortunately, the toughest part of the market is as things ramp up and wind-down is when we bring in many people just to say goodbye in a few years. So sad.

Project Management (PM)

One often overlooked job in the industry is PM. The PMs in the industry are everywhere. We have them in with the OEMs as well as all the carriers. PMs are everywhere

But, here’s the thing, as contractors and carriers cut costs, (meaning headcount), they will downsize the PMs as required. This may seem like bad news, but you know what, PMs are used everywhere in literally every industry.

There are PMs in all construction as well as software. Even IT projects have PMs to manage builds, planning, and growth. If you have your PMP then you can take it to almost any industry that does projects. It’s a transferable skill. 

So, if you’re a PM and you see the industry slowing, you have to make a decision. That would be to hang onto where you’re at or start looking. Looking may help align something down the road. If you have a good LinkedIn presence, you have a shot to find something. There are also Facebook groups that target a specific industry. I have a tower climber group but maybe I should add PM Careers.

Be willing to look beyond your present capabilities. You have the opportunity to expand your experience beyond wireless or telecom. 

Construction Management (CM)

I broke this out of PM because it’s a different skill set and job. Although, in most organizations, this falls under PM. 

We will need CMs to go out and inspect sites, but growth determines jobs. We’ve had growth and I know many climbers were able to move into this but they have to have an understanding of spreadsheets and BOMs to make it happen. They also had to be able to walk the site and estimate the work with accuracy. They also had to be able to update CDs, what we call redlining CDs to match what’s actually at the site compared with what the architect drew on the original drawing. 

Following the PM lead, as site expansion drops, the CM is less and less in demand. This sucks, I know, but the CM is there to walk sites and red-line drawings. As that work declines, so will the CM head count.

What can the CM do? They can look into other industries and try to transfer that skill set over to electrical or construction. They could try to see if generator deployments are increasing or if DAS construction will take off where they are. I think they have to work outside the industry if this is what they want to do.

They could also try to transition into the PM role and work more on the planning and tracking side of the business. They could look into getting a PMP certification.

What I am saying is they have options. Don’t feel trapped in wireless. Be willing to look at what else is out there.

One more thing, there are transitions to being an estimator, quality inspector, close-out specialist, and so on. You have more options if you can think outside the box.

Site Acquisition

Here is something that seems to be ongoing, site acquisition.

This is generally a combination of paralegals and someone in the market who can go to the local municipality to pull and file paperwork for permits and approvals. This is tedious work and is generally what slows down a lot of deployments. However, it needs to be done. There are so many municipalities in the US and each one seems to have their own rules and requirements. When deployments ramp up, it all starts with the site acquisition teams. They not only have towers and rooftops, but they also have ROWs, parking lots, and lamp posts. 

This will slow down but never stop completely. This doesn’t mean much though, they downsize these teams quickly because when it slows, you need a much smaller team very quickly. 

These are usually people who can do more than just site acquisition for wireless companies. They also can work in construction, retail, fiber rollouts, and so on. Some go on to take their bar and become lawyers.

One of the key things that come out of the site acquisition team is the CD and structural study. These are required at every site before changes can be made. The structural study validates that the tower can hold the new equipment. 

The CD on the other hand is what most people look at to make sure there is room, and what work will need to be done. Sure, someone has to walk the site, but not all site acquisition teams walk sites anymore due to cost and time. Most just look at the CDs and Google Earth to determine what needs to be done. 

IT Networking and Routing

Let’s not forget the skills that are required for routers, backhaul, and data centers. This is something that most companies require the bare minimum and overwork who they have. Crazy.

Yet, this is going to continue but not like we know it today. 

There are so many IT techs and engineers. This is a very diverse industry, but I am going to focus on the teams that work on wireless deployments. This is generally the techs and engineers that do the planning and deployments. Most don’t even go to a site; they have the installation or C&I teams handle that part.

These teams get super busy prior to and during the rollout. They have to be on top of what’s happening. It’s usually a PM that tracks it and today we have a lot of software that does most of these skills.

When this work tapers off, they generally start looking outside the industry, which isn’t hard to do. They generally have to be where the work is and that could be anywhere.

They should always have work, there are routers and servers everywhere that need to be configured and constantly updated. The Cloud didn’t solve all the problems, although it’s way easier to offload and let the Cloud provider take care of more and more.

Network Operation Centers (NOC)

The NOCs get larger when they’re tracking the rollout. Then when the rollouts slow down, they generally start to downsize and move into a maintain and operate mode. This means you have fewer people and they can be less talented. 

What less talented means is that the software they use has gotten smarter to track alarms and people at sites. So fewer people and fewer engineers.

The troubleshooting engineers are cut back to a minimum or given double duty doing something else until they’re needed. 

So, NOCs will be getting smaller. Many should be coming back onshore in the US but I am not sure how that will work moving ahead. On the upside, every industry uses NOCs, so if they’re lucky, they can find a new NOC or they can level up their skills and take on something new.

Radio Frequency (RF) Engineering

Let me tell you something, RF Engineers used to be the bomb!! They could do almost anything. I’ve done it in the past. Today, for planning anyway, it’s mostly software. You may still need RF guys on the product side or to troubleshoot, but the RF modeling is generally outsourced. In fact, most carriers send that over to a software group that maintains specific software to do the modeling. 

For all the contractors that had a lot of work, it will move over to the private network teams. They will use you, or a software package, to do most of their planning and reports. It all depends on where the strength of that particular company lies. If they have a good team, they may try to do it all themselves if they can support the salary and licenses. If not, then they outsource to an RF contractor.

Either way, this work will continue, business as usual.

Commissioning and Integration (C&I)

OK, for those of you that think everything is plug and play, pull your head out of your ass because it’s not. We have to go to the site and power up the equipment. What equipment? Radios, BBUs, routers, servers, and so on. There is still physical equipment that needs to be powered up and configured.

This is no longer hard to do, but as of today you still need someone who knows how to read schematics, test DC power, verify DC power, and configure the equipment. It takes some skills. I’ve seen BBUs burn when it’s done wrong.

Why don’t they trust installers to do this? They want to but in the past installers have done things wrong and when things are wrong with DC power, things burn up, (see my note above). It can get ugly. So, it’s a different skill set that requires someone who knows what they’re doing. Crews used to have this skill set, but that changed when their pay got reduced. 

Anyway, these guys usually ramp up and ramp down. It’s usually an independent contractor or a GC that has teams that do this. 

I’ll be honest, I am not sure what these guys will do, but they always seem to find work. They can transition over to other jobs. CMs have done this in the past and some installers can be trained to do this work. I would think when the work dries up, these guys move into something else pretty quickly.

If you do C&I, don’t fret, plenty of carriers may be doing this and private networks are starting to take off. I would think there’s always work out there for you.

Structural Engineers

Towers and rooftops need to have the structure measured and approved. You can put too much on the tower and expect it to hold up in a hurricane. The same goes for a rooftop holding all that weight and the weight of the batteries that need to be added to support 4 to 8 hours of downtime.

In the old days, when there were towers without structural drawings nor proper paperwork to support them, we had to send climbers and engineers out there to do the tests. Today, almost all others have structural studies attached to them, in some way, all we have to do is make changes in the program to see what needs to be done. 

If it can’t hold the additional weight, then we need to get a structural team to design an upgrade, if it can be upgraded, then do the work. 

If the tower is maxed out then a new tower has to be found or someone has to be removed. Maybe build a new tower, if it makes financial sense. When adding new, this is more common than you may think. Although, many offload to a smaller pole. Today’s radios transmit more spectrum so fewer radios may be needed.

So, this work may have ramped up, but these guys are always busy. The crews that do structural upgrades generally get paid more than other crews, this is a special skill set that requires precision changes, maybe welding, and the ability to match the drawing to a tee. 

Data Room, Rack, and Site Design

For the tower sites, believe it or not, at one time this was a big deal. Now there are so many sites built that we’re adding more than designing new sites. We put shelters on the sites and we made drawings of the inside and out. We had to look at cooling and heating depending on where you’re located.

We then had people who would make drawings for racks and cabinets, using CAD to determine if something would fit. They were supposed to look at the power draw and heat to see if the cabinet or shelter could take the additional power draw and heat. 

The idea is that the carrier can show where each piece of equipment is placed. Then identify it in a drawing by name, make, and serial number. Then have an identifier showing which rack it was in and possibly how it was wired.

Today, Data Rooms are a huge thing, especially now that we’re moving over to the Cloud and Edge data rooms. Something has to show where each piece of equipment is located so some remote NOC can understand what the site looks like and possibly look at a drawing to determine what needs to be changed. 

When the tech shows up, you want them to know exactly where to go to swap out a unit. Someone has to document and it has to be available quickly.

Trust me, in the past many units were powered down because of confusion. Crazy! FYI, powering down a live unit means customer disruption, and that means complaints, and that means unhappy bosses.

Power requirements matter. The data center will need a lot of power and the power has to be ready for expansion. This has to be considered when designing the room. As you add equipment, you may need to add and expand power. Planning for this in advance really helps. If you’re lucky, this is a problem you will have because growth is good.

Cooling has to be considered and designed around. Data centers, just like tower sites, can overheat if not designed properly. Heat design is critical to any electrical equipment. Remember when laptops would get hot? I do! This is a critical component to be considered. You must cool everything. We have to make sure that the data room or closet handles all the heat and has proper heat removal.

Today we use Air Conditioners, which also need to be maintained. We’re moving towards new models of liquid-cooled units, which I have worked on in the past. This means that the equipment’s heat exchangers sit in a liquid. The liquid is constantly moving from the equipment’s heat sink to a heat exchanger outside somewhere. Picture the outside heat exchanger looks like an older car’s radiator. This also requires maintenance but the power requirements for a liquid system are much lower without any pressurized cooling chemicals. However, liquid cooling may require antifreeze in colder regions, so the chemical requirement never goes away. However, maintaining liquid cooling is way easier to maintain and the skill set isn’t as critical as HVAC techs.

The location of the equipment matters in the design. Why? Power distribution is one, you want to have the right power at the right location. For instance, if you have power rectifiers, they may require 220VAC (in the USA) to use less current. More expensive to run, and cheaper to operate. 

Also, location matters in heating. If you start stacking all the hottest servers on top of each other, you could have overheating issues in a specific rack. Then you have to relocate when everything is live, meaning customer disruption and unhappy bosses.

Electricians

We’ve always had electricians do wiring at sites. They have to be licensed and certified for the area they’re working in. This is still growing as we expand power at sites and continue to put in new sites. They could be small sites, like poles here and there, lamp posts, or brand-new macro tower sites.

We need electricians to do designs for sites, plan power, and install everything. 

These people are always busy and have plenty of work. To be honest most teams just hire the local contractor and don’t have any staff unless they do a lot of builds or extensions. 

These guys always have work, they like the extra work, but in most cases, they aren’t married to telecom.

Cabling

Inside cabling for DAS and indoor networks will gain traction. This has been mostly handled by electrical wiring teams that have telecom experience. Now they have to wire fiber inside mostly, but also electrical and possibly data cables, like CAT 5 or CAT 6.

Think of all the systems to be built in arenas and buildings that will be added.

Beyond wireless coverage and expansion, think about the data centers. There is an explosion of Cloud and Edge data centers that will be added across the US. As we try to minimize latency, we have to add equipment closer to the end user. This means servers will be everywhere. 

Most companies hire contractors to do the initial wiring and then may have their techs do additional wiring. This is an industry in itself. They may have to design your runs and cable racks along with the rest of your design team. I am not sure how you would do it. Generally, most teams have an architect design this stuff up front and then just tell the Cabling contractors what to do. 

Sales Teams

Don’t think I forgot the sales teams. They are a critical component to growth but if they can’t sell anything, they’re out looking for work like everyone else. 

As private networks start to grow, they will need sales teams. The DAS companies are primed to move these types of networks. While they want to continue pushing DAS, they could easily add a small cell portfolio to be installed. They know the business of building design and probably just want more work. They may or may not have the proper sales teams, but if the business grows like we hope it does, then they will be selling even more systems to new and existing customers. 

Presales teams are critical when companies get bigger. These are the teams that may create the initial design and quotes to support the sales team’s account executives. If these teams work well, they can turn around quotes quickly. 

It’s not about just doing the quote; it’s about listening and understanding the customer. So, the first quote may be a fact-finding document to help the customer understand what you think they mean. Then as they push back, they begin to envision what they really want. Turning around a quote quickly helps the customer see what their vision is. Then, as you go back and forth you align. Sure, sometimes you get it right the first time, especially if you have a tenured relationship with your customer. Then you may already understand each other. But new customers or new ideas change the scope.

Remember, without sales, you don’t have customers. That means you don’t have any income. No income means no paycheck and people lose jobs. Keep that in mind.

Unfortunately, when the need for services or products declines, everyone takes a hit.

Financial Teams

Believe it or not, we need financial people like accountants, Chief Financial Officers, and financial planning analysts. Some PMs do the financial part of a project. These people and teams are needed in larger deployments and smaller companies to help get the finances in order.

The good news is most don’t need telecom, even though they may specialize in it. These people can work anywhere.

These guys get downsized too, that’s when things are really tough. 

Luckily, they can migrate to almost any industry. 

Panic in the Industry

People are starting to panic now because there are larger layoffs, and more coming this year. Let’s face it, this is the slowdown we all expected. It still hurts when it actually happens. We have to ride the wave when it’s coming and learn to deal with the drop-off when it crashes.

We are losing wireless talent. We’re seeing companies reduce people more now than ever as they migrate to a cheaper model where growth is less important and managing the operation is important. Some companies move down in the talent and pay scales.

I am seeing a lot of people retire now, some by choice and some by force. This is the nature of any industry. 

The problem, as I see it, is that this industry is mature. When an industry is mature, they find it hard to differentiate and innovate. Everyone becomes locked into the “business-as-usual” concept. I think that is why we love to look at Silicon Valley’s innovation. In those markets, people became millionaires quickly through innovation by only using laptops and writing code. When you’ve been in telecom for years, you are doing things the way you’re told to. Silicon Valley found ways to make great things happen without all the bureaucracy and bullshit.

When will the Talent Shift?

We’re seeing it in 2023. In 2024, even slower. In 2025, it will probably be even slower. That means people lose jobs and they have to either shift to another industry or change job descriptions. 

Sure, some went to college and have a thousand certifications, but what good is that as the work dries up? Use that to move to the next industry.

As the world evolves you have to adjust your talents, maybe take a new position, or be daring enough to do something on your own. Playing in the status quo works for many of us, but not all. This is where we have to get creative.

The way I see it, we won’t ramp up before 2027 unless something happens. Chances are good it will be around 2030 before the industry takes off again. If it happens sooner, great, but I am not holding my breath.

In the Talent Pool Shrinking for Real?

To me, this is the real question. Why? Because if the cycle repeats, like it did for 4G and 3G before that, things slow down. It doesn’t matter what WIA, NATE, or CTIA says. It’s the nature of the telecom beast.

I would bet we see a ton of people leave wireless and telecom. Especially the tower climbers. They may shift to something in a similar field, like running fiber, pole work, specialized steelwork, or become project managers (PMs). They could even be estimators. They have options.

Unfortunately, so many people have to find new roles. Just because they can become a CM or PM doesn’t mean those positions will be there. It’s the entire industry slowing down. We will lose many people to other industries and (unfortunately) that’s normal.

If the 6G migration is anything like the 5G upgrade, then you will see all the industry associations crying about the talent shortage. 

I will say one thing, it seems like the salaries didn’t grow like I thought they would. If anything, they seem to be maintaining or even dropping. I don’t have official numbers, but let’s face it, inflation quickly outpaces the salary of the working people in this field. Is that fair? Of course not, in fact, it sucks. 

As the great downsizing happens, we’ll see even more salary erosion.

There was a day when experience could earn you a lot more money. It was not that long ago that went from a “well-deserved pay increase” to “helping you get a job at the same pay if you’re lucky”.

So yes, the talent pool will continue to shrink for the next 4 to 6 years. 

Is it for the lack of work? That’s part of it but it’s not the only thing, is it? Systems are more automated than they ever were, greenfield sites are not common but site upgrades are. AI and ML make troubleshooting and network upgrades easier than ever. Alarming is so automated that we set it and forget it.

Think about the site and system automation we built in.

  • We can get more into one radio and we’re deploying more and more integrated antennas than ever before.
  • We don’t run coax up the tower, now it is power and fiber.
  • No coax in an integrated antenna.
  • RF Engineering and optimization is moving into the radios and UEs, (as if we even care anymore).
  • Testing is done by the radio, not externally in most cases.
  • Optimization is automated and done by crowdsourcing devices in many cases.
  • Monitoring is being done by automated systems using ML and AI to prioritize alarms before they are sent and someone is dispatched to look at the problem.
  • Remote testing and troubleshooting are the norm, sending someone to the site is a last resort.
  • We can almost remotely do Integration.
  • Site walks are becoming automated, using drones and loading up to a server where someone uses AI to build the drawings and BOMs.
  • Data is kept in the cloud where anyone can access it to look at the site before determining if a dispatch is needed.

So much automation makes things hands-off.

On the other hand, parts still fail, we still have interference issues, and the carrier executives still find dead spots in their homes. This helps keep the field staff busy. 

What will the next talent pool be?

I see AI coming along. This means that developers, code writers, network security engineers, and IT teams will be a requirement. This is new tech and it can’t be offloaded altogether, in fact, I read that 6G may require AI to be the devices, not at the Edge or in the Cloud. 

Sure, we still need the people I mentioned earlier in this article, but times are changing and we will see some new developments in the future that are more about technology making decisions so we don’t have to. We need new skills.

  • I would think automation would be a key thing. AI and ML will be a factor.
  • I see security being a huge factor moving ahead. We need network security specialists.
  • Using apps will be critical. We need developers.

I am hoping the government gets smarter. I feel so often that the federal, state, and local governments are against tech advances. Unfortunately, they hold back a lot of advances unless there is something in it for them. Some see the advances as good, some don’t care, and some push back until they get something in return. 

How to move ahead.

You also have to think about where the growth is. Let’s look at where I see growth for the above talent.

  • The network infrastructure that the government is putting money towards will be huge. The BEAD funding to deploy fiber is going to eventually spark a lot of growth. Unfortunately, it may not happen until 2025 or 2026. That will spark tons of money for fiber, data rooms, and last-mile connections. 
  • We also have private wireless network growth happening in the US, thanks to CBRS. This could be outdoor networks or larger indoor networks. It could mean carrier-grade equipment as well. This should ramp up in 2024.
  • We have indoor wireless taking off beyond the larger venues, this is good for the industry as a whole. Today people rely heavily on Wi-Fi, which will include 5G going forward. It’s going to be a combination of the 2 technologies moving ahead. This is happening now but should see real growth in 2026.
  • Data rooms are always being added thanks to Cloud and Edge centers. This is an ongoing process. 
  • Smart energy will be growing as utilities start connecting everything they can. This is an ongoing process.
  • Cities will allow more private networks and encourage local businesses to build for various reasons. This is slow but hopefully, it will take off in a few years.
  • This may not help you, the reader, but as manufacturing takes off in the US, more manufacturing plants will pop up in the US. Probably in Arizona, Texas, and Florida. Nokia is expanding its plant in Wisconsin. This will help because in the US plants have to be automated and robotic. We can’t rely on a lot of workers, but a lot of machines that need to be maintained. This is where you guys come in. All of this will be built around a network, wireless or wired, maybe both.
  • But these things generally take a lot of time to develop, keep that in mind. It won’t happen tomorrow. All of the above will have long sales cycles and perhaps longer funding cycles.

Now, how should you look at this? I am not sure where you are in the talent pool, but here are some ideas.

  • These are smaller projects, so they have to be managed accordingly. One PM may manage many projects simultaneously. 
  • The jobs will be very local for installers, with no per diem or unnecessary travel expenses. Costs have to be low.
  • Use your expertise and contribute where you can.
  • As you take on more responsibility, make sure you know the local permitting requirements and laws. You have to understand that doing this work may be different than doing larger projects. Everyone will be looking at the people working just as closely as the people running the project.
  • Support design where you can, they need to walk sites, make drawings, give estimates, and so on.

How do we bridge the talent gap?

I would love to say training, but let’s face it, this is not always the way, regardless of what DC tells you.

When it comes to software and IT, the younger generation will be moving into that arena. The exception will be the people already doing it. 

Don’t worry about Silicon Valley moving into telecom, they would not take the huge pay cut.

No, it will come from colleges, internships, and the self-taught coders. It has to be something that can be learned quickly or adapted from an existing technology. 

If it is to be, it is up to me. So, look in the mirror and make it happen. You know you have options. Keep pushing ahead.

Definitions

  • AI means Artificial Intelligence.
  • BOM means Bill of Materials.
  • C&I means Commission and Integrate.
  • CAD means computer-aided drawing.
  • CM means Construction Manager.
  • DAS means Distributed Antenna System.
  • Edge means servers that are close to the end user. 
  • GC means General Contractor
  • IT means Information Technology.
  • ML means Machine Learning.
  • NOC means Network Operation Center.
  • PM means Project Manager.
  • PMP means Project Manager Professional certification.
  • RF means Radio frequency.
  • ROW means Right of Way as in a tower or pole is put in the right of way of a city street.

Reference Articles

Frenemies: Cablecos and MNOs

I hear a lot of talk about Cable companies, (Cablecos), competing with Mobile Network Operators, (MNOs). This makes the Cablecos Mobile Virtual Network Operators, (MVNOs). I think what most people in the industry know is that they are working together. It’s a partnership that makes them frenemies. But who needs whom?

The One-sided Love Affair:

Cablecos rely on the carriers’ networks for their mobile plans. Comcast and Charter rely on Verizon for their mobile network.

What I don’t see is where the MNOs need the Cablecos which makes it a one-sided love affair. Sure, Cablecos added a lot of mobile subscribers in a short period of time, but let’s face it, they’re not new subscribers, are they? We didn’t just see millions of people pop up without mobile service, did we? Of course not. They came from the MNOs. I would find it hard to believe that someone who had cable and internet one day realized they needed a smartphone. Possible, but not likely. 

People migrate to Cablecos for mobile service because it’s cheaper and they offer another bundle. That’s right, yet another bundle all in one bill. One thing Cablecos do well is marketing to their locked-in subscribers. Can you go a day without an email, commercial, or billboard pushing their mobile plans? Comcast has shoved their mobile plans down my throat. It appears to be working because they’ve added tons of mobile subs.  

Is it true growth? Subscriber penetration in the States is pretty high. I don’t know what else the carriers can do to increase subs for mobility. We see Cablecos stealing mobile customers, I’m sorry, shifting customers away from the MNOs. This may not be bad for the MNOs! Why? Because they aren’t losing so much as they are shifting and possibly gaining. They are losing some revenue when a customer goes with any MVNO, but it’s not all bad. I believe they have a strategy. 

Let’s look at Verizon. I would bet they lose customers to Comcast and Charter, meaning less revenue for Verizon but they still get money from their MVNO Cablecos. It’s not totally lost revenue, just a small decline when Cableco takes its margin away from Verizon. Verizon’s dream would be that Cablecos are stealing subs away from AT&T and T-Mobile. Lose 2 to gain 4, something like that. It’s a strategy that seems to be working. However, the MNOs still have tons of stores everywhere which means they didn’t learn a thing from the MVNOs. Some people love the stores, but they are a huge cost. That’s another article.

Now, we see Cablecos shifting customers away from the MNOs, hence the one-sided love affair. What can the MNOs do? Don’t cry or pout for the MNOs, they have a plan. 

It’s to be the frenemy of the MNO by stealing fixed customers away by creating something new, like fixed wireless access, FWA. This is a new market, more or less, for MNOs. 

Truth be told the MNOs offered this for years, but now they’re creating new services with more bandwidth than ever. They are finally able to compete with other fixed broadband ISPs, like Cablecos. 

Oh, a note to all of you smaller wireless ISPs in major markets, make a choice. One – you can retreat to the poorer neighborhoods or out to the poorer coverage areas. Wherever the MNOs will limit investment into their network. Two – you can lower your costs to $10 or $15 a month and hope for the best. Three – just wait until they come and steal your customers, making them work for it before you quit. I know you think your customers are loyal but don’t get hurt if they bail. 

Back to the point.

Why do you think MNOs are relying on FWA? Where else will growth come from? They intend to steal customers away from Cablecos. They also want to get the very mobile younger generation to literally cut the cord and use their FWA bricks instead of an attached cable box. The ones that want to cut the cord may be able to do it with a brick. 

Remember when I said the love affair was one-sided? This is what I meant. MNOs need to increase subs for this 5G network where they spent billions on spectrum, equipment, and deployment. 

Wait, but why?

OK, I mentioned mobile subs going to Cablecos. This makes sense because in spite of how they treated their customers over the last 10 years, the MNOs didn’t do much better. They have a lot in common in that respect, am I right?

Think about it, how often do you get your mobile or cable bill to see charges that outrage you? With MNOs, it’s usually overages. With Cablecos it’s usually some new fee or a subscription you thought was free. Either one can set you off when you get the bill. 

Don’t deny it, you know there was a time you went nuts looking at the bill!!

How many times have you called either one to complain and all they do is make you sign up for more crap? How many times do they say there’s nothing they can do, but they have something new or a new bundle that you don’t want but you sign up for it because you think it’s a better deal? I can’t believe it’s just me. I mean they can convince you that your plan is the best and cheapest bundle, which deep down, I feel they’re full of crap. 

My point is, Cablecos and MNOs have treated their customers with little respect in the past. Sure, the customer service rep is sweet and nice and oh-so-helpful but is really selling you on a new bundle. 

This is why we had a rise in MVNOs, (Mint and Visible), along with streaming choices, (Netflix and Amazon). They did what they said they were going to do.

How little respect did MNOs give their subs? So little that they are losing their subs to MVNOs that literally use the MNOs network. It may be cheaper and they won’t hit you with data usage overages. Not yet anyway. Most MVNOs have packages that make sense, not what MNO corporations sat around in a conference room for the last 6 months with a team of CFOs and Legal to build a contract that you will never read but hate all the same. You know I’m right about this!!

Now, imagine the same scenario, but with Cablecos and streaming service providers. Cablecos piss you off because of all the hidden fees. Do you think Amazon and Netflix passed them because they were a pain in the ass? No!! Spoiler alert, they treat their customers way better, have a lower barrier of pain to make changes, and they don’t screw you over when you call them for help. Also, they let you package things the way you want. I don’t have to sign up for an Alaskan sports network if I want to watch my local sports network for some freaking high fee. 

Sure, when you call the MNOs and Cablecos for customer service they’re nice, but then their directive is to get you to pay more whereas MVNOs and streaming providers are looking for ways to have flat fees and a simple shopping cart. That means letting you look online, compare, and then make your own decision. 

Simple versus complex. Plans made by normal people versus corporate boardrooms. Flat fees or complex contracts with crappy loopholes. 

OK, now that we understand why the ground has shifted and that the MNOs and Cablecos do own their respective networks, their playgrounds are not what they once were.

FYI- MNOs own Cablecos mobile networks, will this be a problem in the future?

Cableco Shift

Meanwhile, Cablecos are constantly improving their networks as they shift from being entertainment providers to Internet Service Providers, and ISPs. While you may think this is hurting them, I believe in the big picture it will offload so many costs and problems.

Did you ever lose a network because of some stupid argument over costs between a network and a provider? This used to be headline news when a network would go off a cable network or satellite network right before a big football game. 

Back to business. 

Cablecos are in a solid position to hold on to their customers for a very long time since they are wired to them. The local governments paved the way for these companies to have monopolies for years. Up until the last 10 years or so, no other companies could get in to build anything new in Cablecos territories. Comcast has a pretty solid hold in many areas.

However, now Cablecos have to constantly upgrade their bandwidth thanks to the competition. They are being pressured into competing with the competition just to hold on. And their new competition is their own Frenemy, the MNO. 

What about the wired competition? Unfortunately, today it is so expensive to run beside these companies. We all know that Verizon, AT&T, and Google all tried to compete as local ISPs, but it was so expensive to move into a market and hope to get a 30% share. Competition is almost as fierce as the legalities of moving in. 

  • Who owns the right to the poles? Yes, this is a thing. 
  • What agreements are in place with the local governments? More legal battles.
  • Can large Cableco lower their prices in one city to drive out the competition? Of course, they can.

But now you have the MNOs, offering wireless speeds in the gigabit range using FWA, and have become a competitor without all the legal hassles. Stealing customers like Casanova stole the ladies’ hearts. 

MNOs can offer internet access to anyone within coverage. Frenemies, competing yet they are also working together. I see this as very favorable to the MNOs. I guess something good came out of those boardrooms after all.

What can the Cablecos do?

I think they won’t admit it but they know they need to do something. 

I believe most MNO/MVNO agreements are a package deal, a huge package of minutes for a flat monthly/annual fee. Let’s keep it high-level here. Probably commitments on subs as well, connectivity, and cores. 

Cablecos have different motivations than normal MVNOs. Some MVNOs are built to be sold like Mint and Ultra. Some serve specific niches like Kajeet and Consumer Cellular. 

Like most MVNOs, Cablecos need profitability and growth to survive. Unlike most MVNOs, they have an existing subscriber base and don’t need the wireless sector to survive. 

However, Charter and Comcast are trying to build a sustainable business. This time they are relying on Verizon’s partnership to build their business and revenue. 

I believe their plan was to offload to Wi-Fi as much as possible. That makes sense because all of their customers offload to their home Wi-Fi. Both have hotspots all over their respective regions.  So, they will reduce MNO minutes by offloading to the nearest Wi-Fi hotspot.

Did I mention the CBRS PAL spectrum they bought? Charter and Comcast collectively spent over $900M for CBRS PAL licenses. I would assume to offload from Verizon’s network. So, what does that tell you? It tells me that their mobile business is taking off as an MVNO and Verizon is getting a huge piece of the pie. Big enough that Charter and Comcast are willing to pay a lot of money for “air” before investing anything in equipment. 

One more thing, Charter and Comcast won’t build national systems, they are simply building small cells in their respective regions. I would think they intend to build CBRS radios like their Wi-Fi radios, strand mount and in-building models like they have today. Why pay tower and rooftop rent if you don’t have to, let Verizon pay for that.

I also know that Charter and Comcast will work together to cut their costs to Verizon. Remember that Cablecos don’t compete with each other as MNOs do. So, I would think that when they do build, they will allow each other to roam onto their CBRS networks along with every other Cableco. 

However, I have been reading that Charter is in no hurry to build their network. I find this interesting because they literally spent over $450M on CBRS PAL spectrum. So they spent that much money and they’re in no hurry for payback. How crazy is that?

Could it be a bargaining chip to use with Verizon? Remember that Verizon is falling behind the competition in performance and coverage, specifically T-Mobile (TMO). TMO is spectrum rich and could be the #1 carrier very soon. Would that surprise anyone? TMO’s leadership has done a great job and continues to improve the network and expand the customer base while Verizon seems to be floundering, especially when it comes to bandwidth. These Cableco deals could help save them and additional CBRS spectrum couldn’t hurt. 

The other thought with Charter and Comcast using the PAL licenses could be for indoor private networks. They already have tons of small and medium business customers, the CBRS spectrum could sweeten the deal for special customer needs. They could offer Private 5G (P5G) services along with their Wi-Fi and mobile packages. (More packages!) 

BEAD Funding for Cablecos 

Some Cablecos wanted to score RDOF funding, which I believe Charter is getting some. Now I would imagine that they would get a huge chunk of BEAD funding. In fact, I would think that BEAD will go to the MNOs and Cablecos. Why? They have solid relationships with the state and local governments. The states will be the ones deciding how to distribute the money. 

Whatever you think about that, I believe that they have a shot of reaching a lot of unconnected people. It’s all about what the upfront investment could be. Does it make sense in these areas? There’s a cost to maintain the network which may stop them from building in remote areas, unless the payback makes sense. Even if it has “free” funding up front. 

The funding by state can be found here and a random sample is listed below.

  • Texas $3,312,616,455.45
  • California $1,864,136,508.93
  • Missouri $1,736,302,708.39
  • Michigan $1,559,362,479.29
  • North Carolina $1,532,999,481.15
  • Virginia $1,481,489,572.87
  • Alabama $1,401,221,901.77
  • Louisiana $1,355,554,552.94
  • Georgia $1,307,214,371.30
  • Pennsylvania $1,161,778,272.41

Yes, it is a lot of money flowing from the feds to the states to ISPs. Can we get in on that? I hope so! I think we’re all counting on doing a piece of the ISP deployment.  

One more thing, the equipment has to be American-made. That’s why Nokia is expanding manufacturing in Wisconsin. Eat your heart out Ericsson and Samsung!! Oh, Wisconsin is also getting just over $1B dollars of BEAD. Sweet!!

To be honest, who knows how much money the Cablecos will get, but if they take it, they will have to build in remote areas. I know CenturyLink has done a good job of this, but we’ll have to wait and see if the rest do well. I wonder if Frontier will be a player. I guess we’ll see. 

More Links:

Shifting 5G Deployment Optimism

As many of you have seen, there have been a lot of cutbacks in the American 5G upgrades.

I don’t want anyone to panic because we’ve been predicting this for about a year now. When it actually happens, it hurts because people lose jobs. The climbers are the first to see cuts, along with the OEMs, material providers, suppliers, warehouses, and so on. This trickle-down effect is very real. 

Unfortunately, this means the trickle-up effect will also take place. Why?

As you saw above, Ericsson is changing their model from employee to contractor. This is one move. Towercos, like Crown Castle and American Tower, are rolling back their construction services to concentrate on leasing which they’re good at and it makes them a lot of money. Let me tell you, construction is generally high risk and low margin. Towercos’ profit percentage will increase even though revenue may drop by losing construction. Take my word for it, this is a good problem to have.

What it means is that first, the construction teams slowly get reduced, in this case, climbers and the foreman. Then the construction managers, then the project managers, and then the support teams. Not necessarily in that order. 

Don’t fret though, many of you can find work among the new services and in other branches of wireless. Keep in mind 5G added a lot of work in this industry, (wireless industry), but it’s not the only thing. Public safety is upgrading, some broadcast work is happening, and the removal of 3G and 4G will happen. There are still network expansions for the next 2 to 3 years along with all the sites that had long delays. 

There will be a ton of outside fiber work coming up. That is a completely different skill set, but all the same, wire carriers will need to remove and replace all the lead cables they laid for the last 50 years or so. Surprisingly it’s a lot of cable. This is something that can’t be overlooked. It is very labor-intensive and costs a lot of money. The labor will be very local I believe.

Don’t forget venue upgrades and DAS expansions. While this is a different skill set, you should be able to adapt quickly. Just keep in mind it will be very local meaning no per diem for a lot of that work.

Hey, sometimes change sucks, but the industry is cyclical. This is my 4th rodeo. 

Now, we have to look to the future. For some of you, it may be a new company or maybe a new career. For others, this may be your industry, moving from carrier work to public safety tower upgrades, or microwave.

Don’t stress out too bad, it’s not all going to end. The migration to 5G has stages. I listed some of them below. If you partnered with the right company, then you should be working for the next 2 years. 

Keep in mind that the nature of the work is changing. Now the carriers have to prepare for changes and upgrades in the network. Yes, the tower work is waning but the rise of software and networking updates will continue. 

Migration away from 3G and 4G

That means that 3G has been shutting down with equipment being removed aggressively. There will be more, but how much? When? 

Looking ahead, maybe 2 years down the road, shuttering 4G will happen much faster and probably with force. What does force mean? It means that carriers will work hard to remove older technologies from their networks. They already had a serious 5G push. Getting 5G devices into as many hands as possible. 

To be clear, I am not sure how much 3G is still out there. I do know it’s still supporting some services, mostly IoT and automotive. It should have been shut down in 2022, with Verizon holding on the longest, per this article. I am not sure if all were actually completely turned off. Most carriers tried to remove what they could when they did the 5G upgrades. I think there were some lawsuits that made some carriers hold on longer than they wanted to. 

Shutdown dates per David’s blog:

  • AT&T 3G GSM/UMTS – February 22, 2022
  • Sprint 3G CDMA – March 31, 2022
  • Sprint 4G LTE – June 30, 2022
  • T-Mobile 3G GSM/UMTS – July 1, 2022
  • Verizon 3G CDMA – December 31, 2022

Unfortunately, any 3G removals that didn’t already happen, won’t happen for a while unless tied to a 5G upgrade. Remember, they won’t remove it unless they can save money. That means unless there is a reason, like saving rent, they will do it when they have other work at the site.

Unfortunately for all of you climbers, I would think the 4G shutdown would not involve much tower work. I think it’s going to be a software change to reallocate spectrum, for the most part. Don’t get me wrong, some carriers will have to replace radios, but I would like to think that most of them were smart enough to do it during the 5G upgrade. 

I have been wrong before, but who would be stupid enough to go to the site to upgrade to 5G radios and not plan a migration from the existing 4G to 5G? New spectrum and radio consolidation made all of that possible. 

You have to love software, BBUs, and progressing technology that can be remotely upgraded.

Backhaul and Network Changes

Backhaul upgrades are happening. They’ve been happening, I mean the spectrum changes aren’t the only reason you’re getting 1Gbps throughput at a site. The backhaul had to be upgraded to increase your throughput. Cool, right?

However, as we expand coverage using small cell and mini macro sites along with the increasing FWA (Fixed Wireless Access) offerings, carriers need to get creative. This means that we have to connect more sites with high-speed backhaul. The fringe sites are no longer fringe, but key to supplying homes with internet access. 

While many of you think fiber is readily available everywhere, sniff the reality! Fiber is not available anywhere and lead times in common places can be excruciatingly long for carriers. There are many places where fiber upgrades can’t happen or have been delayed for such a long time. Then there are the crazy expensive installations. So, what can we do?

Well, we can wait. That seems to be a common theme all across America. Wait for it and it will come. Wishful thinking? I mean, the BEAD funding is happening for a reason, right?

For those of you relying on LEO birds to save the day, it’s usually a workaround, not an end solution. Even LEO has a very high latency for a carrier’s site that expects less than 10ms per hop.

Let’s add microwave where we can if fiber is near. As I said in other articles microwave is a suitable replacement for fiber, at least for a little while. It’s cheaper over time but expensive upfront. Rent usually isn’t a big deal if the carrier has other equipment at the site, it’s usually included in the original rent models and not adding additional OpEx. It is a reasonable alternative and backup to fiber. So once it’s in they can keep it in just in case fiber drops. Yes, fiber still gets cut. 

We will be adding more and more fiber. I am hoping fiber providers have thought about where we would be 5 to 10 years out. Fiber has been here for a very long time. Most carriers have extra strands at every site just in case they need to add capacity quickly.

However, that isn’t the only thing. The network equipment has to be able to handle the additional services and bandwidth. While we all think it just happens, there are teams of IT people making sure the upgrades happen in the background. They need to be sure the hardware can handle it and that the routers and switches are robust enough to handle the additional traffic. That requires planning, hardware upgrades, and software upgrades. 

This is ongoing work. It’s not something that is one-and-done. While it won’t have the economic impact 5G upgrades did, it should be steady work for the next 5 years or so. Probably longer.

If the FWA models take off, then microwave or multipoint are great solutions for MDUs, (Multi-Dwelling Units), like apartment buildings, condos, row homes, and so on. 

Data Center Upgrades

Who loves the cloud? I do!!

Who loves the edge? I do!!

Who loves edge services and cloud services? I do!!

Low latency baby! Offloading to the max. Major application performance improvements. Who doesn’t love that?

Yeah, baby! (Picture Austin Powers).

In fact, I bet we all love edge and cloud services. I know the networks of today rely heavily on the cloud and the edge is becoming more important to all of us. 

Most apps you use run in the cloud today. That’s why they’re so quick and can have awesome graphics without loading up your phone or laptop with tons of data. 

That’s why Chromebooks are so popular. Offloading to the cloud has changed data storage and processor usage in your laptop forever. Awesome! 

Long live the cloud. Wait, here comes the edge to bring it even closer to me. Nice!

So, what this means is that smaller data centers are popping up all over. Tower companies like Crown Castle and American Tower have been investing in tower companies over the past few years. They saw that it would be an integral part of the communication service provider (CSP) model for years to come. 

There is a reason why. They looked at AWS and Google Cloud to see the massive growth. Microsoft joined in and even IBM has a viable cloud solution. 

Data centers tie all of them together. They are the new telco colocation sites. Remember the days of CLECs? (Competitive Local Exchange Carriers) That is the same concept, only now they not only share space in the data center, but they can share space in the same machine!! How crazy is this?

Let me tell you one more thing, there is no sign of this slowing down. We have had a lot of logistics issues along with the fear of China attacking Taiwan within the next 5 years which caused a lot of disruption. All the more reason we have to upgrade as fast as we can. 

I believe that we are in the infancy of data center growth. Why? AI and Nvidia are changing so much. I thought it would be Quantum computing. Now it seems AI will incorporate Quantum computing into its web of processing tools. 

Whether you’ve read about ChatGPT or not you already know it’s a game-changer. It’s hard not to hear about it with all the press and Terminator movies out there. 

Fun fact – the GPT in Chat GPT stands for “Generative Pre-trained Transformer”. Congratulations, now you know. If you want to know more, it’s another article that I started here but would like to do a deep dive. 

My point in this section is that we need to expect data center growth. 

For your information, I had stock in CoreSites before American Tower bought them and took them private. I don’t fully understand why they took them private instead of rolling them into the tower business, but they’re smart people and I am sure they had good reasons. I am a believer in data center growth. 

Not just the large data centers, but I see a world where we have smaller data centers wherever tower sites and small communities are. Even spread all across sites. 

I also see lower power solutions along with air cooling solutions. We can’t keep using inefficient air conditioners at these sites. We have to get way more efficient to cool these data ovens. Talk about global warming! I feel the same way about tower sites. 

We need better cooling solutions, like Nokia water-cooled equipment. Amazing advances are sometimes doing what other industries, like Broadcast, did years ago. Again, another article but I did touch on it in this article

Power – Back up and more

Power is a huge issue. Not just for upgrades at the site, but sites are demanding more and more power.

In this section, I don’t just mean wireless sites, but data center sites as well. They are all sucking more power than ever. So much electricity is being used as demands become greater and we’re adding more and more equipment. While today’s equipment is more efficient than ever, so is the demand. It takes more power as each user may be on their phone and laptop at the same time with maybe a dozen or more apps running simultaneously. Now, imagine all that from a work-at-home family using work and personal devices simultaneously. 

Demand requires power!

I am a fan of using nuclear power, especially with all the advances. Very small nuclear plants should be built across the USA as soon as possible to meet the demand required. The new nuclear power plants are safer and way more efficient than anything we have out there today. The reality is that renewable power needs so many batteries that it’s unreliable as a large-scale solution. Luckily in America, we have so much natural gas and that is the major power source for electricity. Thanks to all of America’s fracking companies!! 

One thing the carriers are doing moving forward is to upgrade backup power. They realize it’s more important now than ever!

This was not necessarily done when the radio upgrades were done because of cost and skill set. Utility upgrades take longer than fiber upgrades. It’s expensive and time-consuming. It may require lease changes along with coordination between the utility company and landlords.

It can be a huge cost upfront, but in the long run, it will probably pay for itself. Just ask anyone whose site has dropped because they could not get more fuel to a site. It sucks! The KPIs for uptime are critical in the carrier world, especially in disasters. 

OpEx will increase because now you have testing, maintenance, and monitoring costs. Maintenance does require a new skill set to visit the sites. Think about it, you have to do an inspection, replace the oil, and maybe lube the generator like you do your car annually. Luckily, your NOC or existing system should be able to remotely test and monitor for alarms.  

Why do we need to upgrade backup power? Hurricanes and high heat are serious problems now. Fires rage out of control all across North America. We need to be prepared.

Remember that carriers don’t only provide mobile communications but home internet as well. Carriers have a heavy burden and the expectation of having a very reliable system during a crisis matters. Not just to one carrier, but to all of them. Remember that FirstNet rides on AT&T’s network but all carriers have contracts with public safety teams. Reliability in disaster matters more than ever.

Now broadband and mobile are considered a utility, not a luxury, especially in disasters. People need to stay in touch, although most people just want to be able to play games and get email since they can’t go anywhere or do anything with all the stores closed. It’s more about the data and less about actually making a call.

Don’t try to tell them to read a book, it won’t work. 

However, it’s not just the cell site that has to stay up. It’s the data centers, cloud and edge servers, and all the fiber connections in between. Now we expect the entire network to stay up and work properly during and after disasters. That is what makes the USA great, even in disasters, we continue to maintain communication for a very long time. That’s why this country is so great.

Thank you, on-call workers, everywhere!!

Thank you, disaster relief workers!

Thank you, public safety workers! 

Oh, also, we have to do upgrades at the sites to replace all those nasty lead-acid batteries. The new Lithium-Ion batteries are so much more efficient, smaller, and last much longer. Did I mention they can last over 10 years? Way longer than lead acid last. Remember to dispose of batteries responsibly by recycling them properly with a death certificate. There is a process and the environment matters, at least in America.

One more change is taking place here. New generators and if you have the money, go with renewable power to assist power generation. It probably won’t replace anything you have today but it can help lower bills.

Renewable power is a pain in the ass. Why? Let me list the ways!

  • Solar and wind take a larger footprint and need special permitting.
  • You need batteries to store the power.
  • You need the proper transfer and backup systems just for renewables.
  • You need a renewable power expert.
  • So many things can affect power storage, lack of sun or wind, and bad weather. 
  • Renewable power is generally worthless in bad weather and in disasters. In fact, chances are good you will spend a fortune to repair it after a store. 

OK, let’s move on.

New Applications

There will be so many new applications on smartphones for 5G. I wanted to write more on this but it’s so new I don’t have many examples. I think this is something we all expect but haven’t really seen yet. After all, broadband is broadband. 

Network Operation Centers (NOCs)

I am speaking of network monitoring and support here, not data centers. 

You may not realize this, but with all of China’s spying allegations, NOCs are moving back to the US for security reasons. Not all of them and not all services, but it appears that onshore is the only perceived safe shore. I say perceived because we all have opinions, I’ll let you answer your own questions about security.

NOC services will be ramping up stateside. We expect to see NOC growth inside America with 5G growth.

Most people that read this may not like to hear that, but it’s a reality that more and more services will be migrating back to the USA.

What else?

Well, as you know, a lot of the players that were in the business are now getting out of the business. 

If you’re in the fiber business, I mentioned that lead cabling will be replaced or removed. Good news. If not in 2024, then definitely in 2025. The carriers may wait for a few lawsuits before they realize they signed all those ESG agreements.  

Fun fact, ESG means using Environmental, Social and Governance factors to assess the sustainability of companies and countries. These three factors are seen as best embodying the three major challenges facing corporations and wider society, now encompassing climate change, human rights and adherence to laws. Thank you, Robeco!

Let me tell you what that means to you if you’re part of a big company. Your big company sank a lot of money into new policies and a new department that will focus on these issues and force you to sit through endless explanations on why it’s so critical to businesses going forward. Oh, that and the Western governments put a lot of pressure on large corporations to follow ESG policies. In the US the SEC requires all companies to report their ESG policies to their investors. That means as an employee and investor you’re going to hear how wonderful ESG policies are and what it means to that specific company.

Are all policies the same? Not really, hence the transparency. 

Per Robeco, “The basis of ESG comes from the United Nations World Commission on Environment and Development – known as the Brundtland Commission – which is most notable for coining the term ‘sustainable development’. This was defined as “development that meets the needs of the present without compromising the ability of future generations to meet their own needs.”

Tying this into corporate activities later led to a concept of the ‘three Ps’ – People, Planet, Profit – gaining traction in the 1990s. This argued that a focus on each of these three words (and not just profit) was equally important for any commercial enterprise to be sustainable. This morphed into a more specific focus on environmental (planet), social (people) and governance (profit) factors.”

Right now, in the 2020s, this seems like something all corporations worldwide are really into, especially if they’re public. 

What does it mean to me and you? Well, to me it means I have to sit through their presentations every quarter and gloss over it in every company update.

To you, it probably won’t mean much unless you are part of an ESG solution. 

To each corporation, they define it in their own unique way but basically, it’s a form of social responsibility that each company commits to. 

I think it’s a good thing, but I don’t worry about it as much as I do the network. I want to build networks, that’s the dream, right? I want to do it responsibly.

What can we do today? Moving ahead, let’s get through this together. Let’s help each other out. This industry is a community. We need to work together.

If someone needs a job, help them if you can. 

If someone is getting laid off, be kind.

If someone is forced to start a new career, do what you can to teach them. 

We’re all in this together and trust me, this is a small industry. Show some love.

Pay it forward if you can. 

Private Network Integration Notes

As we move along the wireless network’s roadmap, we see that carrier deployments have slowed down. This means that there will be more focus on private networks. Not only because the OEMs will be looking for new revenue opportunities, but because they can move their talent pools over to focus on smaller networks. That and 4G have matured while 5G is maturing.

On that note, keep in mind that private networks are a different beast, mainly because they are generally built for a specific need. That is unlike the carrier’s network which is built for the masses. There’s a good chance a private network won’t be for broadband but for data-only IoT or as a gateway for other networks to ride on like a backhaul.  

Also, 5G may not be an option, at least not in 2023. The thing is, not many devices for IoT are 5G capable, some are still 4G, LoRa, BLE, or maybe Wi-Fi. The private network may be put in for convenience, cost, or security. It doesn’t matter if it’s 4G or 5G, just so long as it serves the designated use case(s). So, the G doesn’t matter as much as the gateway. 

Also, think about the options you can offer, realistically. There should be a roadmap that ties the use case(s) with the technology. Then a timeline. Set expectations. Don’t over-promise something like Push-to-Talk (PTT) because they are working in 4G but are new to 5G. Be realistic.

I still see the push-pull between deploying DAS or small cells. That’s still an issue that depends on who implements the site. Will it be someone more comfortable working on a DAS system? Usually, if that’s the case the network will be public & private. If they lean towards small cells chances are good it will be private only.

Just so you know, I see a lot of DAS deployments when the carriers are involved. It’s what they know. It’s what they’re comfortable with. They don’t want to change their model unless forced to.

One more thing to keep in mind. These private networks can be very local. When you win one, it is normally in a specific city, Campus, or building. I believe that having a local installation and integration partner is the best. They can help keep costs down and build an ongoing relationship with the customer.

What do I know about partnerships?

Here is one thing I do know, partnerships matter. Carriers usually don’t want to take this on themselves because they want someone local and cost-effective who is experienced in DAS and/or private networks. As do OEMs that are used to working with carriers.

The local partner should be someone who knows what they’re doing. Sure, there may be a learning curve, but the carriers and OEMs want to work with someone they trust and will stick it out past the deployment into support. That is why I think the partner should be identified upfront and not treated as a contractor. They work better when they have skin in the game. By giving their name to the customer on day one, both reputations are on the line.

Another thing I know is that so many private networks that have been built so far are not living up to their full potential because of the way they were built. The teams built it hoping more customers would come. This is not the best plan. It would be good to have a use case with a roadmap that leads to adding use cases to the existing network. Groups that build this way can have a long-term sustainable network that could grow.

Finally, I see the carriers are increasing their interest in private networks. In fact, it’s becoming a priority in the larger carriers in the US. The budgets are moving away from the builds and expanding and now moving into the private network where the carrier can offload to a partner, share the cost of the build, and have a public/private network with their spectrum and a slice for the end user to use. Suddenly they’ve locked in customers for a very long time and will be expanding their footprint inside a venue. 

Create “the” Plan.

When working with the customer, have all involved to explain each team’s role and what the timeline for deployment will be. Be realistic. If you’ve gotten this far, then you should have already worked out a roadmap for services and features as well as devices.

Partners are key! My plan would be to have the partnerships become the new normal. We do it for DAS networks and we should do it for private networks. A local partner who knows the business would be of great value before, during, and after the deployment.

These partnerships could be with OEMs, integrators, carriers, and customers. This is a team effort and if everyone’s name is on the end product, then everyone will be responsible for the network and the growth of said network. 

Carriers and OEMs will need to work with integration partners. Do not treat them as vendors, but as partners that know what they’re doing and will put their names on the network. I see OEMs that think it has to have their brand all over it and then they bleed money because they have to call the integrator back in to resolve many issues that may have been overlooked.

Integrators will generally stick with the customer. They also have the patience to sit with a customer that wants to cover a warehouse, say maybe 10 radios, whereas an OEM may scoff at such a small project. They won’t want to spend 6 weeks closing this deal when the integrator will have the patience to see the bigger picture. 

Local teams build relationships and are always looking for new business. It makes sense to partner with them and be there when they call on you. Think of the days when Motorola had so many smaller local shops pushing their public safety networks. That was a great sales model back in the day. Sadly, Motorola let that model fall apart. While they still have partners, things just declined. Oh the good old days.

Cores are hard!

I think today’s choices of cores are wonderful. So many people say they want to do it on their own. My GOD that is a lot of work and money right up front.

The only reason the core decision today is hard is because you have so many choices. On the high end, you could choose to build one, Nokia or Cisco, but why? Doesn’t it make sense that the customer has something simple and cost-effective? Something they won’t have to maintain themselves. They don’t understand the core and will rely on the integrator to recommend something that makes financial sense along with being super reliable.

I think that Nokia’s NDAC is a great way for someone to get started. But, you also have Microsoft, Amazon, and Google that have options for a core. Why not start there and then grow as you need to? Decide what your use case requires, shop features, price, and migration details. That’s when you can move ahead.

When picking a core, understand that the OpEx is what is going to drag you down. Make sure the end user understands the ongoing costs of maintaining a core and all the devices attached to it. This is where so many OEMs and carriers miss the mark. They tend to gloss over the ongoing costs, which, for the small business it will be a big part of their IT budget. What OEMs often forget is that these small businesses IT budget could be a part-time kid that works for low wages, (high school or college kid), not an actual IT professional. I am not being a jerk, just stating that this is how many businesses operate and grow. 

Build a roadmap sooner than later;

You often hear the term, “Build it and they will come”. Who will come? Why will they come? For what? When?

I think it’s best to lay out a high-level roadmap upfront. This won’t limit you, but it can help you expect realistic expectations. Be honest about moving ahead.

I don’t mean saying, think of what you can do, you can do this or that. No, what I mean is to be realistic, then identify where they are today, and then what they might want to add in the near future. Then, show them the timeline of when it will be available. You could do so much but don’t sell them what they will never want or need. Remember, features cost money. Build them a roadmap and timeline.

Most private networks are for data-only IoT, but voice may be an option. I am not sure if 5G PTT is necessarily ready for private networks, but it may be on the roadmap. They also have apps you could drop on a device. Understand the larger picture but keep focused on what they need today.

Don’t stop there, explain the cost impact of each service they may want. It may cost more to simply add a feature or maybe a new radio is required. Not just CapEx, but also OpEx.

How do all the networks tie together? What data can be collected? How will it be collected? What is the process for these new analytics?

The increase in OpEx should be explained. As you add services, radios, applications, or additional data collection, it may add costs. 

Data collection is often overlooked. This should be one of the key components for small manufacturing or shipping businesses. It should be looked at and built upfront. Everyone likes more data to track in real-time what’s going on in the business. This is a feature that’s often undersold. Data is a key factor of having your own network. It is something the customer understands and might be using today. Data has value!!

Create a partnership roadmap!

What I mean by this is to work with the team, including the customer and end-user, to define where you expect to be in a year, 3 years, and 5 years. Not just the network, but how you all can be working together in the future. Defining this isn’t like adding Wi-Fi where a year later you could be buying from someone completely different. For something like this, you would expect a long-term partnership. That would include ongoing maintenance, upgrades, and support. That should be considered in the customer’s OpEx budget you create for them.

The integrator may be facing the customer for the next 5 years. They will be maintaining a constant dialogue with the customer over the maintenance and upgrades. They may even support data collection. 

For those of you that may have worked in public safety years ago, this is what it is like. You build out the radio systems and maybe the e911 system. Then you support the customer for years to come. They rely on you and to be honest, you make some great friends in the process.

Unfortunately, the carrier side is like that to a point, but not like solid partners. They have taken a lot of the friendship out of it in exchange for saving a few dollars.

So, what I would recommend is to build a roadmap to move ahead with the network and the relationship. A relationship roadmap if you will. This will help set the expectations for the next 5 to 10 years beyond integration. 

Carrier Network Updates

I have been asked a lot about the recent work freezes in the carrier networks. While there are a ton of conspiracy theories out there, I thought I would drop in a few words of common sense to help ease the pain of all of you that are seeing a slowdown in work.

There are two main things carriers are looking at moving ahead. Obviously, they want to add revenue and cut costs. How will they do that? Break into new markets and look for network efficiencies.

One is to add new subscribers since it appears that the mobile market is near capacity. So, the next market is the fixed wireless access market, (FWA). They might as well cut into the ISP market. (ISP = Internet Service Provider for all you newbies).

They also want to cut network costs. The obvious winner is energy savings. This will be more complicated but there are several things that the carriers are looking at. 

Many people are questioning the way the industry is moving. To me, history repeats itself and it’s almost as if everyone complaining has never seen this before. The industry moves in cycles. I didn’t think you had to be as close to 59 years old as I am. (I’ll be 59 this year so excuse me for being a grumpy old man!!)

I’m just saying, open your eyes, and look at the progression of 3G to 4G to 5G. The cycle repeats itself. I just don’t know how 6G is going to happen with the spectrum pool so depleted. 

One thing that slowed this cycle down was the government’s lack of operating properly. I don’t blame the FCC, in my opinion, they were held back by the Senate and Congress by not adding another commissioner until now and tying their hands in the spectrum auctions. The US government seems to be incompetent at times, getting in its own way. Again, just my observation. 

This is bad news for many in the industry. It is very hard to weather a storm that may last for a few years. 

Let’s look at what is being done and address these issues as we move forward, shall we?

Let the network pay for itself!

The beauty of building a network is that at some point it will pay you back. The carriers would like to stop spending billions so that they can start making money off the network.

Let’s face it, they were able to build these networks and still turn a profit. At least T-Mobile was able to do that. I don’t know what’s going on at Verizon.

Anyway, the network now can run, growth can slow, and the income will grow slightly over the next 2 to 3 years. I’m not sure if you figured it out but that should mean very high revenue and expenses will be reduced in a big way.

Show me the money!

Well, here it is. Same plans, new FWA users, increased network efficiency, and profits increase. Sounds like a win!

Is 5G a Failure for the carriers?

Honestly, I don’t think so. For my argument, look at the ecosystem. 

The OEMs obviously made out very well. They are selling a lot of radios, upgrading networks and cores. 

Backhaul providers, fiber providers, and cabling providers have been killing it over the past 5 years.

Tower companies have increased revenue that they can live on for the next 20 years. While the major expansion may be over they will continue to add more revenue and the systems fill in for the next 5 years.

Tower climbers had a boon over the last 5 years. Unfortunately, construction is going to slow down. This hits hard and quickly. Sorry, but some will continue to work while many will be looking for a new job very soon.

New devices are everywhere, new iPhones, new Androids, and new IoT devices. Look how the 5G models have taken off. If anything, IoT devices have fallen behind, but only because tech moved so fast that they’re not getting back their 4G investments in time to pay for the 5G investment. It is hard when you sell a device for under $100 to keep updating the chipsets and then testing the new model. So many SKUs for a low-cost device.

What you may not see is the new features beyond spectrum and technology. For instance, software on the network is constantly being upgraded. We’re moving from NSA (non-stand-alone) to SA (Stand-alone) 5G networks. This will allow new features, such as network slicing, to take place all the way to the end user. We also have new features, new data analytics, and new apps along with the increase in data throughput.

The new doors are opening and showing us new network capabilities.

So I don’t think the carriers are losing anything in technology. However, I do think that the delays in releasing spectrum are hurting the carriers. T-Mobile planned well but had a lot of luck on their side. They were able to purchase Sprint and all that beautiful 2.5GHz spectrum. They got 100MHz in most markets, which today is crushing AT&T and Veriizon’s C Band spectrum. The 600MHz spectrum helped them grow the rural area, but it is definitely not the beachfront property the FCC touted it to be at the time, far from it. Band 41, now that’s beachfront property!!

People are already talking about 6G, which has not been defined yet, but will be in 2027. The great thing about being with an OEM is that we’ll be testing 6G wannabe models up until then and after that. For the OEMs, that is good news, to see what’s possible and what chips will take us into that era. 

Until then, the carriers will start to cut expenses and increase sales where they can. How will they do that? I made a few points below.

FWA = Fixed Wireless Access

We already see this moving ahead. Luckily the carriers are already offering this. As 5G matures and Stand Alone 5G (SA) becomes the new normal, this will be utilizing the network the best way it can be run efficiently.

We’ve all seen the emails and flyers from Verizon and T-Mobile for FWA in the home. This is the new market the carriers are breaking into. It seems to be the fastest-growing segment for the carriers. The great thing is they’re not stealing customers from each other but from the cable companies. That means that many of these customers are net-new greenfield customers. A new market for carriers. 

Don’t get me wrong, I understand that the FWA home user has different requirements than most mobile users. I also understand that in most homes and businesses the local ISP offloads onto Wi-Fi inside the building. Offloading to Wi-Fi actually is an easy way for any ISP to distribute the connectivity throughout the insides without much hassle. The 5G coverage may not penetrate into hard-to-reach areas whereas Wi-Fi in the building is much better.

What the carriers have done is cut the cord outside. They also could allow you to take the hotspot with you, assuming it works in the new location. Cable companies do this with their boxes, but I’ll tell you from experience that it’s a pain in and never works right on the first try.

The data usage of home users is where you would think mobile carriers would have an issue. Luckily, most 5G spectrum is new and offers huge throughput compared to networks just 5 years ago. Now they can offer over 100 Mbps up to 1 Gbps. Very competitive with what most ISPs offer with a wire or fiber to the home. 

Unfortunately, many mobile carriers try to cap that data. That will need to stop. I know they probably make a lot of money by charging overages, or at least they will until enough people say enough is enough and leave them to return to Comcast or Charter. 

You can’t screw them if you want to keep a happy customer. I already have neighbors complaining about AT&T charging them overages for their home network. We had to limit their TV’s connectivity. Even then they were so mad at AT&T, not the TV. So, they may take all of their devices over to someone new, it’s not that hard anymore. They told me they went to AT&T because Comcast was charging a lot of money and their customer service was poor. At least AT&T’s customer service was better, but the surprise charges were not good. They may try T-Mobile next. 

However, from a network perspective, suddenly you have sites that may have had passing mobile users on them offering FWA customers full-time, daily, and in off-peak hours. This makes the network much more efficient than it was before. Each fringe site becomes a profit center.

You have to be sure each site can handle the additional traffic at sustained data rates. 

FWA makes each macro site look sexy!!

Think about it, all those sites that were built for the home user or the occasional mobile drive-by are suddenly someone’s lifeline for internet access for their home network. They will rely on it to watch TV, surf the internet, and maybe even work from home like me!

This makes all those fringe sites even more valuable, bringing in more revenue than ever before. They are no longer a coverage requirement but a new profit center. 

Cutting Energy Costs

While this has been a point of contention for the carriers, albeit a quiet one, it is one that now is being addressed. 

The cost per bit has decreased, but the bits used on a site have increased exponentially. So that means that radios and networks have become more and more efficient, and power consumption has increased. Heat creation has increased. Electricity costs more than ever so energy costs have had a double whammy. 

Site expenses just went way up! A big hit on the old OpEx budget on something pretty much out of their control!

Or is it?

The OEMs heard this and listened. There are several new ideas that have come about in the industry, not just for radios but also for networking equipment. 

First, why are utility costs so high at a site? Hint, it’s not the heating bill, it’s the cooling bill.

There are several things that cause problems. Let’s look at a few of them here.

Radios get hot, but generally, they are atop the tower, so they really don’t count. The OEMs did a great job making sure these units take extreme heat. While there are exceptions, looking at Arizona and Nevada here, most radios can take extreme heat. When they need to avoid the sun, OEMs added a sun shield to minimize solar heating.

The radios are more energy efficient than ever, unfortunately, they are also putting out more data, spectrum, and power than ever before. The energy savings were negated by new performance requirements. 

The answer is to reduce run time if possible. Many sites serve coverage or loading purposes. In an urban environment, the radios might only be fully utilized throughout the day or during rush hours. They may have downtime at night or in the wee hours of the morning. This used to not matter. However, as I said, the OEMs were thinking about this.

Now they have ways of turning down power and shutting radios off at off-peak times. While this could be a manual thing, then you have to pay someone to do. That defeats the purpose of cutting costs.

So, the equipment will learn when its slow times are and cut back on radios, maybe a sector, and lower power automatically. Don’t get me wrong, this may cause an issue, but until the complaints come in, let’s cut utility costs, shall we?

With this feature, the network is trying to save costs automatically without human intervention. The carriers should think this is a win-win, but who knows what some of them are thinking.

Next, the networking equipment in the shelter draws a lot of power and puts off a lot of heat. 

This is getting better. Most equipment now is air-cooled, removing those fans that used to fail all the time. They can take extreme heat, but anyone who runs a data center knows that Air Conditioning is required and it becomes the biggest load on electricity. The AC unit also requires a lot of maintenance and it seems like they fail a lot. Especially when they’re running almost 24/7 in the summer. 

If you’ve ever done fieldwork, then you know how many times you went to a site for overheating. I’ve had many calls like this back in the day and I had to learn to repair air conditioning units and call for repair. Maintenance of these units was critical. Not just filter replacement, but monitoring the temperature trends to see if it was losing cooling power.

It’s labor-intensive just to maintain these things.

Well, OEMs are now coming out with liquid-cooled units. While this is new in the data industry, if any of you have worked on broadcast transmitters, then this is actually old-school.

What you have is liquid, maybe water with antifreeze in it, that pumps liquid through the electronics heat sinks and then out to an external cooling unit. Imagine the cooling unit is outside and looks like a large radiator with a fan connected to it. It should have multiple fans to bypass a point of failure, but many still only have one fan to save costs. All the same, this greatly reduces AC cooling costs because the heat is quickly transferred to the outside of the cabinet or shelter or data center, wherever the networking and BBU equipment is located. 

How cool is this? While it seems like a breakthrough in data, broadcast figured this out decades ago, back when I did broadcast work as a young buck.

The difference was that broadcast back then used kilowatts, that’s right, thousands of watts to broadcast to a region. If you made a mistake, it would kill you, literally. 

Today, low power and you’re worried about the lasting effects. Aside from heart attacks and tower accidents, not many people die at a site, especially from electrocution or radiation.

It’s a brilliant idea that is being released today. Unfortunately, a lot of these ideas are coming out in the middle of the 5G build, but all the same, at least we have options moving forward.

Now, for all of you people crying that we should use sustainable power. That sounds good, but the reality is it’s not practical in most of the United States. Where it is practical, there are still limitations.

In the US where backup power is required everywhere, you don’t see many batteries tied to solar panels for long-term backup. We have generators everywhere. We’re still adding generators everywhere. We love generators in this industry. Do you know why? Because 98% of the time they work. If they don’t work it’s because they failed or someone didn’t maintain it properly. 

A generator will guarantee us days of run time, not hours.

I don’t know of any solar panels that can guarantee me days of unbroken run time, do you?

While solar is popular, it has limitations. You need space, enough to add panels for power collection. 

If you put them on the tower, you threaten the structural integrity of the tower which already holds hundreds of pounds of mounts, clamps, antennas, lines, and radios. Oh, did I mention it has to survive hurricane-force winds with all that crap on it? Well, it does!

In la-la land this sounds great, just use solar and wind and everything will be OK. In the real world, those solutions suck for so many reasons. Yet, we are trying to make them work.

One more thing, it takes a ton of batteries to have that lasts for over 4 hours. Batteries, even the amazing Lithium-ion batteries take up space and are heavy. They also have corrosive chemicals in them and come with a lot of risks. There are also a lot of ethical issues with the sources of Lithium and Cobalt. Make sure the batteries are sourced correctly, preferably not from a Chinese company that will not be truthful about the sourcing.

Summary

At least the industry sees these problems and is addressing them.

New business in the form of fixed broadband opens up a new market. Sure, there are a lot of critics, but what is the alternative, doing nothing? People often critique without looking at the actual downside.

Saving energy at the site to cut costs. This is just plain old common sense. One that is finally being addressed.

The USAs 2023 Telecom Pullback

I am sure most of you have read about the telecom slowdown and pullback in the US telecom market. To me, it was no surprise.

The carriers have built out their networks, probably to 70% capacity of where they want to be. That means the last 30% is going to be less of a drive.

They also have 5G nearly everywhere. Sure, they have to decide on SA and NSA in their networks and they have to deploy VoNR everywhere, but that is already happening. Anyone on the front lines saw this 6 months ago. We wanted our leadership back then but apparently, it fell on deaf ears that believed that the power of positive thinking would win in the end. I want to say to them that we told you so but that doesn’t help anyone. Now they know it and the reality is setting in. The good news is I am still positive, just curbing my enthusiasm to where it can make a difference.

So, what does this mean for our industry? Well, it depends on where you are in the industry.

What about the OEMs? 

As you probably heard Nokia and Ericsson are already paying the price in their stocks tumbling. That’s OK, they are seasoned veterans, and they will bounce back. It’s going to be a revenue shift. They have been pushing software and upgrades that have been ongoing. Smart people run them and they already made smart moves that will move them ahead as 5G services mature and grow. Then there is 6G, which once defined they can start moving towards it, maybe in 3 or 4 years. 

What about the Carrier’s teams?

Well, here is where I believe we’ll see a noticeable shift. The carriers have to downsize at some point. The carriers will want their networks to start to pay for all the investment they put into the network over the past 4 years. 

Carriers to their networks, “Hello, it’s payback time.

All the carrier’s teams that were in deployment, unfortunately, will start to downsize. I think many of them already see the writing on the wall. For some reason, the press seems to be missing the disappearing act. If you’re in the industry you already see it and hear the rumblings of a larger downsizing. That’s OK, we all knew it was coming but it still hurts when it happens.

Specifically, what does that mean? Usually, construction managers (CMs) and project managers (PMs) are the first to downsize. Then some of the engineering staff. Then the NOC staff. Support staff will dwindle but that all depends on how the carrier prioritizes support. 

Sorry teams.

What about the tower crews?

OK friends, this is tough to take but as the deployments wind down the first ones we see downsizing are the tower crews and their staff.

Climbers will start to consolidate into smaller crews. Don’t worry, the prices probably won’t drop, but the demand will, gradually, over the next year or so.

Luckily, DISH will continue to build. Crews working for DISH that have agreed to their pricing model should be able to increase headcount, unless DISH pulls back.

Sorry climbers, but in construction (like war), usually the front line gets hit first. Climbers are the front line. Climbers do the physical work and risk the most. Literally risk the most by climbing and working with heavy equipment in the air. They probably spend the most time away from their families, traveling for work, and have the most health issues. That’s why the industry is always changing players.

Also, they are probably the lowest paid. 

What about the tower companies?

Companies like Crown Castle, American Tower, and SBA are going to do just fine. I think the stock market may punish them a bit for not growing, but they already have the leases locked in with everyone. 

In my eyes, the towercos are in the best position to lock in their customers for a very long time. Sure, they have expenses, but the reality is that if they would stay in their lanes, they could maximize their profit. 

With the slowdown, they may be able to downsize a bit. It makes sense because they’ve expanded into construction and project management. They also had to ramp up their leasing and support services. I would think now they can start to see those departments get smaller.

I think they are going to be just fine moving ahead, just slower growth. Lucky for them upgrades will continue over the 4 years or so. Why does that matter to them? Because lease modifications are a way for them to suck just a little more out of those carriers. 

What about NOCs?

I think that most NOCs for network support will be doing well. I think that deployment support NOCs may have to shift to network support. While this nuance may get lost on you, it is a big deal to the support teams.

FYI: NOC = Network Operations Center. (Don’t mean to insult you, but someone new to the industry may not be clear.)

We may also see the support split between offshore and onshore. This will become a hotter topic with the rise of hacking, see how Nexius and Huawei allowed hacks to happen to carriers because the offshore NOCs allow employees to gain access directly inside of the carrier’s network using a Virtual Private network (VPN) connection. Behind all firewalls!!

For those of you that don’t understand this concept of NOC offshoring, let me explain briefly. When you have an onshore NOC, it usually means it’s in the same country as the carrier, in this case, the United States. That is considered more secure and expensive. Whether you agree or disagree, I don’t care. 

Offshoring is when you take the NOC to another country that is very cheap. These countries are generally India, Pakistan, Romania, Philippines, Viet Nam, or another country that has cheaper labor than the USA. 

When you get those annoying telemarketing or tax scam calls, chances are good they’re coming from Pakistan.

To sum it up:

  • Onshore is more secure but also more expensive.
  • Offshore is cheaper but less secure.

The rise of the onshore NOC may happen again due to these issues. It may or may not help since the Chinese hackers will continue to hack.

The NOC is becoming more and more relevant. The beauty and pain of automation, self-learning networks, and AI is that we will rely less on human interaction. Sorry silly humans, but AI is helping to improve networks, not just create the next Terminator.

What about Labs?

Labs will continue but they will move away from 5G once SA and slicing get resolved, and then move into 6G. While it is very early for 6G, they want to test a lot of theories to see how to define 6G. 

Lab staff may have to figure something out.

If you work in a mechanical lab, things may get tough but the carriers are always adding equipment to sites, so thermal testing is still critical. Will carriers lay the responsibility on the OEMs, suppliers, or their own staff?

What about Test Equipment?

And then we have test equipment. Believe it or not, there is a need to find 2 things, interference and PIM. For some reason, these 2 things are still required in this industry. 

FYI: PIM = Passive InterModulation. 

The broader the spectrum use the more likely the network experiences PIM. Broadband channels are opening up to 100MHz which means interference of all kinds will sneak in and cause headaches for the carriers. The end user may or may not notice, but for the NOCs and network engineers, it will keep them up at night. To be blunt, PIM sucks and it can’t always be resolved. If it isn’t an installation problem, it may be out of your control. 

Let’s say it all together now, “We hate PIM! PIM is bad!” Unfortunately, sometimes we have to live with it. Think of that cousin, uncle, or brother-in-law you have to deal with at every family get-together. Yup, like that. 

The rest of the testing still exists and will be used, but the need for dedicated test equipment will wane as software and crowdsourcing grow. 

The good news is we still have test jigs for the devices. This isn’t going to change and as most devices have more spectrum and additional features to smart devices and IOT devices, we will need to keep them compliant. 

I would guess as 6G gets defined lab equipment will start to rise again. Even labs will slow for the next 2 or 3 years but they will do limited testing.

Summary

Hey, the next few years might get tough. Why?

  • We are all waiting to see if DISH can build a network like they said they could. Charlie Ergen finally has to put up or shut up. Whether you’re a fan of his or not, I would bet that all of you are secretly rooting for his success. The industry needs this fourth player. I would like to see DISH find a partner that will help them grow, like Charter or Comcast but I don’t think that will happen anytime soon.
  • We see T-Mobile appears to have the best network and the largest growth of all the carriers. I believe they want to start building revenue without all the expenses of expansion, upgrades, and mergers. I am assuming the merger is officially over. We expect them to downsize the company to increase profits. They appear to be pushing towards #1, but they have work to do. So far, TMUS has not been a disappointment. 
  • Verizon appears to be in decline. For the last 5 years Verizon has been a disappointment. They dropped out of first place in nearly every category. The stock price has continued to decline. The spectrum shortages are shining through. The transition to 5G has been brutal. Luckily, they have a loyal following. Visible has helped. The partnership with Comcast has been favorable to them, at least it appears that way. I don’t know if that’s enough to keep them at #2.
  • AT&T hangs on, but again, see Verizon above. T-Mobile is pressuring them to be more profitable and improve network performance. However, they appear to be #1 in revenue, but for how long? They have a lot of spectrum, but are they using it wisely?
  • Comcast is a player, since they appear to be pushing into the telecom realm with all the services they offer. I wonder how long the Verizon partnership will hold up? It appears to be solid and the CBRS deployments aren’t exactly happening at lightning speed for Comcast.
  • Charter is another cable company that is making the push into a top telecom company. In my opinion, they have the best potential to see massive growth.

Do you trust your infrastructure?

Huawei’s Woes.

One thing I think we can agree on, at least in the West, is that we want to trust our infrastructure providers. The equipment, the software, the apps, and the provider itself. Trust is a huge issue and one that is not as common as you’d think. 

Here in the USA, law enforcement does have access to records but they are supposed to get a warrant first. I know they probably have ways around that, but I would like to think there’s a system in place to protect our privacy.

I lost a lot of that trust when I found out how the FBI was censoring Twitter. How they had people on staff and cut Twitter a check of taxpayer’s money, to make the stories lean towards one party to hurt the other. This is a failure in America. The FBI, in my opinion, brought shame to a respected and trusted organization. I don’t know if Americans can trust them again. 

Let’s get back to infrastructure, I was trying to make a point. We need to trust our providers. It’s a huge deal. We want them to be trusted advisors and provide quality. It has to be a total package. Sure, they may falter, but we need to have faith that they will get better.

That brings me to Huawei’s woes.

Did you know the reason Huawei is being banned in the West? I mean it’s something we talk about. I remember when I was at Alcatel-Lucent, (ALU), and we were worried about Huawei because all their documentation looked amazing. They promised the world and it was way cheaper than ALU or Ericsson. We didn’t know what to do.

Luckily, the US government pressured Tier 1 carriers at the time not to purchase Huawei. We were saved. Remember that this was back in 2018. 

As far back as 2013 Sprint and Softbank told the US and Japanese governments that they would not use Huawei gear. This must have been hard for Sprint, they loved cheap gear!!

Yet, we wondered why. I mean, did they really think that the Chinese government would use that telecom tool to spy? Apparently, they knew something we didn’t. 

Read on tech fans!!

The skinny.

As you know, American communication companies have been pressured not to use Huawei a decade ago before it became an official ban under President Trump, (Rip and Replace Program).  He eventually had Canada hold the Huawei CFO about something. Then they stopped selling Huawei phones about 5 years ago. However, Huawei denied spying allegations but the CFO admitted to wrongdoing in the Iran Trade sanctions case. 

President Biden got elected. You all thought a new president would mean a new policy. Well, not really. 

Surprisingly, Biden stepped things up in the trade war. He went beyond telecom equipment by stopping American semiconductor chips from going to any Chinese company. This includes all American chips which are made in Taiwan and the US among other countries. WOW!

Huawei officially had a bad reputation in the West!! Things kept gaining momentum to ban Huawei in the West. Loss of market share, loss of revenue, and a serious loss of margin!

For those of you that don’t know, margins are much higher in the West than in Asia, especially in China where margins can run negative. In Asia, especially China, making a decent margin is a gift. Many companies that sell there do so at a loss of margin. China still considers itself a developing country which means they don’t have to follow a lot of the same rules other countries do. Labor is cheap which means they can’t afford to buy many iPhones, although Apple will tell you a different story. The good news is that China’s middle class is rising. 

For some reason, everyone wants to sell in China because of the huge population, but most tech companies bleed money and rely on other countries to make up the margins. This is crazy to me, but I am sitting at home writing this and not actively building a tech company. 

Huawei is a protected company in China, which basically means the home team will always win. That doesn’t mean the competition will be kept out. They can get in, operate at a loss, and expect their tech to be copied. You see, rumor has it that the Chinese competition gets reverse-engineered until they can replicate it. Stealing tech was common throughout Chinese history. You have to love history repeating itself. Don’t feel bad, the rest of the world stole gunpowder and green tea from them. It all worked out. 

One thing you learn is that what China does and what they say are usually contradictory. It seems like honesty is not the best policy, and the PRC likes to be the bully, not bullied.

So, here’s the thing. My past experience with Huawei is that they have a great product and great support. When they have problems, they will send a small army of engineers to work with their customers. I’m not sure why they send so many nor why they feel there is a need to have so many. Now I’ve become suspicious, like, is it to have the PRC represented?

Their equipment was cheaper than the other OEMs and somehow their support was very cheap as well. They have many 4G and 5G patents that helped move telecom into the 21st century. They came a long way in a very short time. Pretty amazing right? 

In the US, they sold to many Tier 2 operators and most were happy with Huawei. They liked the support and cheap managed services offered them. FYI – managed services basically gave Huawei access to the customer’s network, direct from China. Think about that for a few minutes.

But then the trust was broken, first in Australia, then in Denmark. Actually, Britain first had suspicious leaks, but that was not as public as these other two stories.

That’s why these stories are crazy! To risk trust by compromising customer privacy is not common in the West.

Fun fact, Huawei means “Chinese achievement”. It was founded by Ren Zhengfei, a former People’s Liberation Army engineer. We all love when an engineer starts a hugely successful company. However, the fact the government played such a huge role in his success raised a lot of suspicions.  

Also, recently under China’s President Xi Jinping, the internet is being locked down even harder, making VPN usage illegal without the government’s permission. It must be approved by the Chinese government so they have access to all the data. If it’s not government approved, then expect fines and/or jail time. (VPN means Virtual Private Network and gets you access to the destination place where your virtual network terminates.)

By the way, remember that President Xi Jinping has a very wealthy family. So he is making a lot of decisions based on wealth. Not so different from the American leaders. 

Allegations in the UK.

It was the UK that was one of the first to ban Huawei. They had suspicions but only the potential for problems. At least that’s the way I read it.

Huawei said it would address the security concerns and make everything better. Unfortunately, that didn’t make anyone sleep better.  

To be honest, when I researched this, all I could find was a lot of accusations and very little solid evidence. I trust British Intelligence to tell us what’s going on but give me something to work with. They had a huge report that was very difficult to get through. OK, it’s 45 pages, but I just could not read all of it.

It appeared to be allegations and not much more. 

Then, I read about what happened in Australia.

What happened in Australia?

Australian intelligence discovered that Huawei had people spying on Australians, apparently government officials. They had found a security breach 10 years ago that sparked this controversy. 

The US Government had suspicions about Huawei being a spy branch for the People’s Republic of China, (PRC), something that doesn’t surprise anyone anymore, but at the time it was huge! Today we realize that the PRC owns part of almost every Chinese company. 

The Australian government found that their local wireless telecoms had been infiltrated by cyber spies. Apparently, they allege it was a network of cyberspies from China. While this seems crazy, remember that China locks down the internet to their residents, so any hacker in China that has access to the outside world does so with their government’s blessing. 

When I say lock down the internet, you can’t use many things that are open in the USA, like Google. 

Those brilliant Australians found malicious code in a Huawei update on a major Australian telecom company back in 2012. While the Chinese government denies this, it was confirmed by over 2 dozen national security officials.

What happened was the system did an update, pushed to the network by Huawei technicians, which is common in telecom. That’s when someone noticed it was stealing data and targeting specific users. They saw the code and knew what it was doing, but then, a few days later, the code deleted itself. Just like that it was gone. Not completely, there were bits and pieces of the code still there, but the bulk of it disappeared as quickly as it came.

The code acted like a digital wiretap that could target specific people if required. Then, mysteriously, the code deleted itself a few days later. Go figure! 

The code was pushed in and had a self-destruct mechanism to automatically remove itself a few days later. Give a lot of credit to that hacker, that is awesome. Too bad it was noticed right away. 

I wonder whose phone information was targeted before it deleted itself. Do you think it was someone in the intelligence community or in government to be noticed so quickly? Maybe a ranking US official in Australia at the time. I mean, to notice something like that in time to see it before it auto-deleted itself must have been because it triggered something. Do you think?

Luckily Australia’s intelligence agency caught it and documented what they saw. They immediately alerted US Intelligence. Apparently, US Intelligence confirmed that they saw the same thing a year earlier, also from Huawei gear, and pushed the same way, AND with the self-destruct mechanism. 

Both noticed a high degree of Chinese spy activity each time. 

Both agencies said there were more incidents where they kept quiet. They said it was becoming more and more common that patches were actually spyware. These “patches” seemed to come from Beijing. 

According to the Bloomberg article “Chinese Spies Accused of Using Huawei in Secret Australia Telecom Hack” released 12/16/2021, written by Jordan Robertson and Jamie Tarabay, Michèle Flournoy said she was aware of this from public forums. She was under-secretary of defense for policy at the Department of Defense under President Barack Obama and co-founder and managing partner of WestExec Advisors LLC. She has some clout!

So, here’s the thing, according to the article, Huawei and ZTE may not be aware that their employees are doing this since they could be working directly for the PRC. There could be factions at these companies that have software and network engineers working together to do this. In the article, they state that ZTE’s message was, (quoting the article) “employs many rumors and speculations to prove nonexistent accusations, the committee rests its conclusions on a finding that ZTE may not be ‘free of state influence’. That standard “would apply to any company operating in China.”

They also refer to a Wall Street Journal article where Huawei technicians helped African leaders spy. That article can be found here. How crazy is that? Why would they do that? Or, why not?

Back to Australia. Huawei’s response was “no evidence had been provided to demonstrate that Huawei had undertaken anything untoward in Australia.” Does anyone know what they meant by “untoward”? 

That response may be true, to be honest, I don’t know. I am still trying to figure out what untoward means. Apparently, I am not the sharpest tool in the shed. I mean, I googled it and it says “unexpected and inappropriate or inconvenient”, but that really doesn’t explain crap. The fact there was a spyware virus put there in lieu of a patch was unexpected and inappropriate coming from a telecom infrastructure OEM. Not to mention the inconvenience. Am I right? 

I do know that China put heavy tariffs on Australia and it appears they’re no longer friendly trading partners. To be clear, this is the Chinese government punishing China for the allegations and for banning them.

I think Australia will be happier, in the long run, to lessen trade with China, but that’s only my opinion.

Watch out China, here comes India, a new friendly trading partner. Sorry, I digress. 

FYI – if you’ve ever read the book “Unrestricted Warfare” by Qiao Lang and Wang Xiangsui, this is a step in the updated war strategy for the People’s Liberation Army. I am putting together something on that for the future.

More hack details came out in 2021. Bloomberg did an extensive report on this, https://archive.is/smY8T.

Something is very wrong in Denmark!

OK, now this is like something out of a spy novel!! Strap in and get ready to read. While this is more on manipulation of winning a contract, I think it shows to what length Huawei and their backers will go just to get a major win. 

I got this information from another Bloomberg article found here, called “When a Huawei Bid Turned into a Hunt for a Corporate Mole” written by Jordan Robertson and Drake Bennett on June 15, 2023. I highly recommend reading it!! They have the option to listen to it, also awesome! Listen to the Bloomberg Podcast and they talked about spying on bids in Denmark with TDC Telecom. They also did a YouTube report found here. Kudos to Bloomberg Businessweek for pulling together a great report. 

FYI – the Danes didn’t care what the US thought of Huawei, they were going to do what they thought was best for them. They went with Huawei for 2G and 3G because Ericsson’s service was not living up to their expectations at that time. 

The big story happened in Denmark with the telecom company TDC. It was the Bloomberg report that initially blew open the investigation.

For those of you that don’t know, TDC Group is a Danish Telecom company. They’ve had Huawei in their network since 2013. Huawei was a trusted partner that already got $700M back then. A trusted partner, up until 2019. 

The basis of the story is that in 2019 TDC was looking for bids to upgrade to 5G. In the end, the 2 final players were Huawei and Ericsson who were bidding against each other. Unfortunately, Nokia was not mentioned. 

When reading this, keep in mind this is a $200M contract, it ain’t chump change, it’s a game changer.

So, the bids were in and they were competitively close, but Ericsson had the edge in price. They were considerably lower, but that isn’t always the deciding factor, is it?  These were the final bids in the final round. Again, Ericsson appeared to have the winning bid. Amazingly, Huawei put in another final bid, literally right before the deadline, just lower than Ericsson’s. When I said last minute, it was at 2:52 AM. Similar offer at a much lower price.

It was almost as if they knew what Ericsson’s bid was. But that should not have been possible, or was it?

Red flag time! This late change raised red flags. (In this case, literal red flags since they’re Chinese.) You gotta love the Danes for being cautious and observant! Respect. 

How could Huawei know? It was like Huawei knew what Ericsson’s bid was, privileged TDC information. 

Then, in meetings with vendor teams following submission, there was a noticeable difference in the teams. Ericsson’s team was nervous whereas Huawei’s key guy, Yang Jason Lan, appeared calm and almost overconfident. So overconfident that it set off the TDC executive’s suspicions. This was strange to the TDC team, (again, the Danes are super observant). So strange in fact, they launched an internal investigation.

Call in security!!

Why? Only senior executives had access to the offers and the bid responses. 

So TDC security investigated everyone and everything. Who had access to the bid, the pricing, and the details? Was it only TDC executives and the respective vendors?

They started looking at laptops and phones, the obvious devices. Hacks were suspected. To us it seems obvious, right? Wouldn’t that be the easiest way in and the best way to shut down the leak? 

No dice. They didn’t find any hacks. That would have been easy to explain and quick to shut down, just remove the hacked devices. So now it’s getting scary, no hacks mean it could be an inside job. 

Let’s start by saying that security also suspected someone could have been listening to their conversations using surveillance techniques. So, they swept the boardrooms and found microphones hidden in the boardroom. Who does that? Sounds like some serious James Bond stuff, am I right?

Can you imagine an employee, cleaning person, or an outsider coming into the conference rooms and installing microphones? Think about the balls it took to do something like that?!?

That’s not all, but what if someone intentionally shared company confidential information with Huawei, specifically the lead, Lan? Who could share those files?

Who would have access to Ericsson’s offer and then share it?

Well, according to GDC security, it was someone named Dov Goldstein, head of special projects, who became a prime suspect. This guy was a 13-year-plus employee, formerly a mechanical engineer that moved up to the head of special projects inside of TDC. Apparently, he got access to all the files and was a close buddy of Lan. Security found all the files on Dov’s laptop. However, they could not figure out how the files got to Lan.

While security was investigating Dov, he went to executives and complained, like he knew what security was doing. It’s like he knew all their moves!! 

So, they had to move the team to a remote location. This was a legal office downtown, away from the main campus, and where the Huawei employees did not have access. Nor did Dov. Secure access. However, they felt they were still being watched. They were!

One night a night watchman looked up and saw a drone above the building. The security guard saw a drone outside the conference room window with a bright light. Inside the room was a whiteboard with the investigation laid out. Whoops!

They were on a high floor and the drone had found the room where they had all the details laid out on a whiteboard. The bright light of the drone enabled it to see inside at that time of night.

It gets crazier! Employees felt they were being followed. Someone started following GDC employees whenever they went out to eat or in a public place.

Guess what, Ericsson got the contract. 

Huawei’s Lan went back to China.

Goldstein left GDC before the investigation was complete. He moved on to his next job. His supervisor stepped down as well.

Huawei’s employees were banned from GDC and all its networks.

Several GDC employees left GDC after the investigation, we’re not sure why. Even many of the investigators left. Even the GDC lead for the investigation left. He would not comment or say why he left. He was the one that oversaw the investigation, yet, he left and would not talk. Why is that? It could be as simple as an NDA, maybe Huawei threatened to sue. Who knows?

Did Huawei have a response? Of course. They said, and I am quoting from the article, “Huawei complies with applicable laws and regulations, and strives for the highest standards of business conduct. We deny any wrongdoing.” No kidding.

Lan had a lawyer respond for him on this investigation, and I’ll quote the article again, “believes that he has acted in compliance with all applicable laws and regulations at all times.” Lan’s relationship to Goldstein was of a professional nature, and one that was appropriate in the circumstances.”

Oh, China’s foreign ministry weighed in, because that’s what they do, and they said, “It is no secret that Huawei has been unreasonably suppressed and treated unfairly in the United States and Europe”.

Also, in 2021 Denmark enacted legislation that gives intelligence officials the ability to block domestic telecom deals involving suppliers that don’t have security agreements in place. Imagine that!

Reputations.

Let me ask you, after all of this, do you trust Huawei as an infrastructure company? In this industry trust is a huge issue. Ericsson lost some of it with allegations of payouts in the Middle East. Alcatel Lucent and Ericsson were getting in trouble decades ago when they would entertain customers. 

However, today it’s all about the privacy of each person, not some corporation making shaky business decisions.

Each person has a choice about their privacy in the West. We rely on the government to support us in many ways, but not to make decisions that could destroy our daily life as we know it just because they disagree with something we say. 

Of course, I guess it depends on what we say and how hurtful it is. Is it defamation? That’s another conversation.

There are reports that Huawei employees are incentivized when they get information about the competition. I don’t mean on the internet, I mean by looking at the competitors’ equipment at a site, logging in, and obtaining privileged data.

I think that if nothing else, Huawei has lost so much trust, especially in the West, that they were actually banned in several countries. How crazy is that?

Ask yourself this:

  • Who do you trust? 
  • Do you trust your smartphone’s privacy settings?
  • Do you use your device for banking?
  • What about Investments?
  • What about retirement?
  • What about private messages to friends, family, or lovers?
  • What about business deals?
  • How much do you trust the device, the company that made your device, the carrier you pay for every month, and the infrastructure that it rides on?
  • Whom can you trust?

Now what?

OK, obviously Huawei is being punished. We really don’t know if it’s the fault of Huawei or if the PRC has infiltrated Huawei and ZTE. I mean, we may never be sure, but this looks quite suspicious now that we know what’s going on. We do know that the PRC has partial ownership of both companies. 

I am pretty sure we, the common public, will never know it all. We’re citizens and I think only ranking government officials really know. We only know what little we’re being told. That’s why the reporters at Bloomberg and Wall Street Journal have my greatest respect. They are taking their chances reporting the truth. 

The PRC is already attacking Bloomberg for the reports. I feel bad for the reporters as China pressures their employers to punish them. We’ll have to wait and see what happens going forward.

All the same, Huawei and ZTE have been replaced in the USA along with Australia and the UK. When I say replaced, I mean banned by the government. Don’t feel bad for Huawei, most of the world is allowing them to be used. They just don’t care about what happened in the US and Australia. I mean even Denmark is on the fence, and they had proof of an RFP being tampered with. 

Some carriers across the world are banning Huawei. Making it their choice, not the government’s. 

So believe that you will. In the West, it’s still up to you to decide. I think China has already made up its mind. 

More links about this:

Tips to Work With