Less Busywork, More Productivity: The Art of Automating Telecom CPE Provisioning

There are many ways in which Bill Gates' frequently cited quote—"lazy programmers are often the best because they find the easiest and most efficient ways to solve problems"—can be misunderstood, but really only one interpretation gets it right 🎯.

It's understandable that confusion emerges since the word "lazy" has generally negative connotations. Gates wasn't saying programmers should simply wait around for problems to solve themselves; instead, he was highlighting an attitude: skilled programmers try to minimize unnecessary work and streamline processes. In reality, perhaps we aren't lazy enough!

The Problem 📌

It was 2015, and I was nearing the end of my studies at university while completing an internship at a telecom company called Cartago Telecom (back then, it was known as Electrónica Martínez). My colleague and I had recently deployed a Speedtest.net node to allow customers to test their Internet throughput themselves—and also so that our metrics wouldn't be limited by momentary peering constraints or uplink throughput, improving customer satisfaction somewhat sneakily in the process 😏.

Around that time, I started proactively focusing on the company's Customer Premises Equipment (CPEs). I realized there was plenty of room for improvement in our deployment workflows and provisioning processes for new customers. Simultaneously, I was searching for a solid topic for my final project at the university. It was a perfect match 🤝.

The company was deploying three different CPE devices: two from the brand Tenda and one from Grandstream. One of these devices used DHCP Option 66 (DHCP66) for provisioning, another one tried a fixed URL, while the third one relied on TR-069.

DHCP66

What's DHCP66? 🤔 It's a configuration setting used within DHCP servers that specifies the provisioning server for client devices—commonly a Trivial File Transfer Protocol (TFTP) server.

In our case, the Tenda W300D device expected a TFTP server, so after appropriately configuring the proprietary DCHP implementation of MikroTik devices, I actually needed to set up a TFTP server. After evaluating several available open-source solutions, I decided to go with Xinetd ("extended Internet daemon").

Xinetd isn't exactly a TFTP server itself; rather, it's a versatile "super-server" that listens for various incoming network requests, activating the appropriate subordinate servers as needed. To make it work specifically for TFTP provisioning, I installed a plugin called tftpd.

The Grandstream HT502 device introduced an interesting twist 🌀. By default, these devices would attempt to provision and update themselves by reaching out to Grandstream's default provisioning URL fm.grandstream.com/gs. Instead of allowing these units to contact external servers, I"hijacked" this hostname using our internal DNS servers, redirecting their provisioning requests toward our in-house infrastructure. I provided a modified firmware binary designed specifically to discard this external URL and instead rely directly on DHCP66.

After implementing this redirection, provisioning became straightforward. Upon startup, the Grandstream device would leverage the same DHCP66 mechanism I already had set in place, retrieving its specific configuration file cfgMAC.xml via our existing TFTP server infrastructure.

TR-069

Lastly, we had a second Tenda W308R router that relied first on DHCP66, but then relied on the TR-069 protocol for automated provisioning and management. Unlike DHCP66, TR-069—also known as CWMP (CPE WAN Management Protocol)—is designed as a bidirectional protocol, offering ongoing management, firmware upgrades, and configuration updates actively pushed by an Auto Configuration Server (ACS).

In several ways, TR-069 is better than DHCP66, since DHCP66 doesn't let you make specific changes easily. For example, if someone tries changing the configuration when using DHCP66-enabled routers, the CPE will have to be reset first for it to get the new settings 🔌🔄😩.. This basically means clients will lose their internet connection temporarily, causing a frustrating user experience. As a workaround, the new configuration had to be applied manually via the webUI, which consequently leaves the configuration inconsistent against what's currently in the TFTP server.

The workflow for TR-069 is pretty different, but still, DHCP66 was needed for the initial configuration for the Tenda router. After successfully setting the local ACS URL (I used GenieACS as the ACS server implementation) and other connection parameters—like the dedicated VLAN for high-priority 🚀 TR-069 management—the router would contact the ACS server, sending details like the device model, firmware version, and hardware ID. Then, our ACS replied, sending down further configuration profiles, firmware updates (if it needed them), and other customized settings built specifically for our deployment.

Beyond initial provisioning, TR-069 allows to periodically track device status, monitor performance metrics, modify device settings dynamically, or remotely trigger device reboots whenever needed for troubleshooting 🛠️. GenieACS provided an overview of all deployed units, enabling centralized visibility and powerful automation in managing our customer's equipment, drastically reducing manual effort while at the same time, improving end-user support.

Thanks to my experience with GenieACS, I later joined another company where GenieACS itself became my main focus. My responsibilities there included turning GenieACS deployments into a production-ready system, troubleshooting and fixing CPEs whose TR-069 implementations did not fully comply with standards (high five to all non-standard implementations), and everything in between. Some of the results of that work can be seen here and here. I was the one who, for the first time in the community, made GenieACS work on Red-Hat based OSes with the help of Docker 🐳!

R and Shiny ✨ for the Web UI

Since standardization was, let's say, flexible in our case (to not say that it was totally custom for each CPE model...🤢), managing all this neatly became the next challenge. I had provisioning covered: the test routers were now happily configuring themselves at startup. But once you want to replicate it for the next client, the task ahead was clear.

I identified the need for a management interface to provision all our equipment from a single web dashboard, with all the needed info to also configure the RADIUS server (for PPPoE sessions and speed limits 🚦) and the ticketing system software. So I decided to build a customized web application to onboard new clients.

I chose the R programming language with Shiny, out of my desire of working with R language, like I explained in a past post. Shiny allowed me to quickly spin up an interactive front-end without writing endless amounts of interface code. For the tougher bits, such as certain scripting tasks and integration with network scripts, Python filled in wherever R needed a helping hand.

I was able then to create the DHCP66 configuration files, manage GenieACS CPEs via API, connect via API as well to the RADIUS server and automatically add all the client's new information. All, from the same management interface.

Sadly, the same couldn't be said for our ticketing system, which lacked a usable API at that time 😭. To get around this limitation, I ended up using Selenium to automate interactions with the ticketing interface, which was error-prone, and it would prevent this solution from scaling, but that was the only option available for now. Interestingly enough, the Selenium expertise I gained through this workaround later proved very useful in my career, helping me crawl and gather data for various analytics services 🤓.

Conclusion

All in all, it turned out great, and it now it's open source. You can check it out here. In the end, my solution saved hours of work for a single onboard—from assistants all the way up to engineers—and earned my university project's Honorific Mention 🏆. As I've previously mentioned in another post, one of the professors evaluating my project reacted with frustration when realizing the company wasn't paying me a dime for this work. But compensation aside, the overall experience and learning I gained made it worth it.

Looking back at this experience, I'm convinced Bill Gates was absolutely right: the best programmers really are the "lazy" ones. By investing the time to build automated and streamlined solutions, the results spoke for themselves: significant improvements in efficiency, less repetitive work for the team, and recognition of my project within the university and within the company 😊.