June 25, 2023 9:10 AM
Image Credit: Created with Midjourney
Join top executives in San Francisco on July 11-12, to hear how leaders are integrating and optimizing AI investments for success. Learn More
Modern IT networks are complex combinations of firewalls, routers, switches, servers, workstations and other devices. What’s more, nearly all environments are now on-premise/cloud hybrids and are constantly under attack by threat actors. The intrepid souls that design, implement and manage these technical monstrosities are called network engineers, and I am one.
Although other passions have taken me from that world into another as a start-up founder, a constant stream of breathless predictions of a world without the need for humans in the age of AI prompted me to investigate, at least cursorily, whether ChatGPT could be used an effective tool to either assist or eventually replace those like me.
Here’s what I found out.
I started by getting the opinion of the best source I could think of about how ChatGPT could add value to network engineers: ChatGPT. It didn’t disappoint and generated a list of three areas it determined it could help:
Event
Transform 2023
Join us in San Francisco on July 11-12, where top executives will share how they have integrated and optimized AI investments for success and avoided common pitfalls.
Register Now
Configuration management
Troubleshooting
Documentation
I then developed a set of prompts — admittedly not optimized — to determine whether or not the tool could, in fact, be an asset to network engineers in one or more of these areas.
Configuration management
To test ChatGPT’s ability to add value in configuration management, I submitted the following prompts:
Can you generate a complete example configuration for a Cisco router with the purpose of starting an internet exchange from scratch?
What about Juniper?
Can you create a Jinja template for each vendor?
The ChatGPT results are extensive, so space — and my respect for the boredom limits of those reading this — limits an exhaustive reproduction of them here, but I have posted the complete transcript of all of the ChatGPT prompts and results on GitHub for those searching for a non-pharmaceutical substitute for Ambien.
So, in the case of configuration management, ChatGPT performed fairly well on basic configuration tasks, and I concluded that it is aware of vendor-specific syntax and can generate configurations. However, the configurations generated by the system should be carefully inspected for accuracy. The generic prompts I tested would be akin to building a quick lab, a task most young networking engineers find tiresome at a minimum and clearly a chore that can be handled by the technology (with, again, some human oversight).
Troubleshooting
To test ChatGPT’s prowess at troubleshooting network engineering challenges, I turned to Reddit, and specifically the /r/networking subreddit to find real-world questions posed by network engineers to their peers. I pulled a few questions from the thread and proposed them to ChatGPT without optimizing the prompt, and the chatbot handled the easier questions well, while it struggled with the more difficult challenges.
Notably, I specifically asked a question that required knowledge of STP, or the Spanning Tree Protocol, a switch capability responsible for identifying redundant links that could result in unwanted loops. Frankly, my opinion is that ChatGPT understands STP better than many networking professionals I’ve interviewed over the years.
At present, ChatGPT can’t replace experienced networking professionals for even slightly complex issues, but it wouldn’t be alarmist to suggest that it might result in the obsolescence of many subreddits and Stack Overflow threads in the coming years.
Automating documentation
This was the area of highest deficiency for ChatGPT. The chatbot initially assured me that it could generate networking diagrams. Knowing it is a text-based tool, I was obviously skeptical, a prejudice that was confirmed when I asked it to generate a diagram and it explained to me that it doesn’t have graphical capability.
Further prompting for network documentation led to the realization — confirmed by ChatGPT — that I needed to provide a detailed network description for it to provide a network description, clearly not a value add. Thus, in the case of automating documentation, the chatbot not only failed, but was guilty of generating lies and deception (so perhaps it’s closer to demonstrating human characteristics than we think). In fairness to AI in general, there are AI applications capable of generating images, and it’s very possible one of those may be capable of producing a usable network diagram.
I then asked ChatGPT if it could generate a network description based on a router configuration file, and it provided a decent summary of what’s configured until it apparently reached the limits of its computational commitment to my prompt, a limit likely implemented by its designers. It is, after all, a free tool, and resources are expensive, especially for an organization burning meaningful cash these days.
Conclusions
A few of the challenges I encountered in my brief experiment when using ChatGPT for network engineering include:
Ensuring accuracy and consistency
Handling edge cases and exceptions
Integration with existing systems and processes
My guess is these issues are not unique either to ChatGPT or AI applications generally, and some cursory research may explain why. Cornell researchers have been studying large language models (LLMs) for some time and “draw a distinction between formal competence — the knowledge of linguistic rules and patterns — and functional competence, a set of skills required to use language in real-world situations.”
Also from some of their research summaries: “Too often, people mistake coherent text generation for thought or even sentience. We call this a “good at language=good at thought” fallacy. Similarly, criticisms directed at LLMs center on their inability to think (or do math or maintain a coherent worldview) and sometimes overlook their impressive advances in language learning. We call this a “bad at thought=bad at language” fallacy.
This analysis is consistent with my experience preparing this article: Specificity reigns supreme when it comes to putting ChatGPT to work. Large, open-ended prompts on complex topics highlight a lack of “functional competence” in the chatbot, but that reality doesn’t neutralize its impressive capabilities when employed for specific tasks by an individual skilled in using it properly.
So, can ChatGPT replace network engineers?
Not yet.
Mike Starr is the CEO and founder of trackd.
DataDecisionMakers
Welcome to the VentureBeat community!
DataDecisionMakers is where experts, including the technical people doing data work, can share data-related insights and innovation.
If you want to read about cutting-edge ideas and up-to-date information, best practices, and the future of data and data tech, join us at DataDecisionMakers.
You might even consider contributing an article of your own!
Read More From DataDecisionMakers
>>> Read full article>>>
Copyright for syndicated content belongs to the linked Source : VentureBeat – https://venturebeat.com/ai/how-chatgpt-could-replace-it-network-engineers/