Here’s how top CIOs build highly effective AI teams

A few years ago, executive sponsorship was one of the biggest roadblocks to getting AI initiatives off the ground. Today the volume of business demands for AI solutions is high enough to make any CIO’s head spin, but they’re rising to the challenge by taking a strategic approach to initiatives while building high-performing AI teams.

For instance, Katrina Redmond, CIO at power management firm Eaton, created an “AI factory” tied to specific areas of the business to meet pressing needs. Subject matter experts on the AI team don’t directly report to IT, although they do closely collaborate. “We jointly decide what activity to prioritize based on value generation, and go after it accordingly,” she says.

Eaton

Each team at Eaton consists of an AI lead, product owners, ML engineers, data engineers, and cloud and DevOps engineers who work with business domain experts. Having a VP of AI and innovation is also key to building highly effective teams, Redmond says.

But having domain experts on AI teams won’t be enough if they can’t frame the business problem in terms that technical teams can understand, says Kathy Kay, EVP and CIO at Principal Financial Group. Someone needs to wear the hat of product manager — the person with sufficient domain expertise to define the business problem — and translate it. “That may be a product manager on the business side, in the IT organization, or another line of business product manager who’s been down this road before and can work with the product manager who needs the problem translated,” she says.

Data scientists and AI engineers are important, adds Hugh Burgin, AI automation and analytics leader at EY Americas, but the biggest ROI, he says, doesn’t come from any one skill but all skills combined to transform the business.

5 categories where AI teams should include roles

What are the essential components of effective AI teams? You need expertise across five broad categories, starting with executive sponsors for each initiative, says Burgin.

Another key role that’s sometimes overlooked is the inclusion of people early on who will directly use the AI. In other words, end user engagement. “Most AI projects can be delivered technically, but that doesn’t mean the end user will adopt it,” he says.

Then there’s transformation engineering. These roles are subject matter experts on both the business process and the AI solution. Sometimes referred to as product owners, product managers, or functional leaders, these people work with the team to develop the AI application and have the functional knowledge to transform the business with it.

The people responsible for product delivery and change management constitute another category. Common titles include project managers, change management professionals, and scrum masters.

EY

Finally, IT leaders need AI support professionals whose responsibilities span all use cases to ensure consistency with respect to responsible AI, governance, and financial operations. “These support roles already exist today in IT but the job is changing and needs to be upskilled for AI,” Burgin says.

Essential roles every AI team should have, or at least consider

Within the core AI team, the most critical roles to have onboard are data scientist, data engineer and AI engineer. But enterprises may also need AI architects that focus on overall AI solution architectures, as well as model managers, validators, testers, and ethicists, says Arun Chandrasekaran, distinguished VP, analyst at Gartner, which just published a research paper on AI job roles, responsibilities and skills.

It’s important to keep in mind when hiring that the role of data scientist is changing. “Models are now massively pretrained, so data scientists are spending more time tuning and operationalizing the models,” says Chandrasekaran. “Model managers have a deep understanding of the vast array of models available and must select the most accurate, affordable, and most performant for their application. AI validators assess fairness, bias, transparency, and explainability of AI apps, and ensure they meet business, regulatory, and ethical standards. And AI testers test AI components like models, APIs, and pipelines for bugs, performance issues, or unexpected behaviors.”

Gartner

Then product managers act as intermediaries between engineering and end users. “They talk to the users, understand the requirements, and make sure products meet the needs of those users,” he adds.

And then AI ethicists develop and enforce guidelines for ethical use of AI. “They monitor for bias, toxicity, and harmful output,” and are especially critical for regulated industries, Chandrasekaran says.

Platform vs. product teams

Global engineering, consulting and construction company Black & Veatch’s focus on the activation of AI spans two different models, says Mike Adams, its EVP and chief digital technology officer. Platform teams ensure the company benefits from native agentic and gen AI capabilities made available by its strategic platform providers — Microsoft, Salesforce, Oracle, and ServiceNow — while the digital product teams develop business-specific AI capabilities that generate enduring and differentiating value.

Platform teams also ride the coattails of AI investments made in strategic platforms by leveraging some of the APIs they offer, he says. Specifically, his teams work to expose strategic platform APIs to Microsoft Teams and Copilot, which is becoming the company’s user interface to agentic AI. “We’re still in the early experimentation phase,” Adams says.

Roles include platform-specific architects, application portfolio managers, and some platform engineering and development roles. “As part of our strategic platform ecosystem, the heavy lifting and investment in AI capability is done on the platform provider side, so a lot of what we’re trying to focus on is managing organizational change to drive appropriate adoption,” he says. But as soon as you start tailoring an AI solution to change the business or have it learn your business, says EY’s Burgin, you need to have a full breadth of skillsets working on that.

At Black & Veatch, that innovation falls on the digital products side of the house. “We’re in the early buildout of a BV digital product operating model,” Adams says. Team roles include a digital product manager or owner, and digital product delivery lead, who heads up a delivery team that includes roles such as scrum master, quality assurance engineer, solution and product architect, software developers, as well as DevOps and cloud engineers that manage the iterative development, build, and release processes of digital product software, which, in many cases, include gen AI, applied AI, and MLOps components.

Black & Veatch

Other roles may have data engineers and data architects if the digital product incorporates generative or applied AI. “Successful AI product teams are similar to digital product teams,” Adams adds. “The only difference is you have domain-specific skillsets around data, analytics, and AI engineering and architecture.”

Black & Veatch’s digital product operating model isn’t fully developed yet, so for now, business relationship managers help to fill in on digital product management roles. “Ultimately, the end-to-end digital product operating model will scale to support the first BV digital product, BV Ask, an internal version of ChatGPT that’s been trained on [internal] data, including engineering best practices, IP from engineering, project execution, estimation, and other unstructured data,” Adams says. Engineers can access the information through a natural language or conversational interface in order to democratize access to information across the engineering community.

As in many organizations, the firm’s AI teams are still evolving as well. “Areas we have to beef up are formal product management and product ownership,” Adams adds. “We need just a bit more rigor there,” adding that these roles need to be firmly anchored to the customer. “The opportunity is so big that if we don’t ground it with customer needs, we’ll be in danger of nibbling around the edges of every AI opportunity versus materially changing the game for our business and customers.”

Grow your own team: training vs hiring

Core AI team roles are in high demand, so training IT professionals to move into AI-related positions is a strategic necessity. “Cultivate your internal talent because external talent is very hard to find,” says Kay at Principal. Her organization helps IT professionals learn new roles by pairing them with experienced people and giving them “stretch assignments.”

Black & Veatch is cross-training engineering teams on AI as well, and it hasn’t been a heavy lift. “These are technologists, so it’s not like LLMs are mystifying to them,” says Adams. “These capabilities are now ingrained in software engineering, data engineering, and analytics.” Adams’ strategy is to align adjacent skill sets by allowing IT professionals to learn through experience.

Eaton is following a similar path, too. “We’ve already transitioned our existing innovation team, which was already working on robotic process automation, and we’ve focused a core area of our chief data officer’s team on AI efforts, and accelerating data stability and governance,” Redmond says. The teams gain experience with pilot projects and working in sandbox environments.

Every CIO should have a clear plan on how to remap skills for the future, says Chandrasekaran. “Look at what skills you’ll need two or three years down the line and plan for how you’ll take people down that line,” he says.

Organizations may still need to recruit from outside to fill some roles, however. When hiring AI engineers, Kay says to look for people who like to tinker, because with these models, you have to constantly change your approach. Look for engineers familiar with the models you’re using or would like to use. And don’t nail down job titles too tightly, as you need people to be flexible enough to play multiple roles. “Engineers need to stay flexible and be able to do more than one function,” she says.

Principal

At the moment Kay is debating a strategic hiring approach in a CIO leadership group. “Do we need a very experienced developer or do we want someone with perhaps a little less depth but a creative side?” she asks. “Would that combination of creativity and some AI development experience give us way better outcomes? We just don’t know yet.”

The outsourcing option

Of course it’s not always possible to train up or hire the skills needed. “Data scientists, data engineers, and AI engineers are extremely sought after in the market, and it’s difficult to quickly train someone,” says Burgin, so bringing in an external partner can close the skills gap while also give internal team members valuable experience. “Look for partners that can bring the level of proficiency you need to your team,” he adds. “Whenever you have a nascent capability you’re trying to build, bring a third party along to help establish a beachhead of competency,” and to work with and extend the capabilities of your team.

Consultants should then come into play when your team hasn’t done a lot of work in a given area, says Chandrasekaran. And while many organizations are outsourcing to traditional consulting firms such as Deloitte, the model companies, including Hugging Face and OpenAI, are moving into services as well. “They take a product and make sure it’s properly configured and engineered,” he says. “They’re not traditional consultants that bring domain knowledge, but software engineers who can customize and optimize the product for the user.”

There are key differences between using internal versus external AI teams, Redmond says. “Internal teams offer long-term benefits such as in-depth knowledge of company systems, faster iteration, and alignment with business strategies,” she says. “On the other hand, external hosting provides rapid prototyping, niche expertise, and it supplements bandwidth.” Eaton takes a hybrid approach, combining internal subject matter experts and AI skills with external specialists.

Combine good strategy and strong leadership

No team can be effective without solid, focused leadership. Some organizations spend time building up massive data and AI engineering teams, but without real business problems to solve, those resources may devolve into experimentation.

“Don’t build depth and assume the applications will come,” Adams says. So start by partnering with a third party on a few clearly defined projects to build momentum and team skills before scaling it.

 “A clear strategic vision and a communicable pathway to achieve it using impact and measurables are essential,” says Redmond.

Let teams also learn through trial and error. Early on, for example, one of Principal’s AI teams working on a project didn’t understand the data as well as they needed to. The system ended up summarizing things users didn’t need and pulling in irrelevant documents. “It’s important to understand the data and how it’s being managed to avoid bad results,” says Kay. “We realized you have to spend a lot more time understanding the data.”

Regarding reporting structures, a highly effective team doesn’t require having everyone report to the same manager, as long as everyone has the same priorities and commitment to transforming the business, Burgin says. “But you need a cross-functional team that’s working together, and dedicated to that transformation,” he says.

Unfortunately, many companies still have rigid silos, which can mean AI may fall under the exclusive purview of the data analytics team, says Chandrasekaran. In those situations, he adds, there isn’t a lot of collaboration with other teams.

His recommendation, then, is to federate AI teams more because having a centralized AI and data engineering team doesn’t work in many cases. “There are some things you should centralize, but AI and data science should be decentralized into the AI teams to gain domain knowledge,” he says.

Ultimately, it’s up to leadership to set clear priorities. “Think about the 10 parts of the business you want to transform with AI and reimagine that process through AI-powered capabilities from the ground-up,” says Burgin. “For each process, develop an integrated working team that  focusses on how to transform the business and work together on a daily basis despite possibly reporting to different parts of the business.”

Ultimately, don’t only train your AI team. Bring those outside along. “We’re rolling out AI and data literacy to the entire company, and some personas are getting additional training,” says Kay. But most importantly, CIOs should invest in their people as they build AI teams. “It’s really important, especially if AI is a big part of your organization’s roadmap,” she adds. “Your teams can learn quickly, and make a huge contribution.”