Peeking Inside the Black Box How AI Labs in Malaysia Are Actually Built

Nowadays, we frequently hear about the expression “AI Lab”. Whether it be from technology companies establishing facilities, universities constructing buildings, or even entrepreneurs operating out of Cyberjaya. They all sound pretty cool — they all sound very “canggih.” However, if you were to walk inside one of those facilities (e.g., the recently opened NAIO Lab unveiled by the government, or a private lab like Daythree AI Labs) what would you actually find? Would it simply be a roomful of hoodie-wearing, code-sharing guys? Or would there be some kind of clarity?
In reality, AI research lab structure is a lot like making the ideal teh tarik. You have to have the right balance of complementary components. If you have too much foam (the marketing component), and no tea (the actual compute), then the creation will collapse. On the other hand, if you have bitter tea (just pure theory), but no sugar (just pure practice) then nobody is going to consume it. So let’s take a closer look at how these facilities have been organized in the local area around KL, as well as throughout the region.

The Two Tribes: Dreamers vs. Builders

In a successful AI implementation, one of the first similarities they will notice is their division. It isn’t just one large unified team but more of two diverse cultures working together yet speaking in a common tongue. The Core AI research teams are one of the factions. These researchers typically find pleasure in reading and understanding various literature in data science. They also have an interest in researching ways of developing a model by using as little data as possible or coming up with innovative ways to explore language processing. This group, though, typically performs their research with no immediate or obvious financial return. They are conducting basic science research, akin to typical research processes—slow.

On the other side of the division from the core researchers, there are the Applied AI project teams. These individuals take the approach of looking at a timeline to help farmers in Terengganu or determining the best way of automating customer service replies. They have a more direct focus or use of this research in the marketplace. The degree and tension between these two teams have. The Core team’s interest in perfection and the Applied team’s focus on delivery. Good labs will have a combination of teams. They are just typically on different floors. If you only have Core, you will eventually die with no products and, therefore, no way of generating revenue. If you only have Applied, you will eventually die because you cannot innovate and exceed your competition. Your lab requires that level of tension.

AI research lab structure is All About the “Rigs” and the “Pipes”

The part that most people find surprising is how much money is tied up in hardware & computing resources in an AI lab. Most people assume that the person with the Ph.D. is the single biggest “asset”. In an AI lab, the vast majority of costs stem from purchasing hardware and compute resources. “Rigs” in the form of clusters of NVIDIA GPUs (the same type of GPUs used in University labs, but hundreds at a time), with just the cost of running one training session for a “big” model in terms of electricity alone costing thousands of RM. However, without data infrastructure & management, no hardware or computing resource is worth anything. The boring and unexciting likes of data infrastructure & management make up 80% of your work in an AI lab. You cannot simply “throw” raw text at an AI model and “hope” it learns.

It is very much like cooking. You have to wash the vegetables (clean the data), chop them (label the data), and store them (database management). As I mentioned previously, Malaysia presents specific challenges in creating AI models that can understand “Manglish”. For example, or that know the definition of “tapau.” These types of datasets are not available in global datasets. So, a majority of the data ops/Data operations in an AI lab are dedicated to cleaning up, organizing and formatting this type of localized data to enable the AI machine to learn from our culture.

AI research lab structure No Lab is an Island

Building data centres in isolation will not work because large companies. Such as Google or Microsoft partner with universities. Recently, the local universities have partnered with companies like SDS AI Lab, while institutions like UCSI have created dedicated AI lab environments with dedicated GPUs. This collaboration with the academic sector is one of the most common ways to recruit talent and have that talent learn on industry problems. Laboratories are constantly looking for new graduates that have knowledge of developing software through the TensorFlow or PyTorch frameworks. In addition, universities are looking for ways that their students can solve real-world problems that they will face on the job, as well as additional funding for the laboratory. The additional funding is usually found in the partnership between the private and academic sectors. Since the pure research laboratory can be repeatedly referred to as a cash drain, the government will be providing a boost of almost RM5.9 billion to R&D in the upcoming Budget 2026.

However, the vast majority of the money that comes from private sector partnerships will be utilized almost immediately. The most ideal example of this cooperation is the recent restructuring of Daythree AI Labs. By separating the business operations from their AI innovation unit, they can avoid the immediate pressure to generate profits and use their tools for improving their existing business. It has turned out to be a win-win situation for both parties. Additionally, there has been some interesting cross-border activity surrounding the “China-ASEAN AI Lab”. Where major technology companies like DeepSeek and Huawei are collaborating to take their technology to market in Southeast Asia. These technology companies and universities are also creating a unique strategy to build a new product by borrowing, adapting, and developing that technology for the Asia-Pacific market.

Who Watches the Watchmen?

Research governance and oversight is the last piece of the puzzle. While it may not be the most exciting, it is important. Artificial Intelligence (AI) possesses the potential to cause harm to individuals, businesses and society as a whole when used incorrectly. If your lab creates a chatbot, and it gives bad financial advice or racist comments, you are liable for the consequences. Therefore, there exists a layer of governance which has the responsibility to check for biases in the chatbot and ensure it is not creating false information. In addition, there is also the issue of intellectual property (IP) ownership. For example, if you used publicly available datasets to train the chatbot, who owns the resulting output? If an intern makes a modification to the code of the chatbot that results in a 5% improvement in performance, does that intern own that line of code? Lawyers love these issues whereas they are a headache for engineers. An example of governance of AI in the government sector is the National AI Office (NAIO). The NAIO strives to draft and establish standard guidelines for using and creating AI in Malaysia. It to ensure that any AI that is created in Malaysia is safe, does not go outside the borders of Malaysia (data is sovereign) and ethically created.

The Future is Hybrid

What does this mean? The traditional AI research lab, where academics worked independently creating code, and passed it to engineers to implement, is no longer the norm. Today’s AI labs, such as the EY AI Innovation Centre and the NAIO Lab in Malaysia, will have an increasingly hybrid structure. Funding streams for these labs, including EY’s and NAIO’s, are more diversified with sources coming from both government-granted funds through budget appropriations, as well as from private equity. Staffing in these labs will similarly be diversified, with contributions being made by both new graduates and seasoned scientists. The objectives of the labs will also be hybrid in nature – combining the goal of publishing research papers with delivering production-quality software.

If you are looking to begin your career within an AI lab, you will want to think beyond just acquiring programming skills. Understand how your lab’s operations intersect with those of the lab’s data management function. As the data management function is as critical to the laboratory’s success as is the delivery of algorithmic solutions. Ultimately, remember that an AI lab is not a magical place that creates solutions from pure research. But rather consists primarily of teams of individuals working together on very powerful computing platforms to bring order to a chaotic and nebulous environment.