Hyderabad’s Commissioner of Police, V.C. Sajjanar in an X (formerly Twitter) post today has issued a warning about the evolution of artificial intelligence (AI) agents into agentic AI systems. This has been reported by Dainik Bhaskar and Siasat Daily, but we haven’t been able to confirm it independently. We tried contacting Hyderabad Police for a confirmation, but there was no response on their numbers.
Sajjanar says that AI chatbots that until recently only answered questions or wrote text have now evolved into agentic AI systems: which are AI systems capable of taking decisions and executing tasks on their own. Key points that he has made:
1. Risk of losing control over autonomous AI agents: “Autonomous robot agents have entered highly critical sectors such as banks, hospitals, and power grids. However, with these digital agents performing tasks independently without human intervention, there is widespread concern that we are at risk of losing control over them”.
2. Agents can act without being directed to: “For instance, if you ask a regular AI, ‘How is the weather outside?’, it merely provides information. But Agentic AI is different; realising the weather is bad, it might close the windows itself and adjust the AC temperature” Sajjanar added.
“Freezing accounts if suspicious transactions occur in the financial sector, altering medication dosages based on a patient’s BP (blood pressure) and sugar levels in hospitals, and controlling machine performance in industries—these [agentic AI] agents are doing such things on their own,” Sajjanar warned.
3. Potential consequences of mistakes by agentic AI: “This autonomy has now become a major challenge. These agents, which take thousands of decisions in minutes without human supervision, can lead to massive danger if they make even a small mistake anywhere.”
“For example, if an AI agent makes a wrong decision in the stock market, losses worth crores could occur in mere seconds,” Sajjanar explains. “Similarly, there is a threat of cybercriminals hijacking the behaviour of these agents and forcing them to commit wrongdoings,” he adds.
“There is also the risk of ‘Unintended Behaviour,’ where the agent misunderstands the task assigned by the owner and takes a wrong path to achieve the goal,” Sajjanar writes on X (formerly Twitter).
4. Digital identity for AI agents? “Every AI agent must have a precise ‘Digital Identity.’ Just as a human has an ID card in an organisation, these software agents must also have identification,” he says.
“Which agent opened which file? When did it make changes? To whom did it send information? Every such movement must be recorded (Logging). Because of this, if an accident happens by mistake, we can immediately identify which agent caused it and rectify the issue.”
5. Governance rules and ‘Trust Frameworks’ the need of the hour: “Ultimately, there must be strict rules (Governance) regarding how much freedom these (agentic AI) agents should be given and what data they are allowed to access. ‘Trust Frameworks’ must be designed to ensure human permission is sought before making crucial decisions.”
“While speeding up work through technology is important, security is even more critical. Tech experts warn that if we do not wisely control the speed of automation, the losses it brings will be greater than the benefits.”
Also Read:
Support our journalism by subscribing
For YouSource link