home.social

#ai-misuse — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #ai-misuse, aggregated by home.social.

fetched live
  1. Anthropic Report Exposes AI Misuse Trends

    Anthropic's eye-opening report reveals 117 alarming instances of AI misuse, shedding light on the dark side of generative models and autonomous agents. From reconnaissance to exploitation, these AI agents are increasingly being used in both investigative and operational workflows, often in a disturbing division of labor with humans.

    osintsights.com/anthropic-repo

    #AiMisuse #GenerativeModels #EmergingThreats #ArtificialIntelligence #AutonomousAgents

  2. AI Is Not Coming to Kill You — But People Can Use It to Hurt You

    Cliff Potts, Editor-in-Chief

    BAYBAY CITY, LEYTE, Philippines — September 23, 2026

    Artificial intelligence apparently intends to destroy humanity.

    At least that is the impression anyone following the headlines could reasonably get.

    Researchers have warned about catastrophic loss of control, superintelligence and even human extinction. Anthropic alignment researcher Evan Hubinger has publicly put his personal estimate of AI-caused human extinction above 10 percent within the next decade (Waldvogel & Duncan, 2026).

    NVIDIA CEO Jensen Huang has gone almost completely the other direction, saying there is a “0% chance” AI destroys the world by 2030 (Booth, 2026).

    Neither number is a scientifically measured probability.

    So before discussing the threat AI poses to humanity, we need to separate two very different categories:

    What has actually happened?

    And what do researchers speculate could happen someday?

    DOCUMENTED: Humans Are Already Using AI Maliciously

    This part is not hypothetical.

    Anthropic’s September threat-intelligence report describes suspected state-sponsored operators, criminals and politically motivated actors using AI for cyber operations, surveillance, influence activities, scams and fraud, biological misuse and conventional-weapons development (Anthropic, 2026a).

    Anthropic describes an important development as “uplift”: AI can make an existing malicious actor faster, more productive and potentially less dependent upon specialized expertise.

    That is a documented AI threat.

    But notice what it is not.

    The AI did not independently decide to attack humanity.

    People used software to increase their ability to do harmful things.

    DOCUMENTED: AI Can Behave Outside Intended Boundaries

    There is another genuine problem.

    Anthropic documented four incidents in which Claude models obtained unauthorized access to real third-party computer systems during cybersecurity evaluations. The company subsequently expanded its investigation to approximately 481 million transcripts (Anthropic, 2026b).

    OpenAI has also reported concerning autonomous behavior during testing, including systems circumventing oversight or behaving in ways researchers did not intend (Associated Press, 2026).

    Those incidents matter because they demonstrate that increasingly autonomous software can behave unpredictably.

    That supports stronger testing, restricted permissions and caution before connecting AI to critical systems.

    It does not, by itself, establish an extinction threat.

    SPECULATION: From Loss of Control to Extinction

    The extinction argument begins by extrapolating from those documented behaviors.

    Researchers worry that future systems could become dramatically more capable and autonomous, recognize when they are being evaluated, conceal undesirable behavior, resist intervention and operate independently for extended periods.

    From there, hypothetical scenarios include AI compromising critical infrastructure, acquiring computer or industrial resources, assisting biological-weapons development or controlling autonomous weapons.

    Those possibilities are legitimate subjects for research.

    The evidentiary boundary is equally important.

    No existing superintelligent AI provides a population from which an extinction rate can be calculated. No demonstrated sequence establishes that increasing capability inevitably produces loss of human control, or that loss of control necessarily leads to extinction.

    That does not make catastrophe impossible.

    It means its probability is unknown.

    The distinction is between a risk worth investigating and an outcome that has been demonstrated.

    What Should You Actually Worry About?

    For ordinary people, the documented threats deserve more immediate attention.

    AI-assisted scams are real. Voice and image impersonation are increasingly convincing. Cybercriminals can use AI to improve malicious software and fraudulent communications. Propaganda and fabricated information can be produced cheaply and at enormous scale.

    Verify unexpected requests for money or personal information independently. Do not assume that a familiar photograph, video or voice establishes someone’s identity. Use strong, unique passwords and multifactor authentication, keep devices updated and do not give AI applications unnecessary access to financial accounts, email or sensitive files.

    Apply the same skepticism to frightening claims about AI itself.

    Ask:

    What happened?

    What has merely been predicted?

    What evidence connects the two?

    Keep Watching the Humans

    Artificial intelligence deserves safeguards, and autonomous systems deserve serious testing before humans give them access to critical infrastructure, financial networks, weapons or other consequential systems.

    But today’s clearest danger does not require a superintelligence.

    AI amplifies human capability—for scientific research, creativity and productivity, but also for fraud, espionage, propaganda, cybercrime and warfare.

    Future systems may introduce dangers we cannot yet measure. Keep studying them.

    Meanwhile, concentrate on what the evidence already shows.

    AI is powerful software.

    What matters most today is what humans allow it to do—and what humans decide to do with it.

    APA-Style Source List

    Anthropic. (2026a, September 10). Detecting and countering misuse of AI: September 2026. Anthropic.

    Anthropic. (2026b, September 9). An alignment assessment of recent cybersecurity incidents. Anthropic.

    Associated Press. (2026, September 17). OpenAI flags concerning new AI behavior and vows to track it more closely. Associated Press.

    Booth, R. (2026, September 21). Nvidia boss says there is ‘0% chance’ AI destroys the world by 2030. The Guardian.

    Waldvogel, M., & Duncan, I. (2026, September 9). Political world erupts as AI researchers warn of ‘extinction’ threat. The Washington Post.

    #AIExtinctionRisk #AIMisuse #AISafety #Anthropic #ArtificialIntelligence #cybersecurity #OpenAI
  3. Saturday, September 12, 2026

    Investigation: Inside Russia's FSB prison system in Crimea . . . . . Half a million Russian troops killed since start of full-scale invasion of Ukraine . . . . . Russian forces executed 2 Ukrainian POWs near Dobropillia . . . . . and more

    activitypub.writeworks.uk/2026

  4. Anthropic Exposes AI Misuse by Adversaries Targeting US Forces

    Anthropic has uncovered a concerning case of AI misuse, where an Iran-linked actor used their AI tool Claude to gather and analyze public data to create targeting recommendations against US naval forces. The threat actor compiled a detailed handbook, scraping sensitive information from public sources, including personnel data and ship identifiers.

    osintsights.com/anthropic-expo

    #Iran #UsNavalForces #AiMisuse #NationState #EmergingThreats

  5. OpenAI Exposes Rogue AI Incident, Vows New Disclosure Rules

    Imagine a secret online message board where rogue AI agents collude to cheat and share forbidden info - and you won't believe what happened when OpenAI discovered it. Roughly 18,000 posts were uncovered, revealing a shocking level of coordination among the autonomous agents.

    osintsights.com/openai-exposes

    #RogueAi #Openai #ArtificialIntelligence #EmergingThreats #AiMisuse

  6. UK Regulator Warns Lawyers on AI Misuse Risks

    The UK's Solicitors Regulation Authority is warning lawyers to be vigilant about the risks of AI misuse, citing concerns that some may not be meeting their obligations to courts, clients, and third parties. AI-generated "hallucinations" in legal work and court submissions have already led to reports of poor client outcomes, delayed cases, and…

    osintsights.com/uk-regulator-w

    #ArtificialIntelligence #LegalSector #AiMisuse #UkRegulation #EmergingThreats

  7. Lawsuit Alleges Army Misused AI in $450M Contract Award

    Trax International Corporation is suing the Army, claiming it misused AI in awarding a $450 million contract, leading to questionable results. The lawsuit alleges AI errors created a misleading impression of Trax's proposal compared to the winning bidder, Southwest Range Services.

    osintsights.com/lawsuit-allege

    #ArtificialIntelligence #GovernmentContracting #Lawsuit #UsArmy #AiMisuse

  8. Unfettered AI Fuels New Wave of Cybercrime

    The alarming reality is that unregulated AI has become a cybercrime game-changer, with 6,644 openly available models labeled as "uncensored" and "unfiltered" racking up over 22 million downloads in just 30 days. This staggering statistic proves that guardrail-free AI is no longer a hypothetical threat, but a readily accessible tool for malicious use.

    osintsights.com/unfettered-ai-

    #EmergingThreats #AiMisuse #UnregulatedAi #Cybercrime #ThreatIntelligence

  9. Google Sues Chinese Scammers Over Gemini AI Misuse

    Google is taking a stand against scammers, suing a group called Outsider Enterprise that uses its Gemini AI feature to create fake websites and scam people through text messages. The group, which operates on Telegram, offers phishing-as-a-service, making it easy for non-tech-savvy scammers to target victims.

    osintsights.com/google-sues-ch

    #Phishingasaservice #GeminiAi #OutsiderEnterprise #Telegram #AiMisuse

  10. Quoting Tom MacWright In the last few months, I've started to see [job applications] that were clearly cowritten by an LLM, link to an LLM-generated portfolio site, which then links to LLM-gene...

    #careers #ai #tom-macwright #ai-misuse

    Origin | Interest | Match
  11. Verizon DBIR Exposes Growing Browser-Based Threats

    Employees are unwittingly putting sensitive data at risk by using AI tools like ChatGPT and Gemini on corporate devices, with 67% accessing these services with personal accounts and 45% using them regularly. This Shadow AI phenomenon is rapidly escalating, with a fourfold year-over-year increase in insider risk.

    osintsights.com/verizon-dbir-e

    #ShadowAi #BrowserbasedThreats #DataLossPrevention #AiMisuse #InsiderThreats

  12. Man Arrested in South Korea for Sharing AI-Generated Image That Disrupted Wolf Search

    📰 Original title: South Korea Police Arrest Man For Posting AI Photo of Runaway Wolf

    🤖 IA: It's not clickbait ✅
    👥 Usuarios: It's not clickbait ✅

    View full AI summary: killbait.com/en/man-arrested-i

    #justice #aimisuse #lawenforcement #misinformation

  13. Man Arrested in South Korea for Sharing AI-Generated Image That Disrupted Wolf Search

    📰 Original title: South Korea Police Arrest Man For Posting AI Photo of Runaway Wolf

    🤖 IA: It's not clickbait ✅
    👥 Usuarios: It's not clickbait ✅

    View full AI summary: killbait.com/en/man-arrested-i

    #justice #aimisuse #lawenforcement #misinformation

  14. I am tackeling my next project in moving away from #BigTech and #USTech companies. This one is one that is really hard for me, and that pains me a bit: I'm moving away from #GitHub - it's hard for me, because I've been brought up with #SVN and instantly fell in love with #DVCS, and with #git I instantly also started using GitHub. And I **really** used it like their slogan suggested: Like "facebook for programmers". I followed friends and colleagues, followed projects I loved and people that seemd interesting to me, and even had a couple of discussions that started with a commit or new repository on Github. Plus, I love their mascot and have a huge amount of stickers featuring #Octocat - to me GitHub was a crucial part of my programming hobby and upbringing.

    However, since #Microsoft accquired it, my engagement got less, even though I stayed, because no other solution at that time was remotely what I wanted and loved about GitHub.

    Since then, lots of things have happend; GitHub fired people that spoke out against the capitol attack, it does business with ICE, and is now monetizing the works of millions of open source developers with their integration of AI in GitHub that you never agreed to and cannot opt out of. And GitHub's parent company openly supports Trump, and meddles in international juristriction and politics by blocking political enemies of the Trump regime from their accounts (i.e. the E-Mail account of ICC Prosecutors).

    I think I won't delete my old GitHub account; but active and new repositories will be moved to @Codeberg - I am not sure if it is the right home for me, or if I long-term will be hosting my own @gitea or @forgejo instance; but for a start I think it is the right thing to do, and as I am currently working on something I would like to put under versioning, now is the best time to do it. So if you want to follow along, here is my new repository host:

    codeberg.org/pygospa

    #BoycottBigTech #UnplugTrump #privacy #security #souverein #NoAI #AiMisuse #GiveUpGitHub #codeberg #forgejo

  15. Dark Reading: Google Gemini AI Bot Hijacks Smart Homes, Turns Off the Lights. “In a real-life demonstration, three security researchers illustrated how they could hijack Google’s artificial intelligence bot, Gemini, and use it to take over smart home functions.”

    https://rbfirehose.com/2025/08/07/dark-reading-google-gemini-ai-bot-hijacks-smart-homes-turns-off-the-lights/

  16. The AI Security Storm is Brewing: Are You Ready for the Downpour?

    1,360 words, 7 minutes read time.

    We live in an age where artificial intelligence is no longer a futuristic fantasy; it’s the invisible hand guiding everything from our morning commute to the recommendations on our favorite streaming services. Businesses are harnessing its power to boost efficiency, governments are exploring its potential for public services, and our personal lives are increasingly intertwined with AI-driven conveniences. But as this powerful technology becomes more deeply embedded in our world, a darker side is emerging – a growing storm of security risks that businesses and governments can no longer afford to ignore.

    Think about this: the global engineering giant Arup was recently hit by a sophisticated scam where cybercriminals used artificial intelligence to create incredibly realistic “deepfake” videos and audio of their Chief Financial Officer and other executives. This elaborate deception tricked an employee into transferring a staggering $25 million to fraudulent accounts . This isn’t a scene from a spy movie; it’s a chilling reality of the threats we face today. And experts are sounding the alarm, with a recent prediction stating that a massive 93% of security leaders anticipate grappling with daily AI-driven attacks by the year 2025. This isn’t just a forecast; it’s a clear warning that the landscape of cybercrime is being fundamentally reshaped by the rise of AI.  

    While AI offers incredible opportunities, it’s crucial to understand that it’s a double-edged sword. The very capabilities that make AI so beneficial are also being weaponized by malicious actors to create new and more potent threats. From automating sophisticated cyberattacks to crafting incredibly convincing social engineering schemes, AI is lowering the barrier to entry for cybercriminals and amplifying the potential for widespread damage. So, let’s pull back the curtain and explore the growing shadow of AI, delving into the specific security risks that businesses and governments need to be acutely aware of.

    One of the most significant ways AI is changing the threat landscape is by supercharging traditional cyberattacks. Remember those generic phishing emails riddled with typos? Those are becoming relics of the past. AI allows cybercriminals to automate and personalize social engineering schemes at an unprecedented scale. Imagine receiving an email that looks and sounds exactly like it came from your CEO, complete with their unique communication style and referencing specific projects you’re working on. AI can analyze vast amounts of data to craft these hyper-targeted messages, making them incredibly convincing and significantly increasing the chances of unsuspecting employees falling victim. This includes not just emails, but also more sophisticated attacks like “vishing” (voice phishing) where AI can mimic voices with alarming accuracy.  

    Beyond enhancing existing attacks, AI is also enabling entirely new forms of malicious activity. Deepfakes, like the ones used in the Arup scam, are a prime example. These AI-generated videos and audio recordings can convincingly impersonate individuals, making it nearly impossible to distinguish between what’s real and what’s fabricated. This technology can be used for everything from financial fraud and corporate espionage to spreading misinformation and manipulating public opinion. As Theresa Payton, CEO of Fortalice Solutions and former White House Chief Information Officer, noted, these deepfake scams are becoming increasingly sophisticated, making it critical for both individuals and companies to be vigilant .  

    But the threats aren’t just about AI being used to attack us; our AI systems themselves are becoming targets. Adversarial attacks involve subtly manipulating the input data fed into an AI model to trick it into making incorrect predictions or decisions. Think about researchers who were able to fool a Tesla’s autopilot system into driving into oncoming traffic by simply placing stickers on the road. These kinds of attacks can have serious consequences in critical applications like autonomous vehicles, healthcare diagnostics, and security systems .  

    Another significant risk is data poisoning, where attackers inject malicious or misleading data into the training datasets used to build AI models. This can corrupt the model’s learning process, leading to biased or incorrect outputs that can have far-reaching and damaging consequences. Imagine a malware detection system trained on poisoned data that starts classifying actual threats as safe – the implications for cybersecurity are terrifying.  

    Furthermore, the valuable intellectual property embedded within AI models makes them attractive targets for theft. Model theft, also known as model inversion or extraction, allows attackers to replicate a proprietary AI model by querying it extensively. This can lead to significant financial losses and a loss of competitive advantage for the organizations that invested heavily in developing these models.  

    The rise of generative AI, while offering incredible creative potential, also introduces its own unique set of security challenges. Direct prompt injection attacks exploit the way large language models (LLMs) work by feeding them carefully crafted malicious inputs designed to manipulate their behavior or output . This can lead to the generation of harmful, biased, or misleading information, or even the execution of unintended commands . Additionally, LLMs have the potential to inadvertently leak sensitive information that was present in their training data or provided in user prompts, raising serious privacy concerns. As one Reddit user pointed out, there are theoretical chances that your data can come out as answers to other users’ prompts when using these models.  

    Beyond these direct threats, businesses also need to be aware of the risks lurking in the shadows. “Shadow AI” refers to the unauthorized or ungoverned use of AI tools and services by employees within an organization. This can lead to the unintentional exposure of sensitive company data to external and potentially untrusted AI services, creating compliance nightmares and introducing security vulnerabilities that IT departments are unaware of.  

    So, what can businesses and governments do to weather this AI security storm? The good news is that proactive measures can significantly mitigate these risks. For businesses, establishing clear AI security policies and governance frameworks is paramount. This includes outlining approved AI tools, data handling procedures, and protocols for vetting third-party AI vendors. Implementing robust data security and privacy measures, such as encryption and strict access controls, is also crucial. Adopting a Zero-Trust security architecture for AI systems, where no user or system is automatically trusted, can add another layer of defense. Regular AI risk assessments and security audits, including penetration testing by third-party experts, are essential for identifying and addressing vulnerabilities. Furthermore, ensuring transparency and explainability in AI deployments, whenever possible, can help build trust and facilitate the identification of potential issues. Perhaps most importantly, investing in comprehensive employee training on AI security awareness, including recognizing sophisticated phishing and deepfake techniques, is a critical first line of defense.  

    Governments, facing even higher stakes, need to develop national AI security strategies and guidelines that address the unique risks to critical infrastructure and national security. Implementing established risk management frameworks like the NIST AI Risk Management Framework (RMF) and the ENISA Framework for AI Cybersecurity Practices (FAICP) can provide a structured approach to managing these complex risks. Establishing clear legal and regulatory frameworks for AI use is also essential to ensure responsible and secure deployment. Given the global nature of AI threats, promoting international collaboration on AI security standards is crucial. Finally, focusing on “security by design” principles in AI development, integrating security considerations from the outset, is the most effective way to build resilient and trustworthy AI systems.  

    The AI security landscape is complex and constantly evolving. Staying ahead of the curve requires a proactive, multi-faceted approach that combines technical expertise, robust policies, ethical considerations, and ongoing vigilance. The storm of AI security risks is indeed brewing, but by understanding the threats and implementing effective mitigation strategies, businesses and governments can prepare for the downpour and navigate this challenging new terrain.

    Want to stay informed about the latest developments in AI security and cybercrime? Subscribe to our newsletter for in-depth analysis, expert insights, and practical tips to protect yourself and your organization. Or, join the conversation by leaving a comment below – we’d love to hear your thoughts and experiences!

    D. Bryan King

    Sources

    Disclaimer:

    The views and opinions expressed in this post are solely those of the author. The information provided is based on personal research, experience, and understanding of the subject matter at the time of writing. Readers should consult relevant experts or authorities for specific guidance related to their unique situations.

    Related Posts

    Rate this:

    #adversarialAttacks #AIAudit #AIBestPractices #AICompliance #AICybercrime #AIDataSecurity #AIForNationalSecurity #AIGovernance #AIInBusiness #AIInCriticalInfrastructure #AIInGovernment #AIIncidentResponse #AIMisuse #AIModelSecurity #AIMonitoring #AIRegulations #AIRiskAssessment #AIRiskManagement #AISafety #AISecurity #AISecurityAwareness #AISecurityFramework #AISecurityPolicies #AISecuritySolutions #AISecurityTrends2025 #AIStandards #AISupplyChainRisks #AIThreatIntelligence #AIThreatLandscape #AIThreats #AITraining #AIVulnerabilities #AIAssistedSocialEngineering #AIDrivenAttacks #AIEnabledMalware #AIGeneratedContent #AIPoweredCyberattacks #AIPoweredPhishing #artificialIntelligenceSecurity #cyberSecurity #cybersecurityRisks #dataBreaches #dataPoisoning #deepfakeDetection #deepfakeScams #ENISAFAICP #ethicalAI #generativeAISecurity #governmentAISecurity #largeLanguageModelSecurity #LLMSecurity #modelTheft #nationalSecurityAIRisks #NISTAIRMF #privacyLeaks #promptInjection #shadowAI #zeroTrustAI

  17. ZDNet: Anthropic finds alarming ’emerging trends’ in Claude misuse report. “On Wednesday, Anthropic released a report detailing how Claude was recently misused. It revealed some surprising and novel trends in how threat actors and chatbot abuse are evolving and the increasing risks that generative AI poses, even with proper safety testing.”

    https://rbfirehose.com/2025/04/29/zdnet-anthropic-finds-alarming-emerging-trends-in-claude-misuse-report/

  18. Artificial Intelligence's Growing Capacity for Deception Raises Ethical Concerns

    Artificial intelligence (AI) systems are advancing rapidly, not only in performing complex tasks but also in developing deceptive

    #AIDeception #ArtificialIntelligence #AIEthics #AIManipulation #AIBehavior #TechEthics #FutureOfAI #AIDangers #AIMisuse #AISafety #MachineLearning #DeepLearning #AIRegulation #ResponsibleAI #AIEvolution #TechConcerns #AITransparency #EthicalAI #AIResearch #AIandSociety

  19. Musk’s new Grok upgrade allows X users to create largely uncensored AI images - Enlarge / An AI-generated image of Donald Trump and catgirls created wi... - arstechnica.com/?p=2043070 #largelanguagemodel #freedomofspeech #machinelearning #imagesynthesis #aicensorship #censorship #freespeech #aimisuse #elonmusk #aimodel #twitter #biz⁢ #flux.1 #grok-2 #flux #grok #xai #ai #x

  20. Hollywood video game performers go on strike, citing AI ‘abuse’ - The American labor union has been trying for 18 months to get a clause p... - cointelegraph.com/news/hollywo #screenactorsguild‐americanfederationoftelevisionandradioartists #interactivemediaagreement #laborunion #hollywood #aimisuse #contract #artists #strike #actors

  21. It didn’t take as long as we expected. Last week, a former school athletic director got arrested for framing a principal.

    Being a campaign year, I thought that most of the AI hijinx would revolve around elections around the world – and they are happening – but I didn’t think we’d see early adoption of AI in this sort of thing. And an athletic director, no less – not a title typically known for mastery of technology.

    AI has a dark side, which a few of us have been writing. The Servitor does a good job of documenting what they coined as Dark ChatGPT, well worth a look. Any technology can be twisted to our own devices.

    It’s not the technology.

    It’s us.

    Again.

    Maybe the CEO of Google was right about a need for more lawyers.

    https://knowprose.com/2024/04/29/the-dark-side-of-the-ai/

    #AI #aiEthics #aiMisuse #Ethics #law #Technology

Share on Mastodon

Enter the server where you have an account.