{"id":29001,"date":"2024-06-11T12:00:21","date_gmt":"2024-06-11T16:00:21","guid":{"rendered":"https:\/\/netfoundry.io\/?p=29001"},"modified":"2024-09-17T22:08:23","modified_gmt":"2024-09-18T02:08:23","slug":"llm-security-development-of-ai-infused-apps","status":"publish","type":"post","link":"https:\/\/netfoundry.io\/ai\/llm-security-development-of-ai-infused-apps\/","title":{"rendered":"LLM Security \u2013 Development of AI-infused apps"},"content":{"rendered":"\n

Securing LLM environments is hard.<\/h3>\n\n\n\n

It involves challenges such as protecting data privacy and confidentiality, implementing robust access control and authentication, ensuring network and infrastructure security, maintaining model integrity against tampering and adversarial attacks, establishing effective monitoring and incident response plans, and balancing security measures with performance and scalability. These measures are crucial to prevent unauthorized access, data breaches, and performance issues while complying with data protection regulations\u200b<\/p>\n\n\n\n

This article is a case study from an R&D team of a large enterprise using a large language model (LLM) from their users\u2019 mobile, web, and thick client apps, without exposing any data or metadata to any network.  Specifically, an Azure OpenAI powered chatbot with APIs to iPhone (iOS) mobile apps, without their Azure VNet being open to any inbound data (so the firewall is set to deny all inbound).  Consider it a sneak preview \u2013 the full case study will be published next.<\/p>\n\n\n\n

Our customer\u2019s requirements went beyond security.  They included:<\/p>\n\n\n\n

    \n
  1. No VPNs, whitelisted IPs, or MPLS dependencies.

    <\/li>\n\n\n\n
  2. No touching the (B2B and B2B2C) mobile phones.

    <\/li>\n\n\n\n
  3. No backhauling all the app traffic \u2013 different microservices and APIs to different destinations, directly from the mobile app.<\/li>\n<\/ol>\n\n\n\n


    Spoiler alert:<\/strong> all requirements met, with an open source-based, software-only solution\u2026with the proof of concept (POC) done in one day.

    If you have requirements like this, then this case study should be worth reading.  Otherwise, this post will be boring.<\/p>\n\n\n\n\n\n\n

    The AI and LLM Security dilemma<\/strong><\/p>\n\n\n\n

    The astounding progress of Large Language Models (LLMs) is creating two opposing forces:<\/p>\n\n\n\n

      \n
    1. We want to leverage the Artificial Intelligence (AI) capabilities in our apps, querying public LLMs (e.g. ChatGPT, Azure OpenAI) and self-hosted LLMs (e.g. Llama, Claude).

      <\/li>\n\n\n\n
    2. We need privacy and security. The data and prompts can contain sensitive, competitive or valuable information, including customer information.<\/li>\n<\/ol>\n\n\n\n


      We have seen this movie before.  In fact, the security force is why our internal enterprise apps are only available via private networks such as our WANs or VPNs.  By shielding our internal enterprise apps, APIs, and data from the Internet, we minimize the attack surface.  Unfortunately, many of the apps that we want to use AI can\u2019t be forced onto our WAN or VPN (and their connections to the cloud such as ExpressRoute or Direct Connect), or doing so would add too much latency, failure points or user experience problems.<\/p>\n\n\n\n

      Solving the AI and LLM Security dilemma<\/strong><\/p>\n\n\n\n

      So, we need a solution to enable our apps and APIs to access the LLMs, but without exposing any of the data to the Internet, and without needing to navigate MPLS or VPN.  Actually, this is exactly what NetFoundry\u2019s OpenZiti (open source) and NetFoundry Cloud (SaaS)  already do for billions of sessions per year \u2013 the LLM on the server side doesn\u2019t make a difference to Ziti.  Ziti enables us to:<\/p>\n\n\n\n

        \n
      1. Minimize the threat surface \u2013 prevent attacks from the Internet.

        <\/li>\n\n\n\n
      2. Secure the road out too \u2013 help prevent data exfiltration.

        <\/li>\n\n\n\n
      3. Provide great UX \u2013 no VPN or MPLS backhaul.

        <\/li>\n\n\n\n
      4. Simplify operations\u2014no complex ACLs, WAF configurations, DNS dependencies, or other day two, bolted-on, infrastructure-dependent, cloud-specific security add-ons.
        <\/li>\n<\/ol>\n\n\n\n


        In other words, the Ziti platform enables us to get both security and simplicity.  In this case, use LLMs without network exposure and without VPN.  It is the same Ziti platform that leaders such as Microsoft<\/a>, LiveView<\/a>, and Intrusion<\/a> use to securely deliver billions of sessions per year for use cases such as:<\/p>\n\n\n\n