Category Archive - DevOps - NetFoundry https://netfoundry.io/category/devops/ Identity-First™ Networking Thu, 26 Feb 2026 14:59:14 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://netfoundry.io/wp-content/uploads/2024/08/netfoundry-icon-color-150x150.png Category Archive - DevOps - NetFoundry https://netfoundry.io/category/devops/ 32 32 Breakneck speed without breaking our necks https://netfoundry.io/devops/devops-meets-secops/ Thu, 26 Feb 2026 14:42:40 +0000 https://netfoundry.io/?p=46637 Why Every DevOps Engineer Should Be Using Ziti Velocity vs security – why can we not have both? After working in the DevOps space for almost a decade, there are a few common traits that you’ll find among most of us: Over the last decade we’ve seen an explosion in technology that allows us to […]

The post Breakneck speed without breaking our necks appeared first on NetFoundry.

]]>
Why Every DevOps Engineer Should Be Using Ziti

Velocity vs security – why can we not have both?

After working in the DevOps space for almost a decade, there are a few common traits that you’ll find among most of us:

  1. We don’t actually know what we do for a living
  2. We love to automate EVERYTHING
  3. We wire systems together and make things work
  4. We don’t like administrating systems – we build systems to administrate other systems

Over the last decade we’ve seen an explosion in technology that allows us to automate and orchestrate on a level that’s never been seen before. We develop faster, deploy faster, we innovate. But in a world that has an ever-increasing need for security, we are also the troublemakers.

In order to wire systems together, we need access to EVERYTHING. Even more than that, we can grant incredible super powers to the automation systems that we build. We create systems that are absolute goldmines for hackers to exploit. Take any of the big technology names in the DevOps space, and imagine the devastation that an exploit can bring if one of these systems becomes compromised. Salt, Kubernetes, Jenkins, Ansible, your Data Warehouse. Something they all have in common – they all have incredible access grants within systems, and they all have access to a disturbing amount of data if it falls into the wrong hands.

Since removing access to these systems is not feasible, how do we continue to build tools that allow us to move at a breakneck speed without compromising our security and creating a massive attack vector? 

Enter Ziti! 

OpenZiti (I use Ziti and OpenZiti synonymously throughout this blog) is an open-source, software-only programmable network overlay based on zero-trust principles.

The Ziti mascot (a yellow ziti noodle) as an old west sherif and as a ninja, along with the OpenZiti logo.

As a DevOps Engineer, I like tools that act like a Swiss Army Knife. Anything that solves a variety of problems and allows me to continue to move quickly makes it into my repertoire and becomes a tool that I come back to over and over again, regardless of where I’m working. OpenZiti provides a way for you to set up access controls quickly and efficiently while raising the standard for “least permissions” at the network level. Read more about Ziti here or Ziggy, our mascot for OpenZiti and his many different outfits here.

Lock Down Your Tools

To make my life easier, I use the NetFoundry platform to provide a cloud orchestrated, programmable operation of OpenZiti – spun up and down minutes based on our needs. NetFoundry is the creator of Ziti, maintains it and can provide a SaaS option for anyone.

Locking down critical resources with Ziti, using NetFoundry-hosted overlay fabric.

If you’re reading this, I probably don’t need to explain the terrifying compromises that happened in the last year with Solarwinds or TravisCI, but just in case you didn’t, you should go read those articles now and then you’ll understand perfectly why this next topic is critical. A CI/CD system is a perfect example of a “system managing a system”. It typically has elevated access and is designed to deploy and execute code in all of the places you care about. Hackers know this, they target these systems and dream about deploying and executing their own code inside of your systems.Knowing that hackers are looking for endpoints just like these would often keep me up at night. We can’t afford to leave our critical systems exposed anymore, we need to make them dark. The methodology for securing any of your sensitive resources is the same whether it’s a data warehouse, a build server, or an API. 

  • Shut down your traditional ingress ports
  • Place an Edge Router (cloud orchestrated software) inside of the private network space
  • Grant service access to only the endpoints that need it 

Seamless Switchover

When I think about “locking things down”,this is typically synonymous to all kinds of breakage and end-user disruption. When doing this sort of work in the past it’s very difficult to validate success short of waiting for users to complain. However, NetFoundry’s console paired with Ziti makes this incredibly easy.  I could see if the project was going to be successful *before* making the cutover. Ziti’s traffic intercept will work whether a service is dark or not. Enroll your end-users, watch the traffic come into the Console, and verify everything ahead of time. After running two migrations with this method, switchover day was a non-event both times because users had already been accessing the services with Ziti for weeks.

Application dashboard showing network utilization with filters, charts, and graphs.

Developer, DevOps and NetOps Access

If you’re putting on your security hat, this phrase alone should make you cringe. As nice as it would be to keep access locked down once you’re in production, sometimes things go wrong, and you need to let your developers log into instances, databases, and message brokers to solve a problem. Most companies don’t like to admit it, but too many are still using shared credentials to interact with their application dependencies, so when it comes to assigning developers secure access, this creates a security nightmare. These types of resources generally live inside of a private VPC or DC, so opening them up to a developer often means exposing the entire VPC to that user. In a “least permissions” world, this is not ideal. What Ziti allows me to do is isolate that network access and separate access by application, team, or environmenti.e. account-based access-control.

Isolating developer access within a VPC or DC: a complex diagram with billing, infrastructure components connected by NetFoundry-hosted fabric.

Step Up Your Visibility 

Have you ever had to determine who accessed a resource based on network traffic? Just in case you haven’t, it’s terrible. Traditional network traffic monitoring is built around IPs and Ports, and from that information you can maybe extract the “who” and the “what”. As a result, in a previous role, I once spent 3 days hunting down the fact one of our employees had gone to work from a coffee shop! NetFoundry introduces a whole new generation of network visibility, where all traffic monitoring operates around trusted identities (endpoints) and services (user-defined slices of traffic). For every byte of traffic that passes over OpenZiti, we know who initiated the traffic, what service is being accessed, and what time the traffic was passed. No additional interpretation is required.

Application dashboard showing network utilization with filters, charts, and graphs.

Zero trust journey

To make our journey to zero trust DevOps as seamless and non-event as possible, we decided to go through an incremental, evolutionary series of steps. We started with our data warehouses before securing our Jenkins CI/CD pipeline. Next, we moved to ‘dark bastions’ by applying OpenZiti to them and closing inbound port 22 – auditing access became so much easier and less cumbersome. To come, we will make all internal system communications dark by applying NetFoundry to our ETL jobs and database to CI/CD connections.

I don’t know what journey you will take with OpenZiti and NetFoundry; that’s your next decision.

More info

Most of us in the security, tech and networking worlds are cynics, and for good reason.  Meanwhile, the marketeers have flocked to use and abuse the Zero Trust terms.  So here are links to enable you to quickly filter through the noise and judge for yourself if you want to learn a bit more before diving into the SaaS or open source:

The post Breakneck speed without breaking our necks appeared first on NetFoundry.

]]>
Accelerating DevOps with Zero Trust: Moving Faster Without Sacrificing Security https://netfoundry.io/devops/accelerating-devops-with-zero-trust-moving-faster-without-sacrificing-security/ Thu, 07 Nov 2024 13:16:03 +0000 https://netfoundry.io/?p=41816 If you’ve ever been in the trenches of DevOps, you know the drill: move fast, maintain reliability, and—somehow—don’t break security in the process. I’ve been doing this for over a decade, and I’ve lost count of the number of times my team and I had to juggle priorities, feeling the pressure of meeting deadlines while […]

The post Accelerating DevOps with Zero Trust: Moving Faster Without Sacrificing Security appeared first on NetFoundry.

]]>

If you’ve ever been in the trenches of DevOps, you know the drill: move fast, maintain reliability, and—somehow—don’t break security in the process. I’ve been doing this for over a decade, and I’ve lost count of the number of times my team and I had to juggle priorities, feeling the pressure of meeting deadlines while knowing every open port or IP address could be an attack vector waiting to happen. Here’s how I’m finally able to move faster and sleep better, thanks to the zero trust connectivity we’ve built at NetFoundry with Ziti.

The Real DevOps Struggle: Moving Fast Without Creating Vulnerabilities

DevOps isn’t just about building and deploying software; it’s about creating connectivity between systems: development environments, production servers, monitoring tools, CI/CD systems, data warehouses, even different cloud providers. Every time we “connect” something, we open another door for attackers to get in. Over the years, I’ve seen VPNs, VPC peering, MPLS connections, you name it. All of these solutions are SLOW to implement, difficult to manage, and they all have the common problem of heavily exposing one site to another the moment you connect them together. 

Take VPNs, for instance. They’re great for creating a secure tunnel, but they’re still just a tunnel—everything on both ends is exposed. When you start adding multiple cloud regions or VPCs into the mix, you end up with a tangled mess of network configurations that’s a pain to secure and scale. And don’t even get me started on managing IPs, firewall rules, and access controls across clusters in Kubernetes. It was clear: if we were going to get serious about DevOps at scale, we needed a different approach.

Why Traditional Networking Slows DevOps Down

To connect our various systems, we were constantly having to open ports, configure firewalls, or set up VPNs to create the connectivity that we needed. Traditional network security is built around static systems that don’t change, but that’s not the world we live in anymore. Every time we open a port or punch a hole in a firewall, we introduce risk. As the Head of Devops, I had to ensure that the developers had access to the resources they needed without opening the entire production environment to everyone. It felt like every tool we added—monitoring, CI/CD, data management—added another layer of complexity and potential exposure.

Some key challenges I kept hitting were:

  1. VPN Limitations: VPNs made us feel secure, but they slowed everything down and weren’t flexible. Managing VPN access across our multi-cloud setup was clunky and constantly prone to configuration issues.
  2. VPC Peering Concerns: AWS’s VPC peering is straightforward at first, but it’s region specific, and as soon as you peer them together, your security is only as good as the weakest link. One compromised VPC will compromise them all.
  3. Multi-Cloud Complexity: As we started using multiple clouds, things got even messier. Peering and direct connections didn’t cover all our needs, and adding load balancers or complex mesh networking solutions only made things more fragile. The more environments we connected, the harder it was to keep everything secure and running smoothly.
  4. Kubernetes Networking Headaches: Managing networking and security within Kubernetes felt like a nightmare at times. Each cluster and pod communication required specific configurations and opened up potential vulnerabilities. Ensuring the right access for internal and external traffic was anything but simple.

 

Every tool we used was supposed to help us move faster, but the underlying infrastructure we had to build around them was slowing us down and constantly raising new security concerns.

Moving Faster with Zero Trust: How Ziti Changes the Game

Then came Ziti, NetFoundry’s zero trust platform. With Ziti, I could connect systems without worrying about exposing them. It’s what we call “making it dark”—there are no exposed IPs or open ports; every connection is identity-based and encrypted. Suddenly, it didn’t matter where something was deployed or how many environments we had, because everything was “invisible” to the outside world, and so much simpler to manage. 

Here’s how it’s improved our workflows:

  1. No More Open Ports or Exposed IPs: Ziti eliminates open ports and uses identity-based access instead. This means I don’t have to leave holes in our firewall or worry about IP whitelisting. Every connection is authenticated and encrypted, so I’m not constantly second-guessing if we’ve secured our environment properly.
  2. Policy-Driven Access Management: Managing access used to mean configuring networks, assigning VPNs, and juggling firewall rules. Now, with Ziti, I just assign policies to specific identities—like users, services, or even applications. Policies control who gets access to what, and they’re way easier to adjust as we grow or change setups. A policy can be as simple as “#devops identities can access “#devops services”. 
  3. Effortless Multi-Cloud and Multi-Cluster Connections: With Ziti, I can connect workloads across AWS, Azure, and GCP without needing VPNs, peering, or load balancers. It’s all seamless and managed through Ziti, which lets me focus on running the applications instead of managing connections. I get the same zero trust security across environments without the hassle.
  4. Secure, Scalable Kubernetes Networking: Kubernetes networking can be a headache, but Ziti makes it simple by embedding zero trust right into the clusters. Whether it’s ingress or egress traffic, it’s all secured without open ports. I can scale up clusters or add new ones without worrying about the networking nightmares that usually come with it.

Six Use Cases Where Ziti is a Game-Changer for Us

Since switching to Ziti, I’ve seen the benefits across everything we do in DevOps:

  • CI/CD: I can secure our CI/CD pipelines so that only authorized entities can access each stage of deployment, preventing accidental exposures in our code deployment.
  • Monitoring: Our monitoring tools are “dark” now, so all data is transmitted without opening any ports, eliminating the risk of data leaks.
  • ETL: We can centrally manage all of our ETL jobs without adding any new exposure to our database connections.
  • Data Warehouses: Ziti ensures that only authenticated users can access our data warehouses, preventing unauthorized data extraction.
  • Configuration Management: Configuration management tools can reach into distributed locations easily without ever opening up a port to the outside world.
  • Developer Access: Devs get least-privilege access via Ziti, so they can work on production issues without risking broader exposure.

Moving with Speed and Security, Thanks to Ziti

With Ziti, I’m finally able to create the connectivity I need at speed without sacrificing a strong security model. We don’t have to slow down for complex network setups or worry about exposed services or ports. Ziti’s zero trust approach is secure by default, so I can focus on building and deploying rather than fighting with access control lists. 

So, if you’re in DevOps and tired of the network hassle, take a look at zero trust with Ziti. It might just change how you work.

The post Accelerating DevOps with Zero Trust: Moving Faster Without Sacrificing Security appeared first on NetFoundry.

]]>
API Gateway Security https://netfoundry.io/devops/api-gateway-security/ Thu, 20 Jun 2024 16:00:56 +0000 https://netfoundry.io/?p=28931 In today’s interconnected world, API Gateway Security fuels the lifeblood of business interactions. Yet while fostering connectivity, APIs’ inherent openness creates a double-edged sword. Moreover, APIs become prime targets for cyberattacks, jeopardizing sensitive data and disrupting critical processes. This article explores navigating this zero-trust API security paradox, leveraging API best practices and innovative solutions like […]

The post API Gateway Security appeared first on NetFoundry.

]]>
In today’s interconnected world, API Gateway Security fuels the lifeblood of business interactions. Yet while fostering connectivity, APIs’ inherent openness creates a double-edged sword. Moreover, APIs become prime targets for cyberattacks, jeopardizing sensitive data and disrupting critical processes. This article explores navigating this zero-trust API security paradox, leveraging API best practices and innovative solutions like private API gateways.

Understanding the API Security Landscape

Galeal Zino’s insightful piece, “Squaring the Circle: How to Make Public APIs Private“, published February 14, 2024, dives deep into the unique challenges APIs face. Additionally, their public accessibility, a boon for business agility, ironically becomes an attack vector for malicious actors. Zino likens APIs to “snowflakes,” constantly evolving and exposing new vulnerabilities. However, traditional security measures, often effective against known threats, struggle to adapt to zero-day attacks that exploit these changes. This necessitates a paradigm shift towards API gateway security and private API solutions

The Rise of Zero-Trust API Networking

Zino proposes a revolutionary approach: Zero-Trust API Networking. This rethinks API accessibility, rendering them invisible to unauthorized entities while maintaining agility. Consequently, this approach eliminates public access, enforces strict identity and authentication measures, and utilizes outbound-only connections. Consider public APIs enjoying the seclusion of private networks without sacrificing the connectivity that fuels modern business. The key lies in private API gateways powered by zero-trust principles.

API Gateway Security in the Software Industry

APIs are the backbone of many software applications, enabling seamless integration and functionality across various platforms. Ensuring API gateway security is vital to protecting intellectual property and user data and maintaining the integrity of software solutions. 

Secure Development Practices
Software companies adopt secure coding practices and perform regular security assessments to identify and mitigate vulnerabilities in their APIs, especially in DevOps. This includes following OWASP API Security guidelines to address the top security risks associated with APIs.

Zero-Trust Networking
Implementing zero-trust networking principles allows software providers to create private API gateways that restrict access to authenticated and authorized users only. This reduces the attack surface and protects APIs from external threats.

Comprehensive API Management
API management solutions provide software companies with the tools to monitor, analyze, and secure their APIs in production or in development. These platforms enable rate limiting, access control, and detailed logging, ensuring that API usage is secure and compliant with industry standards.

Example
Intrusion, a NetFoundry customer,  has a partnership with NetFoundry and has built their Zero Trust Network Access Solution using NetFoundry’s platform. Intrusion Shield Endpoint helps businesses protect vital data – using zero trust architecture.

“The newly released Federal Cybersecurity Strategy reinforces the Zero Trust Approach and makes it clear that the technology industry must begin to take more responsibility for protecting the public from cybercriminals, and other threat actors,” said Tony Scott, CEO of Intrusion. “The continued vitality of the United States economy and its valuable businesses, institutions, and public services will increasingly depend on a much more secure digital infrastructure. The tech industry must take the lead and help shift the burden of poor cybersecurity from the individual to the builders and operators of our technology infrastructure who have the right skills, long-term economic incentives, and the scale to do so. Intrusion is proud to partner with NetFoundry to help accelerate this transformation.”

Another SaaS company uses NetFoundry to embed zero-trust security into its API gateway, protecting its cloud-based applications from unauthorized access. By employing microsegmentation and least-privilege access, the company ensures that only specific applications and users can interact with its APIs, safeguarding its intellectual property and user data.

API Gateway Security in the Banking Industry

The banking industry, with its vast amounts of sensitive financial data, is a prime target for cyberattacks. APIs play a crucial role in modern banking by enabling secure data exchanges between banks, third-party providers, and customers. Ensuring robust API gateway security is essential to protect this data and maintain trust.

Enhanced Security Measures
Banks implement strict API security protocols to protect customer data and comply with regulatory standards such as PSD2 in Europe, which mandates strong customer authentication (SCA) and secure open banking APIs. By leveraging zero-trust principles, banks ensure that every API request is authenticated and authorized, minimizing the risk of data breaches.

Private API Gateways
Banks can shield their APIs from the public internet, making them invisible to unauthorized users. This approach enhances security by enforcing strict access controls and using end-to-end encryption to protect data in transit.

Real-Time Monitoring and Anomaly Detection: Banks employ advanced monitoring tools to detect and respond to suspicious activities in real time. This proactive approach helps prevent unauthorized access and mitigates potential threats before they can cause significant damage.

Conclusion

API Gateway Security is paramount in today’s digital landscape, especially in high-stakes industries like banking and software. Organizations can protect their APIs from evolving cyber threats by adopting zero-trust principles and leveraging advanced security solutions like private API gateways. As Galeal Zino emphasizes, the key to securing APIs lies in rendering them invisible to unauthorized entities while maintaining their operational agility. Embracing these strategies will ensure that APIs remain a secure and reliable backbone of modern business interactions.For a deeper dive into how Zero-Trust API Networking can transform the security landscape for APIs without hindering their operational essence, Zino’s article on the open-source OpenZiti platform offers a beacon of hope. Furthermore, if you take API security seriously, you know the importance of using the right API security tools and following the OWASP API security guidelines. In conclusion, it’s a must-read for anyone vested in securing the digital arteries that power today’s interconnected business ecosystems. For complete insights and technical breakdowns, visit DevOps.com to read Galeal Zino’s enlightening article.

The post API Gateway Security appeared first on NetFoundry.

]]>
This is the Way: Invisible Jenkins https://netfoundry.io/devops/this-is-the-way-invisible-jenkins/ Mon, 04 Dec 2023 16:56:00 +0000 https://netfoundry.io/?p=42552 Why Jenkins? Jenkins is a widely adopted automation tool for building software with a broad array of plugins and integrations with key DevOps tools like Docker and Kubernetes. Here at NetFoundry, we started out using Jenkins to automate our builds, run tests, and promote releases. Eventually, we expanded to use CI as a service too. There were […]

The post This is the Way: Invisible Jenkins appeared first on NetFoundry.

]]>

Why Jenkins?

Jenkins is a widely adopted automation tool for building software with a broad array of plugins and integrations with key DevOps tools like Docker and Kubernetes. Here at NetFoundry, we started out using Jenkins to automate our builds, run tests, and promote releases. Eventually, we expanded to use CI as a service too. There were some builds that needed to happen in GitHub or BitBucket. Some jobs were not as readily migrated to a new CI system, and why bother? It made more sense to keep running those jobs in Jenkins, and so a new problem was introduced: how do we trigger Jenkins jobs from an external system like GitHub?

In our case, the Jenkins jobs needed certain parameters not available in GitHub’s built-in webhooks. We created a custom Jenkins-style webhook generator that we could use in GitHub, but the core problem remained: anyone could still “see” our Jenkins server on the internet. This exposure was intentional, but it could have become a serious problem if our server had been the target of a network-launched attack. One layer of security didn’t seem like enough protection for something so valuable. We knew that we wanted to use our home-grown networking framework OpenZiti to solve this kind of problem, so we used the OpenZiti SDK for Node.JS to create a GitHub Action to send a webhook securely via OpenZiti instead of over the internet.

NetFoundry | Webhook Visible Jenkins

Checkmate: Jenkins is a High-Value Target

I would be surprised if a resourceful attacker failed to deliver a damaging blow after finding a foothold on a Jenkins server. If this were chess, then Jenkins is the queen. One of the reasons the target value is so high is that Jenkins tends to store powerful credentials needed to deploy cloud infrastructure and applications automatically. A compromised Jenkins server is also an opportunity to introduce malicious code into a build. If successful, that infected build could be distributed through the trusted supply chain.

The tools for attacking Jenkins have become more sophisticated and accessible, making Jenkins a softer target. Popularity is a double-edged sword. The rule of thumb for defensive security is to balance the scales with target value on one side and defensive measures on the other. As our Head of Security Mike Gorman likes to say, “You don’t need a $100 lock for a $10 bike.” We knew that losing control of Jenkins would be a severe incident, so we got to work.

A Quick Inventory of the Problems and Alternatives

We knew we’d need to solve several problems before taking the Jenkins server off the public internet. 

How will users access the UI?

How will administrators access SSH?

How will we receive webhooks from external CI systems?

In times past, I’ve used tools like SSH tunnels, VPNs, virtual desktops, and webhook relays to solve these kinds of problems. It’s simpler to make the server intentionally public and trust that the login method will prevent unauthorized use, but that’s just one layer of security. I’m guessing this partially explains why Mikail Tunç found 25K Jenkins servers directly exposed to the internet during his 2018 survey of the internet via Shodan.

Those traditional solutions like VPNs and relays incur hidden, ongoing costs associated with the complexity and rigidity of a configuration that is not tightly coupled to the application’s needs. This slows down work that relies on those solutions because things often break when changes are made. The server is separate from the security, so they always have to be configured separately, which is inherently fragile. Despite all the care taken to make a server private from the internet, the core problem remains. While the application server is still listening on any network port, it is vulnerable to attack.

I’ve personally spent my fair share of time troubleshooting and managing bolt-on security measures like those VPNs. I came to regard them as a necessary evil. Whenever a problem arose with an IPSec tunnel, it was always my top priority because of the high costs associated with a loss of availability. It would have been so much better if those solutions didn’t catch on fire in the first place, and ideal if they were not even necessary.

In the next section, I’ll explain how these problems are solved by combining a strong identity with a service policy. This allows us to keep servers invisible on the network unless a caller has permission to connect.

Why use an Overlay? Identity vs Network Perimeter

OpenZiti is an example of a networking overlay that uses identities instead of a network perimeter as a basis for access control. So what is an identity-based control vs network perimeter control? Metaphorically, basing security on a private network perimeter is like saying “You’re allowed to knock on my front door if you can look up my address in a public directory and walk into my yard.” whereas requiring an identity for an invisible service is like saying “You’re allowed to knock on my front door if I sent you the secret map to my neighborhood and you have your own unique PIN for the front gate.” The identity that permits you to knock on the door is separate from the login credential, like the key that lets you open the front door.

A familiar example of network perimeter control would be limiting access to the Jenkins UI login prompt to private user network addresses on a VPN or the VPC’s private subnets attached to an SSH bastion. An example of an identity control is that your device was issued a certificate that allows you to access the Jenkins server’s login prompt from anywhere. In other words, it’s who you are (your verifiable identity), not where you are (your address with respect to the perimeter).

How Do Users Log in to Invisible Jenkins?

Everyone that needs to log in to Jenkins runs an OpenZiti tunneler on their computer. Each tunneler has an identity installed on the same computer. The OpenZiti network administrator issues these identities via email or chat in the form of one-time enrollment tokens and assigns role attributes that match up with a service policy. The service policies grant permission for those tunneler identities to use Jenkins. We control the service policies through a web console or CLI. As soon as the policy grants access to an enrolled identity, the tunneler automatically configures a private domain name and IP route to the Jenkins server on that user’s computer.

These new controls completely replaced the global DNS and public IP address of the Jenkins server. The Jenkins server stopped listening on an attached network interface address (e.g., 0.0.0.0:443) and is instead only exposed to other processes running on the same device through the loopback interface (e.g., 127.0.0.1:443). The OpenZiti “tunneler” running on the Jenkins server acts like a gatekeeper allowing connections only from OpenZiti identities that are authorized by an OpenZiti service policy.

In an OpenZiti network, your servers and clients are secure network peers and you control the direction of access and which applications are allowed.

How Does Invisible Jenkins Receive Webhooks From GitHub?

Here are two examples of Jenkins jobs in our environment. Both are triggered by a GitHub webhook that is sent via OpenZiti instead of the open internet. I used the OpenZiti Webhook “Action” from GitHub Actions Marketplace in both of these cases. This is a Node.JS application that uses the OpenZiti SDK to contact the Jenkins API in the same way the OpenZiti tunnelers do that we installed on our workstations to reach the Jenkins UI.

NetFoundry | Invisible Jenkins

The Simple Jenkins Job: Just Poll GitHub For Changes

The simplest example of the OpenZiti webhook is our Jenkins job that needs to poll for changes in a GitHub repo whenever commits are pushed to GitHub. This event triggers a workflow that sends the webhook to Jenkins. This webhook triggers the Jenkins job to poll for changes in the GitHub repo. This is a more secure alternative to sending a GitHub webhook to Jenkins over the internet.

Jenkins can poll a Git remote according to a schedule, but waiting for a scheduled job to run also means waiting to learn whether the build will succeed. To shift that build pain as early as possible I needed the Jenkins job to run as soon as the GitHub event occurred. We were using a custom Jenkins-style webhook generator in GitHub, and so I swapped in the OpenZiti webhook Action in its place. The webhook payload/context sent by the OpenZiti webhook Action is the same as GitHub’s built-in webhook generator. Still, the OpenZiti webhook Action runs as a step in a GitHub Actions workflow instead of through the GitHub Actions settings. Here’s an example workflow step that sends a webhook via OpenZiti. I’ll explain the three inputs right after the example.

NetFoundry | Webhook

This is a GitHub Actions workflow that runs on Git push events. The workflow will send the full GitHub event context to the specified URL via OpenZiti instead of the internet. The setup to enable this is as follows:

  1. Create an identity each for:
    • Jenkins server
    • Webhook workflow
  2. Create a service that defines the URL to receive the webhook.
  3. Install an OpenZiti tunneler on the Jenkins host with the first identity you created for Jenkins.
  4. Create Actions secrets for the repo with the three required values:
    • ziti-id: the JSON of the second enrolled identity you created for this workflow.
    • webhook-url: The URL of Jenkins including the service’s OpenZiti domain name, e.g. “someapp.ziti”
    • webhook-secret: A random string that’s only saved in the Actions secret and is used by the webhook sender to compute a data integrity hash for the payload data.

The Slightly More Complex Jenkins Job: Adding in Build Parameters

This job required a little extra setup for Jenkins and GitHub. The result is that I can send custom Jenkins job build parameters inside the same GitHub-style webhook via OpenZiti.

  1. A Git push event triggers the GitHub workflow.
  2. The workflow has a step that uses the OpenZiti webhook.
  3. The OpenZiti webhook accepts “inputs”. We send the Git branch name, the committer, and the computed version as extra data fields to the webhook Action as workflow inputs.
  4. The webhook uses OpenZiti to send the combined GitHub context and extra inputs to Jenkins.
  5. The webhook arrives via the OpenZiti tunneler to the Jenkins server on 127.0.0.1:443 and is parsed by the “generic webhook trigger” plugin for Jenkins.
  6. The JSON fields are parsed as Jenkins job parameters with JSONPath by the Jenkins plugin.

The “generic webhook trigger” adds a new REST endpoint in the Jenkins server that functions similarly to the Jenkin server’s built-in webhook API. I didn’t modify this Jenkins plugin, but if I had imported the OpenZiti SDK to the plugin, it would have simplified my solution even further by eliminating the need for a running OpenZiti tunneler on the Jenkins server. Here’s a sample workflow copied below that has two triggers: workflow_call and workflow_dispatch. The ziti-version input to these triggers is copied to the data input of the webhook Action along with the Git branch name from the GitHub context.

NetFoundry | Jenkins Smoketest

Taking it to the Next Level

I described a solution for the existing Jenkins users to continue using Jenkins after we switched off the internet visibility of the server. I also explained the configuration in GitHub and Jenkins for two jobs that are both triggered by a GitHub event.

What I’ve done here for Jenkins is highly similar to what we did for our SSH bastions. As you can see, this is a familiar pattern here at NetFoundry! No more mystery connections, no more presumed trust for any address or zone, including private addresses. This problem and the solution are equally applicable to GitHub or GitLab. There is already an OpenZiti webhook plugin available for both!

The OpenZiti maintainers are reachable at openziti.org. You can sign up for a free-forever NetFoundry Teams plan to get up and running instantly with a hosted, managed, opinionated deployment of OpenZiti: nfconsole.io/signup.

Please give our GitHub repo a GOLD STAR to let us know you’re glad OpenZiti is a thing, and let us know what you’re building in Discourse so we can brag on it and help out!

The post This is the Way: Invisible Jenkins appeared first on NetFoundry.

]]>