{"id":42552,"date":"2023-12-04T11:56:00","date_gmt":"2023-12-04T16:56:00","guid":{"rendered":"https:\/\/netfoundry.io\/?p=42552"},"modified":"2024-12-05T10:32:43","modified_gmt":"2024-12-05T15:32:43","slug":"this-is-the-way-invisible-jenkins","status":"publish","type":"post","link":"https:\/\/netfoundry.io\/devops\/this-is-the-way-invisible-jenkins\/","title":{"rendered":"This is the Way: Invisible Jenkins"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Why Jenkins?<\/h2>\n

Jenkins<\/strong> is a widely adopted automation tool for building software with a broad array of plugins and\u00a0integrations\u00a0with key DevOps tools like Docker and Kubernetes. Here at NetFoundry, we started out using Jenkins to automate our builds, run tests, and promote releases. Eventually, we expanded to use CI as a service too. There were some builds that needed to happen in GitHub or BitBucket. Some jobs were not as readily migrated to a new CI system, and why bother? It made more sense to keep running those jobs in Jenkins, and so a new problem was introduced: how do we trigger Jenkins jobs from an external system like GitHub?<\/p>\n

In our case, the Jenkins jobs needed certain parameters not available in GitHub\u2019s built-in webhooks. We created a custom Jenkins-style webhook generator that we could use in GitHub, but the core problem remained: anyone could still \u201csee\u201d our Jenkins server on the internet. This exposure was intentional, but it could have become a serious problem if our server had been the target of a network-launched attack. One layer of security didn\u2019t seem like enough protection for something so valuable. We knew that we wanted to use our home-grown networking framework OpenZiti to solve this kind of problem, so we used the OpenZiti SDK for Node.JS to create a GitHub Action to send a webhook securely via OpenZiti instead of over the internet.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"NetFoundry\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Checkmate: Jenkins is a High-Value Target<\/h3>\n

I would be surprised if a resourceful attacker failed to deliver a damaging blow after finding a foothold on a Jenkins server. If this were chess, then Jenkins is the queen. One of the reasons the target value is so high is that Jenkins tends to store powerful credentials needed to deploy cloud infrastructure and applications automatically. A compromised Jenkins server is also an opportunity to introduce malicious code into a build. If successful, that infected build could be distributed through the trusted supply chain.<\/p>\n

The tools for attacking Jenkins have become\u00a0more sophisticated and accessible<\/a>,\u00a0making Jenkins a softer target. Popularity is a double-edged sword. The rule of thumb for defensive security is to balance the scales with target value on one side and defensive measures on the other. As our Head of Security Mike Gorman likes to say, \u201cYou don\u2019t need a $100 lock for a $10 bike.\u201d We knew that losing control of Jenkins would be a severe incident, so we got to work.<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

A Quick Inventory of the Problems and Alternatives<\/h3>\n

We knew we\u2019d need to solve several problems before taking the Jenkins server off the public internet.\u00a0<\/p>\n

How will users access the UI?<\/strong><\/em><\/p>\n

How will administrators access SSH?<\/strong><\/em><\/p>\n

How will we receive webhooks from external CI systems?<\/strong><\/em><\/p>\n

In times past, I\u2019ve used tools like SSH tunnels, VPNs, virtual desktops, and webhook relays to solve these kinds of problems. It\u2019s simpler to make the server intentionally public and trust that the login method will prevent unauthorized use, but that\u2019s just one layer of security. I\u2019m guessing this partially explains why Mikail Tun\u00e7 found 25K Jenkins servers directly exposed to the internet during\u00a0his 2018 survey of the internet via Shodan<\/a>.<\/p>\n

Those traditional solutions like VPNs and relays incur hidden, ongoing costs associated with the complexity and rigidity of a configuration that is not tightly coupled to the application\u2019s needs. This slows down work that relies on those solutions because things often break when changes are made. The server is separate from the security, so they always have to be configured separately, which is inherently fragile. Despite all the care taken to make a server private from the internet, the core problem remains. While the application server is still listening on any network port, it is vulnerable to attack.<\/p>\n

I\u2019ve personally spent my fair share of time troubleshooting and managing bolt-on security measures like those VPNs. I came to regard them as a necessary evil. Whenever a problem arose with an IPSec tunnel, it was always my top priority because of the high costs associated with a loss of availability. It would have been so much better if those solutions didn\u2019t catch on fire in the first place, and ideal if they were not even necessary.<\/p>\n

In the next section, I\u2019ll explain how these problems are solved by combining a strong identity with a service policy. This allows us to keep servers invisible on the network unless a caller has permission to connect.<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Why use an Overlay? Identity vs Network Perimeter<\/h3>\n

OpenZiti is an example of a networking overlay that uses identities instead of a network perimeter as a basis for access control. So what is an identity-based control vs network perimeter control? Metaphorically, basing security on a private network perimeter is like saying \u201cYou\u2019re allowed to knock on my front door if you can look up my address in a public directory and walk into my yard.\u201d whereas requiring an identity for an invisible service is like saying \u201cYou\u2019re allowed to knock on my front door if I sent you the secret map to my neighborhood and you have your own unique PIN for the front gate.\u201d The identity that permits you to knock on the door is separate from the login credential, like the key that lets you open the front door.<\/p>\n

A familiar example of network perimeter control would be limiting access to the Jenkins UI login prompt to private user network addresses on a VPN or the VPC\u2019s private\u00a0subnets attached to an SSH bastion. An example of an identity control is that your device was issued a certificate that allows you to access the Jenkins server\u2019s login prompt from anywhere. In other words, it\u2019s who you are (your verifiable identity), not where you are (your address with respect to the perimeter).<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

How Do Users Log in to Invisible Jenkins?<\/h3>\n

Everyone that needs to log in to Jenkins runs an OpenZiti tunneler on their computer. Each tunneler has an identity installed on the same computer. The OpenZiti network administrator issues these identities via email or chat in the form of one-time enrollment tokens and assigns role attributes that match up with a service policy. The service policies grant permission for those tunneler identities to use Jenkins. We control the service policies through a web console or CLI. As soon as the policy grants access to an enrolled identity, the tunneler automatically configures a private domain name and IP route to the Jenkins server on that user\u2019s computer.<\/p>\n

These new controls completely replaced the global DNS and public IP address of the Jenkins server. The Jenkins server stopped listening on an attached network interface address (e.g., 0.0.0.0:443) and is instead only exposed to other processes running on the same device through the loopback interface (e.g., 127.0.0.1:443). The OpenZiti \u201ctunneler\u201d running on the Jenkins server acts like a gatekeeper allowing connections only from OpenZiti identities that are authorized by an OpenZiti service policy.<\/p>\n

In an OpenZiti network, your servers and clients are secure network peers and you control the direction of access and which applications are allowed.<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t

\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

How Does Invisible Jenkins Receive Webhooks From GitHub?<\/h3>\n

Here are two examples of Jenkins jobs in our environment. Both are triggered by a GitHub webhook that is sent via OpenZiti instead of the open internet. I used\u00a0the OpenZiti Webhook \u201cAction\u201d from GitHub Actions Marketplace<\/a>\u00a0in both of these cases. This is a Node.JS application that uses the OpenZiti SDK to contact the Jenkins API in the same way the OpenZiti tunnelers do that we installed on our workstations to reach the Jenkins UI.<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"NetFoundry\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

The Simple Jenkins Job: Just Poll GitHub For Changes<\/h3>\n

The simplest example of the OpenZiti webhook is our Jenkins job that needs to poll for changes in a GitHub repo whenever commits are pushed to GitHub. This event triggers a workflow that sends the webhook to Jenkins. This webhook triggers the Jenkins job to poll for changes in the GitHub repo. This is a more secure alternative to sending a GitHub webhook to Jenkins over the internet.<\/p>\n

Jenkins can poll a Git remote according to a schedule, but waiting for a scheduled job to run also means waiting to learn whether the build will succeed. To shift that build pain as early as possible I needed the Jenkins job to run as soon as the GitHub event occurred. We were using a custom Jenkins-style webhook generator in GitHub, and so I swapped in the OpenZiti webhook Action in its place. The webhook payload\/context sent by the OpenZiti webhook Action is the same as GitHub\u2019s built-in webhook generator. Still, the OpenZiti webhook Action runs as a step in a GitHub Actions workflow instead of through the GitHub Actions settings. Here\u2019s an example workflow step that sends a webhook via OpenZiti. I\u2019ll explain the three inputs right after the example.<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"NetFoundry\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This is a GitHub Actions workflow that runs on Git push events. The workflow will send the full GitHub event context to the specified URL via OpenZiti instead of the internet. The setup to enable this is as follows:<\/p>\n

    \n
  1. Create an identity each for:\n