4.1   Replace Cookie and Authorization headers¶

While the auth_request handler cannot remove headers, NGINX and the Kubernetes Ingress annotations do provide a way of replacing headers in the request with headers returned by the auth_request handler before passing them to the backend. We could make use of this by having Gafaelfawr return a Cookie header that is the same as the incoming request but with the Gafaelfawr cookie stripped out of it, and then configure the ingress to replace the incoming header with that header via:

nginx.ingress.kubernetes.io/auth-response-headers: Cookie

(plus any additional headers from Gafaelfawr that the protected application uses). A simpler approach would work for the Authorization header: listing the header in auth-response-headers without sending that header in the auth_request handler response should strip the header from the request before sending it to the backend service. This would only work for services that do not reuse the Authorization header for their own purposes (JupyterHub does this).

For backend services that must receive a token in the Authorization header (CADC’s TAP service is currently in this category), Gafaelfawr could be configured to return a delegated token in the Authorization header.

Advantages:

• Works transparently with the current Rubin Science Platform design, with no changes required to protected services, routes, or hostnames.
• Addresses both the cookie and Authorization header cases.
• Conceptually simple, and collects all of our request manipulation code in the same place (the Gafaelfawr auth_request handler).

Disadvantages:

• Requires parsing and surgery on the Cookie header in Gafaelfawr, which opens the possibility of unexpected browser interactions or problems with invalid but still working headers.
• Only protects against token leakage to authenticated URLs that are protected by Gafaelfawr. Requests that do not go through an auth_request handler will still leak cookies and Authorization headers. This includes any request to a backend service that uses Gafaelfawr’s OpenID Connect support instead. This could be mitigated by supporting a stripped-down auth_request handler mode that only cleans the headers and sending all requests, even unauthenticated requests, through an auth_request handler, at the cost of additional complexity and possible fragility.
• This adds additional complexity to the required ingress configuration to use Gafaelfawr, which is already very complex. If we take this approach, it may be worth adding Gafaelfawr support for a custom ingress resource and have Gafaelfawr add the appropriate annotations and generate the real Ingress resource from that custom resource.
• Does not isolate the JavaScript of each service. All services are still in the same JavaScript origin, which means that malicious JavaScript injected into any service could still make authenticated requests to other services, even though the attacker would not have direct access to the cookie.

4.2   Strip Gafaelfawr cookie from proxied request¶

It may be possible to add NGINX configuration to remove the cookie from the proxied request. It would still be present in the auth_request subrequest, but would not be sent to the destination host. See, for example, these instructions to remove a specific cookie with NGINX, which use the following snippet:

set $new_cookie $http_cookie;
if ($http_cookie ~ "(.*)(?:^|;)\s*some_cookie=[^;]+(.*)") {
  set $new_cookie $1$2;
}
proxy_set_header Cookie $new_cookie;

A simpler approach also works for the Authorization header:

proxy_set_header Authorization "";

However, the Notebook Aspect also uses the Authorization header for its own internal purposes, so the logic may need to be more complex, or the Notebook Aspect may need to be excluded.

Advantages:

• Works transparently with the current Rubin Science Platform design, with no changes required to protected services, routes, or hostnames.
• Addresses both the cookie and Authorization header cases.

Disadvantages:

• Does not isolate the JavaScript of each service. All services are still in the same JavaScript origin, which means that malicious JavaScript injected into any service could still make authenticated requests to other services, even though the attacker would not have direct access to the cookie.
• It’s not clear how this stacks with auth_request subrequests, which are also done with proxying. The cookie and Authorization header must be included in that request. Some experimentation to find the right configuration may be required.
• Editing a structured field with regular expression matching is error-prone and potentially fragile. There is at least one report that NGINX then escaped the header and broke other cookies, and other problems like that are possible.
• This relatively complex NGINX configuration would need to be added to every ingress definition used in the Rubin Science Platform and kept up-to-date if it needed to change.
• Given the complex interactions between proxying, auth_request, and regex matching, this is the type of configuration that runs a risk of breaking with NGINX upgrades.

4.3   Use separate per-host cookies for each application¶

If each protected service had its own authentication session cookie that was only usable by that service, and only that cookie was sent to requests for that service, that would eliminate the problem.

This could be done as follows:

• Create a separate hostname for each service. In other words, for the Rubin Science Platform instance hosted at data.lsst.cloud, the Notebook Aspect would be at nb.data.lsst.cloud (and username.nb.data.lsst.cloud once a notebook has been launched), the Portal Aspect would be at portal.data.lsst.cloud, and so forth. The authentication system itself would use auth.data.lsst.cloud. (Per-service granularity is ideal from a security standpoint, but this approach works with any granularity of hostnames. We could instead group services into a small number of security domains and accept attacker movement within a security domain. The most important to separate are, first, the authentication system and the notebooks, and then, second, the Portal Aspect. API services could probably be grouped into one hostname without much loss of security provided that the Authorization header is stripped.)
• The authentication session cookie for each of those services would be scoped to only that hostname and would use the __Host- prefix. See the Set-Cookie documentation for more information about that prefix.
• The cookie, encrypted in a key known only to Gafaelfawr, would contain the hostname for which the cookie was valid. Gafaelfawr would not honor the cookie as authentication to any other domain. This would prevent authenticating to one service with a cookie stole from a request to a different service.

This means there would no longer be a single authentication cookie for the entire Rubin Science Platform instance. That, in turn, means that authentication to a given protected service through a web browser would become somewhat more complicated. Rather than simply redirecting to /login and then back to the protected service once the cookie has been set, each protected service would have to follow a login protocol similar to OAuth 2. This could still be implemented in the auth_request handler.

The similar but more difficult problem of authenticating web services at arbitrary hostnames using OAuth 2 is handled as follows:

1. Service sets a cookie containing a random state string. (The state string is required to prevent session fixation.)
2. Unauthenticated user is redirected to the identity provider, including the state string in the request.
3. The identity provider authenticates the user.
4. The identity provider redirects the user back to the protected service, including the state string and an authentication code in the request.
5. The protected service compares the state strings and ensures they match.
6. The protected service presents the code to the identity provider, which returns authentication information about the user.
7. The protected service creates a session cookie containing that now-verified authentication information.

In this case, since the same software component can act as both the protected service and the identity provider, step 6 can be simplified by using shared state. The login protocol would instead look like this:

1. Service creates an encrypted cookie for its hostname containing a random state string.
2. Service redirects the user to the /login route on the separate auth hostname for this Rubin Science Platform deployment and includes the state string and the return URL in that request.
3. The /login route authenticates the user. This may redirect to another provider, or may be immediate if the user has already authenticated to some other service. Store the user’s authentication credentials in a cookie specific to the auth hostname to fulfill subsequent authentication requests. Create a random Redis key. In Redis, under that key, store the domain authenticated, the state string, and the user’s credentials.
4. Redirect the user back to a designated reserved URL on the same hostname as the return URL. Include the new Redis key (which acts as an authentication code) in that request. Unfortunately, the authentication system has to pass state back to the hostname of the protected service, so this intermediate URL is needed.
5. Using the auth_request handler, intercept that request. Retrieve the information from the Redis key. Verify that the state and hostname match. Delete the Redis key. Set a cookie containing the hostname and authentication credentials from the Redis data, which will act as the authentication session cookie for that hostname going forward.
6. Redirect the user back to the URL they were trying to visit. The user now has a cookie for that hostname whose internal (encrypted) data matches the hostname of the request, and authentication can proceed as normal.

This is the same process as OAuth 2 but without step 6 because external storage is used to retrieve the information instead.

Advantages:

• Also provides protection against malicious JavaScript hosted by one Rubin Science Platform service by separating services into different JavaScript origins. Currently, all services are the same origin for JavaScript purposes, so malicious JavaScript hosted by any service can fool the browser into making authenticated requests to other services on behalf of the attacker. Separating the services into different origins would bring the normal JavaScript cross-origin request policy into play, which would provide substantial protection against lateral movement between services using JavaScript (via CSRF, for example).
• Separates the session cookies into separate cookies for each hostname that only work for that hostname.
• Uses well-understood cookie properties and parallels the well-tested OAuth 2 authentication flow.
• Doesn’t require any special NGINX configuration.

Disadvantages:

• Requires some significant changes to the authentication system to implement this new authentication flow.
• Adds additional complexity to each internal authentication request (akin to using OpenID Connect internally).
• Does not address the Authorization header problem, since we cannot ask users to use per-service tokens. However, it may be possible to use a combination of this approach and either auth_request handler stripping or NGINX configuration to hide the Authorization header from protected services.

4.4   Use path-restricted cookies¶

Theoretically, a variation of the previous design can be done with path-restricted cookies instead. This would allow all protected services to use the same hostname, but maintain separate cookies for each protected service. Rather than issuing the cookies to different hostnames, the cookies would use a path restriction, limiting the cookie to only the route prefix used by that application.

The rest of the design would be identical to using per-host cookies except the cookies could not use the __Host- prefix (since it forbids path-restricted cookies).

In practice, path-restricted cookies provide little security benefit because they are stricter than the same-origin policy of JavaScript and thus can be bypassed by using malicious JavaScript.

This approach would have all of the disadvantages of per-host cookies without the benefit of site isolation against malicious JavaScript. The only advantage would be to avoid needing to create and expose separate hostnames per service, which is not a sufficiently compelling advantage.