summaryrefslogtreecommitdiff
path: root/modules/caddyhttp/reverseproxy/reverseproxy.go
AgeCommit message (Collapse)Author
2022-03-09reverseproxy: copy_response and copy_response_headers for handle_response ↵Francis Lavoie
routes (#4391) * reverseproxy: New `copy_response` handler for `handle_response` routes Followup to #4298 and #4388. This adds a new `copy_response` handler which may only be used in `reverse_proxy`'s `handle_response` routes, which can be used to actually copy the proxy response downstream. Previously, if `handle_response` was used (with routes, not the status code mode), it was impossible to use the upstream's response body at all, because we would always close the body, expecting the routes to write a new body from scratch. To implement this, I had to refactor `h.reverseProxy()` to move all the code that came after the `HandleResponse` loop into a new function. This new function `h.finalizeResponse()` takes care of preparing the response by removing extra headers, dealing with trailers, then copying the headers and body downstream. Since basically what we want `copy_response` to do is invoke `h.finalizeResponse()` at a configurable point in time, we need to pass down the proxy handler, the response, and some other state via a new `req.WithContext(ctx)`. Wrapping a new context is pretty much the only way we have to jump a few layers in the HTTP middleware chain and let a handler pick up this information. Feels a bit dirty, but it works. Also fixed a bug with the `http.reverse_proxy.upstream.duration` placeholder, it always had the same duration as `http.reverse_proxy.upstream.latency`, but the former was meant to be the time taken for the roundtrip _plus_ copying/writing the response. * Delete the "Content-Length" header if we aren't copying Fixes a bug where the Content-Length will mismatch the actual bytes written if we skipped copying the response, so we get a message like this when using curl: ``` curl: (18) transfer closed with 18 bytes remaining to read ``` To replicate: ``` { admin off debug } :8881 { reverse_proxy 127.0.0.1:8882 { @200 status 200 handle_response @200 { header Foo bar } } } :8882 { header Content-Type application/json respond `{"hello": "world"}` 200 } ``` * Implement `copy_response_headers`, with include/exclude list support * Apply suggestions from code review Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2022-03-06reverseproxy: Dynamic upstreams (with SRV and A/AAAA support) (#4470)Matt Holt
* reverseproxy: Begin refactor to enable dynamic upstreams Streamed here: https://www.youtube.com/watch?v=hj7yzXb11jU * Implement SRV and A/AAA upstream sources Also get upstreams at every retry loop iteration instead of just once before the loop. See #4442. * Minor tweaks from review * Limit size of upstreams caches * Add doc notes deprecating LookupSRV * Provision dynamic upstreams Still WIP, preparing to preserve health checker functionality * Rejigger health checks Move active health check results into handler-specific Upstreams. Improve documentation regarding health checks and upstreams. * Deprecation notice * Add Caddyfile support, use `caddy.Duration` * Interface guards * Implement custom resolvers, add resolvers to http transport Caddyfile * SRV: fix Caddyfile `name` inline arg, remove proto condition * Use pointer receiver * Add debug logs Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2022-03-06reverseproxy: Implement trusted proxies for `X-Forwarded-*` headers (#4507)Francis Lavoie
2022-03-03reverseproxy: Make shallow-ish clone of the request (#4551)Francis Lavoie
* reverseproxy: Make shallow-ish clone of the request * Refactor request cloning into separate function Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2022-01-04admin, reverseproxy: Stop timers if canceled to avoid goroutine leak (#4482)Денис Телюх
2021-12-02caddyhttp: Make logging of credential headers opt-in (#4438)Francis Lavoie
2021-11-24reverseproxy: Adjust defaults, document defaults (#4436)Francis Lavoie
* reverseproxy: Adjust defaults, document defaults Related to some of the issues in https://github.com/caddyserver/caddy/issues/4245, a complaint about the proxy transport defaults not being properly documented in https://caddy.community/t/default-values-for-directives/14254/6. - Dug into the stdlib to find the actual defaults for some of the timeouts and buffer limits, documenting them in godoc so the JSON docs get them next release. - Moved the keep-alive and dial-timeout defaults from `reverseproxy.go` to `httptransport.go`. It doesn't make sense to set defaults in the proxy, because then any time the transport is configured with non-defaults, the keep-alive and dial-timeout defaults are lost! - Sped up the dial timeout from 10s to 3s, in practice it rarely makes sense to wait a whole 10s for dialing. A shorter timeout helps a lot with the load balancer retries, so using something lower helps with user experience. * reverseproxy: Make keepalive interval configurable via Caddyfile * fastcgi: DialTimeout default for fastcgi transport too
2021-10-26reverseproxy: Sanitize scheme and host on incoming requests (#4237)Francis Lavoie
* caddyhttp: Sanitize scheme and host on incoming requests * reverseproxy: Sanitize the URL scheme and host before proxying * Apply suggestions from code review Co-authored-by: Matt Holt <mholt@users.noreply.github.com> Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2021-10-18reverseproxy: Prevent copying the response if a response handler ran (#4388)Francis Lavoie
2021-09-27Revert 3336faf2 (close #4360)Matthew Holt
Debug log is correct level for this
2021-09-24reverseproxy: Log error at error level (fix #4360)Matthew Holt
2021-08-23reverseproxy: Remove redundant flushing (#4299)Francis Lavoie
From reading through the code, I think this code path is now obsoleted by the changes made in https://github.com/caddyserver/caddy/pull/4266. Basically, `h.flushInterval()` will set the flush interval to `-1` if we're in a bi-directional stream, and the recent PR ensured that `h.copyResponse()` properly flushes headers immediately when the flush interval is non-zero. So now there should be no need to call Flush before calling `h.copyResponse()`.
2021-08-12reverseproxy: Incorporate latest proxy changes from stdlib (#4266)Francis Lavoie
I went through the commits that touched stdlib's `reverseproxy.go` file, and copied over all the changes that are to code that was copied into Caddy. The commits I pulled changes from: - https://github.com/golang/go/commit/2cc347382f4df3fb40d8d81ec9331f0748b1c394 - https://github.com/golang/go/commit/a5cea062b305c8502bdc959c0eec279dbcd4391f - https://github.com/golang/go/commit/ecdbffd4ec68b509998792f120868fec319de59b - https://github.com/golang/go/commit/21898524f66c075d7cfb64a38f17684140e57675 -https://github.com/golang/go/commit/ca3c0df1f8e07337ba4048b191bf905118ebe251 - https://github.com/golang/go/commit/9c017ff30dd21bbdcdb11f39458d3944db530d7e This may also fix https://github.com/caddyserver/caddy/issues/4247 because of the change to `copyResponse` to set `mlw.flushPending = true` right away.
2021-06-15reverseproxy: Fix overwriting of max_idle_conns_per_host (closes #4201)Matthew Holt
Also split the Caddyfile subdirective keepalive_idle_conns into two properties so the conns and conns_per_host can be set separately. This is technically a breaking change, but probably anyone who this breaks already had a broken config anyway, and silently fixing it won't help them fix their configs.
2021-06-04reverseproxy: Always remove hop-by-hop headersMatthew Holt
See golang/go#46313 Based on https://github.com/golang/go/commit/950fa11c4cb01a145bb07eeb167d90a1846061b3
2021-05-12reverseproxy: Set the headers in the replacer before `handle_response` (#4165)Francis Lavoie
Turns out this was an oversight, we assumed we could use `{http.response.header.*}` but that doesn't work because those are grabbed from the response writer, and we haven't copied any headers into the response writer yet. So the fix is to set all the response headers into the replacer at a new namespace before running the handlers. This adds the `{http.reverse_proxy.header.*}` replacer. See https://caddy.community/t/empty-http-response-header-x-accel-redirect/12447
2021-05-05reverseproxy: Minor logging improvementsMatthew Holt
2021-05-02reverseproxy: Add `handle_response` blocks to `reverse_proxy` (#3710) (#4021)Francis Lavoie
* reverseproxy: Add `handle_response` blocks to `reverse_proxy` (#3710) * reverseproxy: complete handle_response test * reverseproxy: Change handle_response matchers to use named matchers reverseproxy: Add support for changing status code * fastcgi: Remove obsolete TODO We already have d.Err("transport already specified") in the reverse_proxy parsing code which covers this case * reverseproxy: Fix support for "4xx" type status codes * Apply suggestions from code review Co-authored-by: Matt Holt <mholt@users.noreply.github.com> * caddyhttp: Reorganize response matchers * reverseproxy: Reintroduce caddyfile.Unmarshaler * reverseproxy: Add comment mentioning Finalize should be called Co-authored-by: Maxime Soulé <btik-git@scoubidou.com> Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2021-03-29reverseproxy: Implement health_uri, deprecate health_path, supports query ↵Francis Lavoie
(#4050) * reverseproxy: Implement health_uri, replaces health_path, supports query Also fixes a bug with `health_status` Caddyfile parsing , it would always only take the first character of the status code even if it didn't end with "xx". * reverseproxy: Rename to URI, named logger, warn in Provision (for JSON)
2021-02-25reverseproxy: Fix round robin data race (#4038)Matthew Holt
2021-02-22reverseproxy: Add duration/latency placeholders (close #4012) (#4013)Matt Holt
* reverseproxy: Add duration/latency placeholders (close #4012) (and #2268) Adds 4 placeholders, one is actually outside reverse proxy though: {http.request.duration} is how long since the server decoded the HTTP request (headers). {http.reverse_proxy.upstream.latency} is how long it took a proxy upstream to write the response header. {http.reverse_proxy.upstream.duration} is total time proxying to the upstream, including writing response body to client. {http.reverse_proxy.duration} is total time spent proxying, including selecting an upstream and retries. Obviously, most of these are only useful at the end of a request, like when writing response headers or logs. See also: https://caddy.community/t/any-equivalent-of-request-time-and-upstream-header-time-from-nginx/11418 * Add new placeholders to documentation
2021-02-09reverseproxy: Response buffering & configurable buffer sizeMatthew Holt
Proxy response bodies can now be buffered, and the size of the request body and response body buffer can be limited. Any remaining content that doesn't fit in the buffer will remain on the wire until it can be read; i.e. bodies are not truncated, even if the buffer is not big enough. This fulfills a customer requirement. This was made possible by their sponsorship!
2020-11-25reverseproxy: Handle "operation was canceled" errors (#3816)Daniel Santos
* fix(caddy): Avoid "operation was canceled" errors - Also add error handling for StatusGatewayTimeout * revert(caddy): Revert 504 handling - This will potentially break load balancing and health checks * Handle client cancellation as different error Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2020-11-22ci: Use golangci's github action for linting (#3794)Dave Henderson
* ci: Use golangci's github action for linting Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix most of the staticcheck lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the prealloc lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the misspell lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the varcheck lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the errcheck lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the bodyclose lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the deadcode lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the unused lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the gosec lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the gosimple lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the ineffassign lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Fix the staticcheck lint errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Revert the misspell change, use a neutral English Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Remove broken golangci-lint CI job Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Re-add errantly-removed weakrand initialization Signed-off-by: Dave Henderson <dhenderson@gmail.com> * don't break the loop and return * Removing extra handling for null rootKey * unignore RegisterModule/RegisterAdapter Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com> * single-line log message Co-authored-by: Matt Holt <mholt@users.noreply.github.com> * Fix lint after a1808b0dbf209c615e438a496d257ce5e3acdce2 was merged Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Revert ticker change, ignore it instead Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Ignore some of the write errors Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Remove blank line Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Use lifetime Signed-off-by: Dave Henderson <dhenderson@gmail.com> * close immediately Co-authored-by: Matt Holt <mholt@users.noreply.github.com> * Preallocate configVals Signed-off-by: Dave Henderson <dhenderson@gmail.com> * Update modules/caddytls/distributedstek/distributedstek.go Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com> Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
2020-11-20reverseproxy: Implement cookie hash selection policy (#3809)Dimitri Masson
* add CookieHashSelection for session affinity * add CookieHashSelection for session affinity * register module * reverse_proxy: Add and fix cookie lb_policy * reverse_proxy: Manage hmac.write error on cookie hash selection * reverse_proxy: fix some comments * reverse_proxy: variable `cookieValue` is inside the else block * reverse_proxy: Abstract duplicate nuanced logic of reservoir sampling into a function * reverse_proxy: Set a default secret is indeed useless * reverse_proxy: add configuration syntax for cookie lb_policy * reverse_proxy: doc typo and improvement Co-authored-by: utick <123liuqingdong@163.com>
2020-10-13reverseproxy: Fix dial placeholders, SRV, active health checks (#3780)Matt Holt
* reverseproxy: Fix dial placeholders, SRV, active health checks Supercedes #3776 Partially reverts or updates #3756, #3693, and #3695 * reverseproxy: add integration tests Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
2020-10-01reverseproxy: allow no port for SRV; fix regression in d55d50b (#3756)Mohammed Al Sahaf
* reverseproxy: fix breakage in handling SRV lookup introduced by 3695 * reverseproxy: validate against incompatible config options with lookup_srv * reverseproxy: add integration test cases for validations involving lookup_srv * reverseproxy: clarify the reason for skipping an iteration * grammar.. Oxford comma Co-authored-by: Francis Lavoie <lavofr@gmail.com> Co-authored-by: Francis Lavoie <lavofr@gmail.com> Fixes #3753
2020-10-01reverseproxy: Change 500 error to 502 for lookup_srv config (#3771)Aleksei
Fixes #3763
2020-09-17reverseproxy: Correct alternate port for active health checks (#3693)Mohammed Al Sahaf
* reverseproxy: construct active health-check transport from scratch (Fixes #3691) * reverseproxy: do upstream health-check on the correct alternative port * reverseproxy: add integration test for health-check on alternative port * reverseproxy: put back the custom transport for health-check http client * reverseproxy: cleanup health-check integration test * reverseproxy: fix health-check of unix socket upstreams * reverseproxy: skip unix socket tests on Windows * tabs > spaces Co-authored-by: Francis Lavoie <lavofr@gmail.com> * make the linter (and @francislavoie) happy Co-authored-by: Francis Lavoie <lavofr@gmail.com> * One more lint fix Co-authored-by: Francis Lavoie <lavofr@gmail.com> Co-authored-by: Francis Lavoie <lavofr@gmail.com>
2020-09-16reverseproxy: Enforce port range size of 1 at provision (#3695)Mohammed Al Sahaf
* reverse_proxy: ensure upstream address has port range of only 1 * reverse_proxy: don't log the error if upstream range size is more than 1
2020-08-07reverseproxy: Minor fixes and cleanupMatthew Holt
Now use context cancellation to stop active health checker, which is simpler than and just as effective as using a separate stop channel.
2020-08-03reverse_proxy: fix bidirectional streams with encodings (fix #3606) (#3620)Kevin Lin
* reverse_proxy: fix bi-h2stream breaking gzip encode handle(#3606). * reverse_proxy: check http version of both sides to avoid affecting non-h2 upstream. * Minor cleanup; apply review suggestions Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
2020-07-20reverse_proxy: flush HTTP/2 response when ContentLength is unknown (#3561)Kevin Lin
* reverse proxy: Support more h2 stream scenarios (#3556) * reverse proxy: add integration test for better h2 stream (#3556) * reverse proxy: adjust comments as francislavoie suggests * link to issue #3556 in the comments
2020-07-17reverseproxy: Restore request's original host and header (fix #3509)Matthew Holt
We already restore them within the retry loop, but after successful proxy we didn't reset them, so as handlers bubble back up, they would see the values used for proxying. Thanks to @ziddey for identifying the cause.
2020-06-04reverseproxy: Enable changing only the status code (close #2920)Matthew Holt
2020-05-27reverseproxy: Pool copy buffers (minor optimization)Matthew Holt
2020-05-27reverseproxy: Enable response interception (#1447, #2920)Matthew Holt
It's a raw, low-level implementation for now, but it's very flexible. More sugar-coating can be added after error handling is more developed.
2020-05-20reverseproxy: Don't overwrite existing X-Forwarded-Proto headerMatthew Holt
Correct behavior is not well defined because this is a non-standard header field. This could be a "hop-by-hop" field much like X-Forwarded-For is, but even our X-Forwarded-For implementation preserves prior entries. Or, it could be best to preserve the original value from the first hop, representing the protocol as facing the client. Let's try it the other way for a bit and see how it goes. See https://caddy.community/t/caddy2-w-wordpress-behind-nginx-reverse-proxy/8174/3?u=matt
2020-05-18reverseproxy: Make debug log safe if error occursMatthew Holt
2020-05-18reverseproxy: Emit debug log before checking error (#3425)Matthew Holt
This way the upstream request will always be available even if it failed
2020-05-13reverseproxy: Apply response header ops before copying it (fix #3382) (#3401)Matt Holt
2020-04-22reverseproxy: Don't forget to provision embedded headers handlerMatthew Holt
https://caddy.community/t/set-cookie-manipulation-in-reverse-proxy/7666?u=matt
2020-04-21reverseproxy: always set req.URL.Host with upstream (#3297)westwin
2020-04-17reverseproxy: Set X-Forwarded-Proto (closes #3275) (#3276)Matt Holt
2020-04-07reverseproxy: Remove NTLM transport; refactor and improve docsMatthew Holt
2020-03-30Keep type information with placeholders until replacements happenMatthew Holt
2020-03-27reverse_proxy: Upstream.String() method returns either LookupSRV or DialMatthew Holt
Either Dial or LookupSRV will be set, but if we rely on Dial always being set, we could run into bugs. Note: Health checks don't support SRV upstreams.
2020-03-24reverse_proxy: Add support for SRV backends (#3180)Matt Holt
* reverse_proxy: Begin SRV lookup support (WIP) * reverse_proxy: Finish adding support for SRV-based backends (#3179)
2020-03-20reverseproxy: Add Alt-Svc to Hop-by-hop headers list (#3159)Paolo Barbolini
Adds `Alt-Svc` to the list of headers that get removed when proxying to a backend. This fixes the issue of having the contents of the Alt-Svc header duplicated when proxying to another Caddy server.
2020-02-23reverse_proxy: Health checks: Don't cross the streamsMatthew Holt
Fixes https://caddy.community/t/v2-health-checks-are-going-to-the-wrong-upstream/7084?u=matt ... I think
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-09 17:27:13 +0000