Log Analysis with netgo
2022 March 13 16:23 stuartscott 177537¤ 1255¤
So you wrote a web server in Go and, like dishes at a fancy restaurant, pages are getting served.
package main
import (
"log"
"net/http"
)
func main() {
// Create Multiplexer
mux := http.NewServeMux()
// Handle Index
mux.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain")
w.Write([]byte("Hello World!"))
}))
// Serve HTTP Requests
log.Println("HTTP Server Listening on :80")
if err := http.ListenAndServe(":80", mux); err != nil {
log.Fatal(err)
}
}
By now you probably have some questions, like;
- What is the most popular page?
- When do most visitors come?
- How long do visitors stay?
- Which features should be added next?
You could use one of the many website analytic services but then the identity and behavior of your visitors is being given away to a third party.
This article will show you how to use netgo
to gain insights into your traffic while respecting your users and keeping your log data confidential.
As always, the code shown is open source and hosted on GitHub.
Step 1: Collect
The most important piece of any data analysis project is of course the data itself, and so your first step is to collect it.
netgo
provides two utilities to help with this endeavor; the first configures the log package in the standard library to write to both standard output and a file so you have a permanent record, and the second utility wraps your handlers so the request data is written to the log:
import (
"aletheiaware.com/netgo"
"aletheiaware.com/netgo/handler"
// ...
)
func main() {
// Configure Logging
logFile, err := netgo.SetupLogging()
if err != nil {
log.Fatal(err)
}
defer logFile.Close()
log.Println("Log File:", logFile.Name())
// ...
mux.Handle("/", handler.Log(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// ...
})))
// ...
}
Step 2: Extract
After running your web server for a while you will have amassed a treasure trove of information in the form of a directory filled with log files. The next step is to parse these files and extract the data.
netgo
includes logparser
which scans through all the log files in the given directory, extracts the request information while ignoring the rest, and populates an SQLite database for easy querying.
$ go install aletheiaware.com/netgo/cmd/logparser
$ logparser logs/
Note: parsing can take quite a few minutes - better go make a nice cup of tea ☕️
Step 3: Analyze
Once all the log data is in a database you'll want to slice, dice, and visualize it so you can recognize trends and identify opportunities.
netgo
includes logserver
which provides a dashboard to examine and understand your server's traffic.
$ go install aletheiaware.com/netgo/cmd/logserver
$ logserver
When you open your browser and navigate to localhost
you'll see a histogram of requests over time, and several bar charts showing which addresses have made the most requests, what are the most popular URLs, and what are the most common HTTP Protocols, Methods, and Headers.
The dashboard in the screenshot below shows the traffic from the first week of the new Perspective website that was announced in a previous article.
-
2022 March 13 16:33 stuartscott 2510¤
Here are some of the other utilities provided by
netgo
to make your life easier.Compression
One of the first insights you might gain from analyzing your logs is that a large proportion of requests will accept a response encoded with
gzip
.Compressing your responses will result in less bytes getting sent across the network, thus shorter page loading times and reduced mobile data charges your users.
netgo
includes a utility to wrap your handler and compress the response if requested:mux.Handle("/", handler.Log(handler.Compress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // ... }))))
HTTPS
Once your project reaches maturity you probably want to consider supporting HTTPS, and redirecting HTTP requests to HTTPS.
netgo
provides a utility that will redirect valid requests of whitelisted routes and return 404s for all others:// Only Requests Addressed to this Host will be Redirected host := "example.com" // Only Requests for these Routes will be Redirected routes := map[string]bool{ "/": true, "/index.html": true, "/favicon.ico": true, } // Redirect HTTP Requests to HTTPS go http.ListenAndServe(":80", http.HandlerFunc(netgo.HTTPSRedirect(host, routes))) // Serve HTTPS Requests server := &http.Server{ Addr: ":443", Handler: mux, TLSConfig: &tls.Config{MinVersion: tls.VersionTLS12}, } if err := server.ListenAndServeTLS(path.Join("certificates", "fullchain.pem"), path.Join("certificates", "privkey.pem")); err != nil { log.Fatal(err) }
Health
Adding health monitoring to your server is as easy as
handler.AttachHealthHandler(mux)
.Any requests to
/health
will return 200.Serving a Static Directory
Serving a static directory (with optional directory listing) is as simple as
mux.Handle("/", handler.Log(handler.StaticDir("html/static", true)))
Setting Response Headers
Set the key/value pairs of your response headers inline;
mux.Handle("/", handler.Log(handler.Header(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // ... }, "key", "value")))
Setting Cache Control Directives
An extension of handler.Header is included for setting cache control directives;
mux.Handle("/", handler.Log(handler.CacheControl(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // ... }, "public, max-age=3600")))
Notice something is missing? Just open a ticket!
Convey is made available by Aletheia Ware under the Terms of Service and Privacy Policy.
Convey is an open-source project released under the Apache 2.0 License and hosted on Github.
© 2021 Aletheia Ware LLC. All rights reserved.