GO Web Services

From bibbleWiki
Revision as of 22:39, 27 August 2021 by Iwiseman (talk | contribs) (→WebSockets)
Jump to navigation Jump to search

Handling Requests

These are the two main ways to handle requests

  • Handle a handler to handle requests matching a pattern
  • HandleFunc a function to handle requests matching a pattern

Handle

Here is the signature for Handle. 

func Handle(pattern string, handler Handler)

Handler is an interface which has the signature 

type Handler interface {
	ServeHTTP(ResponseWriter, *Request)
}

So here is how to use this approach. We create a type and implement the interface required. 

// Creating a handler type
type fooHandler struct {
  Message string
}

// Implementing the function for the interface
func (f *fooHandler) ServeHTTP(w http.ResponseWrite, r *http.Request) {
  w.Write( []byte(f.Message) )
}


func main() {
  http.Handle("/foo", &fooHandler{Message: "hello Word"})
}

Remember Remember we can construct a struct using a struct literal e.g. 

type Student struct {
    Name string
    Age  int
}

pb := &Student{ // pb == &Student{"Bob", 8}
    Name: "Bob",
    Age:  8,
}

Requests

These consist of three parts

  • Method
  • Headers
  • Body

HandleFunc

Perhaps somewhat simpler. We just need to create a function with the correct signature 

func main() {
  foo := funct(w http.ResponseWritter, _ *http.Response) {
    w.Write( []byte("hello world") )
  }

  http.HandleFunc("/foo", foo)
}

Handling JSON

Marshal and Unmarshal

Every GO type implements the empty interface so any struct can be marshaled into JSON. 

func Marshal(v interface{}) (byte[]. error)

To ensure the struct is marshaled, all the fields need to be exported. In this example surname would not be marchaled. 

type foo struct{
  Message string
  Age int
  surname string
}

json,Marshal(&foo{"4Score",56, "Lincoln"})

To Unmarshal 

f := foo{}
json.Unmarshal([]byte(`{"Message":"4Score","Age":56, "Name":"Abe"}`), &f)

Demo Data

Demo Type

Lets make a type to encode 

package product

// Product
type Product struct {
	ProductID      int    `json:"productId"`
	Manufacturer   string `json:"manufacturer"`
	Sku            string `json:"sku"`
	Upc            string `json:"upc"`
	PricePerUnit   string `json:"pricePerUnit"`
	QuantityOnHand int    `json:"quantityOnHand"`
	ProductName    string `json:"productName"`
}

Make Some Demo Data

Here is some demo data. 

[
  {
    "productId": 1,
    "manufacturer": "Johns-Jenkins",
    "sku": "p5z343vdS",
    "upc": "939581000000",
    "pricePerUnit": "497.45",
    "quantityOnHand": 9703,
    "productName": "sticky note"
  },
  {
    "productId": 2,
    "manufacturer": "Hessel, Schimmel and Feeney",
    "sku": "i7v300kmx",
    "upc": "740979000000",
    "pricePerUnit": "282.29",
    "quantityOnHand": 9217,
    "productName": "leg warmers"
  },
...

Below is a example of how to read the json into a map 

func loadProductMap() (map[int]Product, error) {
	fileName := "products.json"
	_, err := os.Stat(fileName)
	if os.IsNotExist(err) {
		return nil, fmt.Errorf("file [%s] does not exist", fileName)
	}

	file, _ := ioutil.ReadFile(fileName)
	productList := make([]Product, 0)
	err = json.Unmarshal([]byte(file), &productList)
	if err != nil {
		log.Fatal(err)
	}
	prodMap := make(map[int]Product)
	for i := 0; i < len(productList); i++ {
		prodMap[productList[i].ProductID] = productList[i]
	}
	return prodMap, nil
}

Implementing Method Types

Create Handler and Determine Mathod

We can now provide a GET method within the handler. The handler determines which method by looking at the request. 

func handleProduct(w http.ResponseWriter, r *http.Request) {
...
	switch r.Method {
            // GET
            // POST
            // etc
        }
}

GET Method

We use the Mnmarshal method to encode the json 

	case http.MethodGet:
		productsJSON, err := json.Marshal(productList)
		if err != nil {
			w.WriteHeader(http.StatusInternalServerError)
			return
		}
		w.Header().Set("Content-Type", "application/json")
		w.Write(productsJSON)

POST Method

We use the Unmarshal method to decode the json. In the source they replaced the ReadAll Unmarshal in favour of json.NewDocder. 

	case http.MethodPost:
		// add a new product to the list

		var newProduct Product

		bodyBytes, err := ioutil.ReadAll(r.Body)
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}
		err = json.Unmarshal(bodyBytes, &newProduct)
		if err != nil {
			w.WriteHeader(http.StatusBadRequest)
			return
		}

		err := json.NewDecoder(r.Body).Decode(&product)
		if err != nil {
			log.Print(err)
			w.WriteHeader(http.StatusBadRequest)
			return
		}


		if newProduct.ProductID != 0 {
			w.WriteHeader(http.StatusBadRequest)
			return
		}
		newProduct.ProductID = getNextID()
		productList = append(productList, newProduct)
		w.WriteHeader(http.StatusCreated)

Managing the Routes

Example Routes

With REST we generally have one route for all products and one for a particular product. e.g. GET /products/123. To manage this we greate different handlers. 

	http.Handle("/products", middlewareHandler(productListHandler))
	http.Handle("/products/", middlewareHandler(productItemHandler))

Managing the URL

Here is an example of getting the 123 from the URL. Note we return a 404 if we cannot determine the resource and not a bad request. 

func productHandler(w http.ResponseWriter, r *http.Request) {
	urlPathSegments := strings.Split(r.URL.Path, "products/")
	productID, err := strconv.Atoi(urlPathSegments[len(urlPathSegments)-1])
	if err != nil {
		log.Print(err)
		w.WriteHeader(http.StatusNotFound)
		return
	}
	product, listItemIndex := findProductByID(productID)
	if product == nil {
		http.Error(w, fmt.Sprintf("no product with id %d", productID), http.StatusNotFound)
		return
	}
	switch r.Method {
	case http.MethodGet:
...

Middeware

Explanation

I liked this example provided in GO. Just made sense to me. This is like the interceptors in Angular. 

func enableCorsMiddleware(handler http.Handler) http.Handler {

  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

  // do stuff before intended handler here 

  handler.ServeHttpp(w,r)

  // do stuff after intended handler here 

})

func intendedFunction(w http.ResponseWriter, r *http.Request) {
  // business logic here
}

func main() {
  intendedHandler := http.HandlerFunc(intendedFunction)
  http.Handle("/foo", enableCorsMiddleware(intendedHandler))
  http.ListenAndServe(":5000", nil)
}

Wrapping Handlers

When using middleware we need to convert our handler to http.handlers. To do this we use the http.handlerFunc. E.g. 

// Before Middleware

//	http.HandleFunc("/products", productsHandler)
//	http.HandleFunc("/products/", productHandler)
//	http.ListenAndServe(":5000", nil)

func main() {
	productListHandler := http.HandlerFunc(productsHandler)
	productItemHandler := http.HandlerFunc(productHandler)

	http.Handle("/products", middlewareHandler(productListHandler))
	http.Handle("/products/", middlewareHandler(productItemHandler))
	http.ListenAndServe(":5000", nil)
}

Yet another CORs

Well love diagrams to here are another two. CORs prevents same-site access. Where means different ports, protocols subdomains are not allowed.


Below is a sample function to enable CORs. Do it, do it, don't do it. 

func enableCorsMiddleware(handler http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Add("Access-Control-Allow-Origin", "*")
		w.Header().Add("Access-Control-Allow-Methods", "POST, GET, OPTIONS, PUT, DELETE")
		w.Header().Add("Access-Control-Allow-Headers", "Accept, Content-Type, Content-Length, Authorization, X-CSRF-Token, Accept-Encoding")
		handler.ServeHTTP(w, r)
	})
}

Persisting Data

Connection to Database

Nothing too difficult here. 

package services

import (
  "database/sql"
  "strconv"
  "log"
)

type DatabaseConnection struct {
  Db *sql.DB
}

func NewDatabaseConnection(
  dbHost string,
  dbPort int,
  dbName, dbUser, dbPass string) *DatabaseConnection {

  var connectionString = 
    dbUser + ":" + dbPass + "@tcp(" + dbHost + ":" + strconv.Itoa(dbPort) + ")/" + 
    dbName + "?parseTime=true&autocommit=false"

   dbConnection, err := sql.Open("mysql", connectionString)
   if err != nil {
    log.Fatal("Error connecting to Database. Error is %s", err.Error())
   }
}

Doing a SELECT

We create a Select Statement using prepare to do this once as it is expenseive 

func CreateSelectStatement(databaseConnection *DatabaseConnection) *sql.Stmt {

  selectStatement, err := databaseConnection.Db.Prepare(
   "SELECT NEW_CASES, NEW_DEATHS, NEW_RECOVERED, TOTAL_CASES, TOTAL_DEATHS, TOTAL_RECOVERED, RECORD_DATE_TIME FROM COVID_DATA WHERE ALPHA2CODE = ? AND YEAR = ? AND DAY = ?")

  if err != nil {
     log.Fatal("Error Creating Select Statement. Error was %s", err.Error())
  }

  return selectStatement
}

Now we can bind the arguments to the statement 

func (covidDataTable *CovidDataTable) Select(
  alpha2Code string,
  year, day int) (bool, *models.CovidData) {

  // Get the rows
  rows, err := covidDataTable.selectStatement.Query(&alpha2Code, &year, &day)
  if err != nil {
    log.Fatal("Error occurred selecting covid data. Error was  %s", err.Error())
    return false, nil
  }
...

And then we can iterate over the results and make sure we close the rows if there is an error. 

...

defer rows.Close()

  ok := rows.Next()
  if !ok {
     log.Fatal("Error occurred Selecting covid data for Country %s, Year %d, Day.", alpha2Code, year, day)
  }

  var newConfirmed, newDeaths, newRecovered int
  var totalConfirmed, totalDeaths, totalRecovered int
  var recordDateTime time.Time

  err = rows.Scan(
    &newConfirmed, &newDeaths, &newRecovered,
    &totalConfirmed, &totalDeaths, &totalRecovered, &recordDateTime)

    if err != nil {
      log.Fatal("Error occurred selecting covid data. Error was  %s", err.Error())
                return false, nil
    }
...

Doing a INSERT

We create a Insert Statement using prepare to do this once as it is expenseive 

func CreateInsertStatement(databaseConnection *DatabaseConnection) *sql.Stmt {

  insertStatement, err := databaseConnection.Db.Prepare(
    "INSERT INTO COVID_DATA(ALPHA2CODE, YEAR, DAY, NEW_CASES, NEW_DEATHS, NEW_RECOVERED, TOTAL_CASES, TOTAL_DEATHS, TOTAL_RECOVERED, RECORD_DATE_TIME) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")

  if err != nil {
     log.Fatal("Error Creating Insert Statement. Error was %s", err.Error())
  }
  return insertStatement
}

And now we can to the insert 

func (covidDataTable *CovidDataTable) Insert(
  tx *sql.Tx,
  alpha2Code string,
  year, day int,
  covidData *models.CovidData) bool {

  // Do the insert
  result, err := tx.Stmt(covidDataTable.insertStatement).Exec(
    &alpha2Code,
    &year,
    &day,
    covidData.NewConfirmed,
    covidData.NewDeaths,
    covidData.NewRecovered,
    covidData.TotalConfirmed,
    covidData.TotalDeaths,
    covidData.TotalRecovered,
    covidData.RecordDateTime)

    var myError = covidDataTable.insertStatement.Close()
    if myError != nil {
      log.Fatal("Yes my son %s", myError.Error())
    }

    // Check for Error
    if err != nil {
      log.Fatal("Error occurred inserting covid data. Error Executing statement. Error was  %s", err.Error())
    }

    rows, err := result.RowsAffected()
    if err != nil {
      logFatal("Error occurred inserting covid data.  Error getting row affected. Error was  %s", err.Error())
    }

    // All good
    return true
}

Managing Database Resources

Connection Pools

These are the parameters available.

  • Connection Max Lifetime - Sets maximum amount of time a connection may be used
  • Max Idle Connections - Sets the maxium number in idle connection pool - default is 2
  • Max Open Connections - Sets the maximum open connections

Using Contexts

These allow you to perform tasks with timeouts. Useful to ensure resource are no exhausted. 

func getProduct(productID int) (*Product, error) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()
	row := database.DbConn.QueryRowContext(ctx, `SELECT 
	productId, 
	manufacturer, 
	sku, 
	upc, 
	pricePerUnit, 
	quantityOnHand, 
	productName 
	FROM products 
	WHERE productId = ?`, productID)

	product := &Product{}
	err := row.Scan(
		&product.ProductID,
		&product.Manufacturer,
		&product.Sku,
		&product.Upc,
		&product.PricePerUnit,
		&product.QuantityOnHand,
		&product.ProductName,
	)

	if err == sql.ErrNoRows {
		return nil, nil
	} else if err != nil {
		log.Println(err)
		return nil, err
	}
	return product, nil
}

File Uploading

To upload files we could

  • encode the file to base64 and convert to Json
  • multipart/form-data type which uses HTTP form submit

Encode

To decode the string we would need to 

str := "SGVsbG8gV29ybGQ="

output, err := base64.StdEncoding.DecodString(str)
if err != nill {
  log.Fatal(err)
}

fmt.Printf("%q\n", output)

Using Multipart

Uploading

The ParsMultiPartForm will start using disk if the limit is exceeded.
For the roll left bit 

op1 << op2 Shift bits of op1 left by distance op2; fills with zero bits on the right-hand side
5    << 20
0101 << 00000000000000000000
010100000000000000000000 (binary) = 50000 (Bytes)


func uploadFileHandler(w http.ResponseWriter, r *http.Request) {
  r.ParseMultipartForm(5 << 20) // 5 Mb
  file, handler, err := r.FormFile("receipt")
  if err != nil {
    log.Print(err)
    w.WriteHeader(http.StatusBadRequest)
    return
  }

  defer file.Close()

  f, err := os.OpenFile(filepath.Join(ReceiptDirectory, handler.Filename), os.O_WRONLY|os.O_CREATE, 0666)
  
   defer f.Close()
   io.Copy(f, file)
}

Downloading

And here is an example of Downloading 

  fileName := urlPathSegments[1:][0]
  file, err := os.Open(filepath.Join(ReceiptDirectory, fileName))
  defer file.Close()
  if err != nil {
	w.WriteHeader(http.StatusNotFound)
	return
  }
  
  fHeader := make([]byte, 512)
  file.Read(fHeader)
  fContentType := http.DetectContentType(fHeader)

  stat, err := file.Stat()
  if err != nil {
	w.WriteHeader(http.StatusInternalServerError)
	return
  }

  fSize := strconv.FormatInt(stat.Size(), 10)
  w.Header().Set("Content-Disposition", "attachment; filename="+fileName)
  w.Header().Set("Content-Type", fContentType)
  w.Header().Set("Content-Length", fSize)
  file.Seek(0, 0)
  io.Copy(w, file)

WebSockets

Introduction

Previously we used to poll for changes. With WebSockets we can solve this problem.

  • Client sends HTTP GET request
    • Connection: Upgrade
    • Upgrade: Websocket
    • Sec-WebSocket-Key: key
  • Server Responds with status code "101"
    • Switching Protocols
    • Upgrade: Websocket
    • Connection: Upgrade
    • Sec-WebSocket-Accept: key

Creating WebSocket

For GO the package is golang.org/x which means
These packages are part of the Go Project but outside the main Go tree. They are developed under looser compatibility requirements than the Go core. Install them with "go get".

I understand there are alternatives. Googling showed Gorilla to be one of the most popular.

So we are going to use golang. Let create a handler. 

package test

import (
	"net/http"

	"golang.org/x/net/websocket"
)

func SetupRoutes(apiBasePath string) {
	http.Handle("/websocket", websocket.Handler(testSocket))
}

The interface is 

type Handler func(*Conn)

Now implement the handler. 

package test

import (
	"fmt"
	"log"
	"time"

	"golang.org/x/net/websocket"
)

type message struct {
	Data string `json:"data"`
	Type string `json:"type"`
}

func testSocket(ws *websocket.Conn) {

	// we can verify that the origin is an allowed origin
	fmt.Printf("origin: %s\n", ws.Config().Origin)

	done := make(chan struct{})
	go func(c *websocket.Conn) {
		for {
			var msg message
			if err := websocket.JSON.Receive(ws, &msg); err != nil {
				log.Println(err)
				break
			}
			fmt.Printf("received message %s\n", msg.Data)
		}
		close(done)
	}(ws)

loop:
	for {

		select {

		// End the infinate loop
		case <-done:
			fmt.Println("connection was closed, lets break out of here")
			break loop

		// Do Send of Data
		default:
			// Construct Data
			testData := TestData{"FRED1", "FRED2"}

			if err := websocket.JSON.Send(ws, testData); err != nil {
				log.Println(err)
				break
			}

			fmt.Println("Send some data")

			// pause for 10 seconds before sending again
			time.Sleep(10 * time.Second)
		}
	}

	fmt.Println("closing the connection")
	defer ws.Close()
}

Testing a Websocket

We can use the devtools within chrome 

let ws = new WebSocket("ws://localhost/localhost:5000/websocket")
ws.send(JSON.stringify({"data":"test data", "type":"Test"})

Templates

Introduction

This is like pug and esj where you provide data and a markup which has placeholders for the data. 

package main

import (
  "html/template"
  "os"
)

type BlogPost struct {
  Title   string
  Content string
}

func main() {
  post := BlogPost{"First Post", "This is the blog post content section"}
  tmpl, err := template.New("blog-tmpl").Parse("<h1>{{.Title}}</h1><div><p>{{.Content}}</p></div>")
  if err != nil {
	panic(err)
  }
  err = tmpl.Execute(os.Stdout, post)
  if err != nil {
	panic(err)
  }
}

Pipelines

This is a way of chaining operations together in a template. 

  // Pass argument to function .SaySomething(text string)
  {{ .SaySomething " Hello "}}

  // Same as
  {{ " Hello " | .SaySomething }}

  // And
  {{ " Hello " | .SaySomething | printf "%s %s" " World"  }}

Looping

I have not checked the code and just wanted an awareness of the concept. I think I would prefer to format on the client but this could be useful when connecting to other reporting system. No surprises but here is an example. Here is the template 

<!doctype html>
<head>
</head>
<body style="height: 100%;" }>
<div style="background: #6a7d87; width: 100%; z-index: 100; top: 0;
  left: 0; box-shadow: 0 1px 20px transparentize($second-color, 0.5);">
</div>

<div style="font-size: 1.5rem; font-weight: bold; color: #212529; display: block; font-family: Roboto, 'Helvetica Neue', sans-serif;font-stretch: normal; font-weight: bold; text-align: left; padding: .75rem 1.25rem; background-color: rgba(0,0,0,.03); border-bottom: 1px solid rgba(0,0,0,.125); ">
Product Summary Report
</div>
  <table style="width: 100%; height: 100%; margin-top: .5em;">
    <tr>
      <th>Row</th>
      <th>Product Name</th>
      <th>Quantity On Hand</th>
    </tr>
    {{range $index, $element := .}}
       {{if mod $index 2}} <tr style="background:#6a7d87;"> {{else}} 
    <tr> {{end}}
      <td>
        {{$index}}
      </td>
      <td>
	{{.ProductName}}
      </td>
      <td>
	{{.QuantityOnHand}}
      </td>
     </tr>
     {{end}}
    </table>

</body>
</html>

And the code for the template demonstrating looping. 

    t := template.New("report.gotmpl").Funcs(template.FuncMap{"mod": func(i, x int) bool { return i%x == 0 }})
    t, err = t.ParseFiles(path.Join("templates", "report.gotmpl"))
    if err != nil {
	log.Print(err)
	w.WriteHeader(http.StatusInternalServerError)
	return
    }

    w.Header().Set("Content-Disposition", "Attachment")
    var tpl bytes.Buffer
    err = t.Execute(&tpl, products)

Functions

There are functions just like ejs and pug. The ones listed and here is the link https://pkg.go.dev/text/template#hdr-Functions 

*and {{if and true true true}} {{end}}
*or  {{if or true false true}} {{end}}
*index {{index . 1}}
*len {{len .}}
*not {{if not false}}
*print, printf, println {{println "hey"}}

Custom Functions

We can write our own function but they must return a single value or value,err. 

// Usage
tmpl := "{{range &index, $element := .}}{{if mod $index 2}}{{.}}{{end}}}{{end}}}"

// Declare
fm := template.FuncMap{"mod": func(i,j int) bool {return i%} == 0 }}
// Add to Template
tmpl, _ := template.New("tmplt").Funcs(fm).Parse(tmpl)
Retrieved from "https://www.bibble.co.nz/mediawiki/index.php?title=GO_Web_Services&oldid=5786"